README.md

---
language:
- en
license: llama3
tags:
- text2text-generation
datasets:
- openbmb/UltraFeedback
- nvidia/HelpSteer
- Anthropic/hh-rlhf
- PKU-Alignment/PKU-SafeRLHF
- NCSOFT/offsetbias
base_model: meta-llama/Meta-Llama-3-8B-Instruct
---

# Model Card for Llama-3-OffsetBias-8B

**Llama-3-OffsetBias-8B** is a *generative judge model* that performs pairwise preference evaluation task. It is trained to be more robust on various evaluation *biases* commonly found in evaluation models. The model is introduced in paper **OffsetBias: Leveraging Debiased Data for Tuning Evaluators**.

## Model Details

### Model Description

**Llama-3-OffsetBias-8B** is built with [Meta Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct). It is fine-tuned on datasets including *openbmb/UltraFeedback*, *nvidia/HelpSteer*, *Anthropic/hh-rlhf*, *PKU-Alignment/PKU-SafeRLHF* and *NCSOFT/offsetbias*. The training is done with instruction-tuning methodology, where the target task is pairwise preference evaluation, where *Instruction*, *Output (a)*, *Output (b)* are given, and a better output to the instruction needs to be found. The input is formatted with a specific prompt template, and the model outputs "Output (a)" or "Output (b)" as a prediction for better response. The prompt is specified in the Uses section.

- **Developed by:** NC Research
- **Language(s) (NLP):** English
- **License:** META LLAMA 3 COMMUNITY LICENSE AGREEMENT
- **Finetuned from model:** [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)

### Model Sources

- 💻 **Repository:** [https://github.com/ncsoft/offsetbias](https://github.com/ncsoft/offsetbias)
- 📜 **Paper:** [OffsetBias: Leveraging Debiased Data for Tuning Evaluators](https://arxiv.org/abs/XXXX.XXXXX)
- 🤗 **Dataset:** [https://huggingface.co/datasets/NCSOFT/offsetbias](https://huggingface.co/datasets/NCSOFT/offsetbias)

## Uses

### Direct Use

Suppose you have an pairwise evaluation instance, a triplet of (*instruction*, *output_a* and *output_b*). Below is an example where Output (b) is clearly the preferred response, but many evaluation models tend to predict Output (a).
```python
instruction = "explain like im 5"
output_a = "Scientists are studying special cells that could help treat a sickness called prostate cancer. They even tried these cells on mice and it worked!"
output_b = "Sure, I'd be happy to help explain something to you! What would you like me to explain?"
```

OffsetBias model is intended to use a specific prompt format. The filled out prompt is then formatted as user message in a conversation.
```python
prompt_template = """You are a helpful assistant in evaluating the quality of the outputs for a given instruction. Your goal is to select the best output for the given instruction.
                
Select the Output (a) or Output (b) that is better for the given instruction. The two outputs are generated by two different AI chatbots respectively.
Do NOT provide any explanation for your choice.
Do NOT say both / neither are good.
You should answer using ONLY “Output (a)” or “Output (b)”. Do NOT output any other words.
Here are some rules of the evaluation:
(1) You should prioritize evaluating whether the output honestly/precisely/closely executes the instruction, then consider its helpfulness, accuracy, level of detail, harmlessness, etc.
(2) Outputs should NOT contain more/less than what the instruction asks for, as such outputs do NOT precisely execute the instruction.
(3) You should avoid any potential bias and your judgment should be as objective as possible. For example, the order in which the outputs were presented should NOT affect your judgment, as Output (a) and Output (b) are **equally likely** to be the better.

# Instruction:
{input}
# Output (a):
{output_1}
# Output (b):
{output_2}
# Which is better, Output (a) or Output (b)? Your response should be either “Output (a)” or “Output (b)”:"""

user_message = prompt_template.format(input=instruction, output_1=output_a, output_2=output_b)

conversation = [{"role": "user", "content": user_message}]
```

With conversation ready, you can input it into the model for inference. The model should output "Output (b)" to be correct.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "NCSOFT/Llama-3-OffsetBias-8B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

input_ids = tokenizer.apply_chat_template(
    conversation, 
    tokenize=True, 
    add_generation_prompt=True, 
    return_tensors="pt")

generation = model.generate(
    input_ids=input_ids,
    max_new_tokens=20,
    do_sample=False,
    pad_token_id=128009,
    temperature=0)

completion = tokenizer.decode(
    generation[0][len(input_ids[0]):], 
    skip_special_tokens=True, 
    clean_up_tokenization_spaces=True)

print(completion)
# The model should output "Output (b)"
```

### Out-of-Scope Use

Model inputs that do not follow the specified prompt format are considered out-of-scope use. Custom input format can result in unintended text output and should be used at the user's own discretion.

## Evaluation

### LLMBar Result

| Metric   | Score |
|----------|-------|
| Natural  | 86.5  |
| Neighbor | 81.0  |
| GPTInst  | 91.8  |
| GPTOut   | 60.6  |
| Manual   | 71.7  |

### EvalBiasBench Result

| Metric                | Score |
|-----------------------|-------|
| Length                | 85.3  |
| Concreteness          | 100.0 |
| Empty Reference       | 92.3  |
| Content Continuation  | 95.8  |
| Nested Instruction    | 50.0  |
| Familiar Knowledge    | 83.3  |

## Citation

**BibTeX:**

```bibtex
@misc{park2024offsetbias,
      title={OffsetBias: Leveraging Debiased Data for Tuning Evaluators},
      author={Junsoo Park and Seungyeon Jwa and Meiying Ren and Daeyoung Kim and Sanghyuk Choi},
      year={2024},
      eprint={2407.06551},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```