---
license: apache-2.0
datasets:
- Anthropic/hh-rlhf
language:
- en
pipeline_tag: text-generation
tags:
- text-generation-inference
---
# Model Card for OpenBezoar-HH-RLHF-DPO

The OpenBezoar-HH-RLHF-DPO is an LLM that has been fine tuned for human preferences alignment using Direct Preference Optimization (DPO), on top of [OpenBezoar-HH-RLHF-SFT](https://huggingface.co/SurgeGlobal/OpenBezoar-HH-RLHF-SFT) model on a subset of [Anthropic's HH-RLHF Dataset](https://huggingface.co/datasets/Anthropic/hh-rlhf).

## Model Details

- Base Model: [OpenBezoar-HH-RLHF-SFT](https://huggingface.co/SurgeGlobal/OpenBezoar-HH-RLHF-SFT)
- Dataset used for SFT: First 100K examples of the [HH-RLHF](https://huggingface.co/datasets/Anthropic/hh-rlhf) dataset
- Alignment Method: [Direct Preference Optimization (DPO)](https://arxiv.org/abs/2305.18290)
- Epochs: 1

### Model Description

OpenBezoar-HH-RLHF-SFT is an LLM that is built upon the OpenLLaMA 3B v2 architecture. This model has been fine-tuned for human preferences alignment using DPO. Alignment has been performed on top of the [OpenBezoar-HH-RLHF-SFT](https://huggingface.co/SurgeGlobal/OpenBezoar-HH-RLHF-SFT) model. For more information please refer to our paper.

### Model Sources

- **Repository:** [More Information Needed]
- **Paper :** [More Information Needed]

## Instruction Format

We follow the typical format for instruction-based prompt templates, with a system prompt followed up by the user prompt. Both begins with a prefix and ends with two newline characters as described below. It is important to utilize this template in order to obtain best responses for instruction fine-tuning related tasks.
```
### System: {system}

### Instruction: {instruction}

### Response:
```

Notice that **no** end-of-sentence (eos) token is being appended.

## Limitations

- The model might not consistently show improved abilities to follow instructions, and it could respond inappropriately or get stuck in loops.
- Although this model is aligned to human preferences and has been evaluated for performance, it is not guaranteed that it will **refrain** from generating harmful content exclusively.
- Caution is urged against relying on this model for production or adjacent use-cases.

## Citation

If you find our work useful, please cite our paper as follows:

```
[More Information Needed]
```