|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- Anthropic/hh-rlhf |
|
language: |
|
- en |
|
pipeline_tag: text-generation |
|
tags: |
|
- text-generation-inference |
|
--- |
|
# Model Card for OpenBezoar-HH-RLHF-DPO |
|
|
|
The OpenBezoar-HH-RLHF-DPO is an LLM that has been fine tuned for human preferences alignment using Direct Preference Optimization (DPO), on top of [OpenBezoar-HH-RLHF-SFT](https://huggingface.co/SurgeGlobal/OpenBezoar-HH-RLHF-SFT) model on a subset of [Anthropic's HH-RLHF Dataset](https://huggingface.co/datasets/Anthropic/hh-rlhf). |
|
|
|
## Model Details |
|
|
|
- Base Model: [OpenBezoar-HH-RLHF-SFT](https://huggingface.co/SurgeGlobal/OpenBezoar-HH-RLHF-SFT) |
|
- Dataset used for SFT: First 100K examples of the [HH-RLHF](https://huggingface.co/datasets/Anthropic/hh-rlhf) dataset |
|
- Alignment Method: [Direct Preference Optimization (DPO)](https://arxiv.org/abs/2305.18290) |
|
- Epochs: 1 |
|
|
|
### Model Description |
|
|
|
OpenBezoar-HH-RLHF-SFT is an LLM that is built upon the OpenLLaMA 3B v2 architecture. This model has been fine-tuned for human preferences alignment using DPO. Alignment has been performed on top of the [OpenBezoar-HH-RLHF-SFT](https://huggingface.co/SurgeGlobal/OpenBezoar-HH-RLHF-SFT) model. For more information please refer to our paper. |
|
|
|
### Model Sources |
|
|
|
- **Repository:** [More Information Needed] |
|
- **Paper :** [More Information Needed] |
|
|
|
## Instruction Format |
|
|
|
We follow the typical format for instruction-based prompt templates, with a system prompt followed up by the user prompt. Both begins with a prefix and ends with two newline characters as described below. It is important to utilize this template in order to obtain best responses for instruction fine-tuning related tasks. |
|
``` |
|
### System: {system} |
|
|
|
### Instruction: {instruction} |
|
|
|
### Response: |
|
``` |
|
|
|
Notice that **no** end-of-sentence (eos) token is being appended. |
|
|
|
## Limitations |
|
|
|
- The model might not consistently show improved abilities to follow instructions, and it could respond inappropriately or get stuck in loops. |
|
- Although this model is aligned to human preferences and has been evaluated for performance, it is not guaranteed that it will **refrain** from generating harmful content exclusively. |
|
- Caution is urged against relying on this model for production or adjacent use-cases. |
|
|
|
## Citation |
|
|
|
If you find our work useful, please cite our paper as follows: |
|
|
|
``` |
|
[More Information Needed] |
|
``` |
