---
license: apache-2.0
base_model: openchat/openchat-3.5-0106
datasets:
- berkeley-nest/Nectar
---
This is openchat/openchat-3.5-0106, tuned with DPO on a subset Nectar. This time with 5000 steps, a full epoch.

Careful attention was paid to make sure the chat template was followed properly.

Data selection and filtering:
- filtered dataset to only include examples with multiple turns, to preserve strength in multi-turn scenarios
- used the 4th ranking response as the "rejected" instead of the 3rd. When I inspected the dataset, I frequently could not find any meaningful difference in quality between the 1st and 3rd ranked responses, so to make the accepted/rejected signal extra clear, I replaced 3rd ranking with 4th ranking.
- I filtered out any examples with "good_natured == False". Why? When I inspected examples with "good_natured == False" in the Nectar dataset, I noticed they frequently include refusals from even the top ranking model. So, counter-intuitively, including "bad natured" entries might actually censor the model *more*, since the top responses (as ranked by GPT-4) to these queries tend to be refusals. Not to mention, the quality of the conversations that are "bad natured" tends to be worse in general, in my own opinion.

Differences from 0.4:
- Trained on 5000 steps instead of 500, with a lower learning rate and slower warmup period.