Hugging Face's logo

sfulay
/
zephyr-7b-dpo-full-hh

Text Generation
Transformers
Safetensors
mistral
alignment-handbook
trl
dpo
Generated from Trainer
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
zephyr-7b-dpo-full-hh / README.md
sfulay's picture
sfulay
End of training
a67679b verified
preview code
|
raw
history blame contribute delete
4.39 kB
metadata
license: apache-2.0
base_model: alignment-handbook/zephyr-7b-sft-full
tags:
  - alignment-handbook
  - trl
  - dpo
  - generated_from_trainer
  - trl
  - dpo
  - generated_from_trainer
datasets:
  - Anthropic/hh-rlhf
model-index:
  - name: zephyr-7b-dpo-full-hh
    results: []

zephyr-7b-dpo-full-hh

This model is a fine-tuned version of alignment-handbook/zephyr-7b-sft-full on the Anthropic/hh-rlhf dataset. It achieves the following results on the evaluation set:

  • Loss: 0.5341
  • Rewards/chosen: -2.5112
  • Rewards/rejected: -3.3276
  • Rewards/accuracies: 0.7295
  • Rewards/margins: 0.8164
  • Logps/rejected: -485.8198
  • Logps/chosen: -398.0228
  • Logits/rejected: 2.4839
  • Logits/chosen: 1.8909

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-07
  • train_batch_size: 8
  • eval_batch_size: 8
  • seed: 55
  • distributed_type: multi-GPU
  • num_devices: 8
  • gradient_accumulation_steps: 2
  • total_train_batch_size: 128
  • total_eval_batch_size: 64
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 1

Training results

Training Loss Epoch Step Validation Loss Rewards/chosen Rewards/rejected Rewards/accuracies Rewards/margins Logps/rejected Logps/chosen Logits/rejected Logits/chosen
0.6755 0.0796 100 0.6758 -0.1176 -0.1618 0.5830 0.0442 -169.2391 -158.6626 -2.4575 -2.4733
0.5965 0.1592 200 0.5934 -1.0644 -1.4934 0.6772 0.4290 -302.3988 -253.3484 -0.2911 -0.5108
0.5621 0.2388 300 0.5712 -1.3390 -1.8901 0.6875 0.5511 -342.0688 -280.8055 0.3164 -0.1607
0.551 0.3183 400 0.5651 -1.2110 -1.8230 0.7192 0.6120 -335.3575 -268.0063 -0.2086 -0.6012
0.5696 0.3979 500 0.5572 -1.8231 -2.3956 0.7229 0.5725 -392.6161 -329.2127 0.5151 0.0872
0.5504 0.4775 600 0.5508 -1.9233 -2.7091 0.7201 0.7858 -423.9663 -339.2298 1.2370 0.4867
0.5387 0.5571 700 0.5417 -2.3329 -3.1152 0.7211 0.7823 -464.5798 -380.1928 1.9107 1.1807
0.5119 0.6367 800 0.5416 -2.6721 -3.5027 0.7276 0.8306 -503.3281 -414.1180 3.4443 2.7209
0.564 0.7163 900 0.5385 -2.6361 -3.3606 0.7183 0.7245 -489.1202 -410.5185 2.5529 1.9952
0.5201 0.7959 1000 0.5347 -2.5021 -3.2845 0.7229 0.7824 -481.5121 -397.1160 2.4306 1.8888
0.5341 0.8754 1100 0.5346 -2.4898 -3.2841 0.7295 0.7943 -481.4664 -395.8830 2.3851 1.8147
0.5394 0.9550 1200 0.5341 -2.5107 -3.3276 0.7295 0.8168 -485.8161 -397.9764 2.4847 1.8912

Framework versions

  • Transformers 4.41.2
  • Pytorch 2.1.2
  • Datasets 2.20.0
  • Tokenizers 0.19.1