mistral_dpo3

This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.6493
• Rewards/chosen: -0.0199
• Rewards/rejected: -0.1207
• Rewards/accuracies: 0.6902
• Rewards/margins: 0.1008
• Logps/rejected: -49.8290
• Logps/chosen: -78.6185
• Logits/rejected: -0.5023
• Logits/chosen: -0.7841

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-07
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 40
• training_steps: 200

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6892	0.1236	40	0.6854	-0.0095	-0.0254	0.6975	0.0159	-48.8761	-78.5142	-0.5066	-0.7881
0.6746	0.2473	80	0.6703	0.0082	-0.0405	0.6957	0.0487	-49.0272	-78.3371	-0.5048	-0.7865
0.6579	0.3709	120	0.6587	0.0022	-0.0744	0.6938	0.0766	-49.3662	-78.3975	-0.5036	-0.7854
0.6539	0.4946	160	0.6516	-0.0128	-0.1074	0.6902	0.0946	-49.6963	-78.5475	-0.5026	-0.7844
0.6464	0.6182	200	0.6493	-0.0199	-0.1207	0.6902	0.1008	-49.8290	-78.6185	-0.5023	-0.7841

Framework versions

• PEFT 0.12.0
• Transformers 4.43.3
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1