mistral_dpo1

This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.6658
• Rewards/chosen: 0.2577
• Rewards/rejected: 0.1951
• Rewards/accuracies: 0.6588
• Rewards/margins: 0.0625
• Logps/rejected: -48.8060
• Logps/chosen: -74.7955
• Logits/rejected: -0.5381
• Logits/chosen: -0.8076

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-07
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 40
• training_steps: 200

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6891	0.1598	40	0.6869	0.0983	0.0855	0.6470	0.0129	-49.9027	-76.3890	-0.5382	-0.8076
0.6782	0.3195	80	0.6772	0.2135	0.1789	0.6494	0.0346	-48.9688	-75.2377	-0.5385	-0.8080
0.6719	0.4793	120	0.6712	0.2187	0.1699	0.6635	0.0488	-49.0587	-75.1852	-0.5383	-0.8078
0.6635	0.6390	160	0.6675	0.2557	0.1974	0.6635	0.0584	-48.7838	-74.8149	-0.5381	-0.8077
0.655	0.7988	200	0.6658	0.2577	0.1951	0.6588	0.0625	-48.8060	-74.7955	-0.5381	-0.8076

Framework versions

• PEFT 0.11.1
• Transformers 4.42.4
• Pytorch 2.3.1+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1