arceeai-cpt-sft-dpo-full

This model was trained from scratch on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.4290
• Rewards/chosen: -2.1400
• Rewards/rejected: -3.3730
• Rewards/accuracies: 0.7680
• Rewards/margins: 1.2330
• Logps/rejected: -667.6970
• Logps/chosen: -599.8088
• Logits/rejected: -3.8772
• Logits/chosen: -3.7995

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-07
• train_batch_size: 4
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 2
• total_train_batch_size: 64
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6051	0.1	100	0.5848	-0.2214	-0.5948	0.6920	0.3733	-389.8707	-407.9477	-3.9800	-3.8777
0.5134	0.21	200	0.5025	-1.5935	-2.5024	0.7160	0.9089	-580.6380	-545.1561	-3.9154	-3.8176
0.4489	0.31	300	0.4614	-1.5620	-2.6097	0.7760	1.0477	-591.3610	-542.0072	-3.7594	-3.6703
0.4359	0.42	400	0.4467	-2.0879	-3.2160	0.7680	1.1281	-651.9947	-594.5918	-3.7022	-3.6221
0.4271	0.52	500	0.4441	-2.0549	-3.2181	0.7840	1.1631	-652.2027	-591.3006	-3.8189	-3.7408
0.4181	0.63	600	0.4366	-1.9876	-3.1678	0.7760	1.1802	-647.1777	-584.5698	-3.7950	-3.7170
0.4	0.73	700	0.4317	-2.1647	-3.3521	0.7640	1.1874	-665.6046	-602.2762	-3.8739	-3.7970
0.4123	0.84	800	0.4291	-2.2039	-3.4491	0.7680	1.2453	-675.3075	-606.1934	-3.8606	-3.7827
0.4394	0.94	900	0.4292	-2.1325	-3.3633	0.7680	1.2308	-666.7250	-599.0574	-3.8777	-3.8001

Framework versions

• Transformers 4.36.2
• Pytorch 2.1.2+cu121
• Datasets 2.16.1
• Tokenizers 0.15.0