--- library_name: transformers license: apache-2.0 base_model: tsavage68/Na_M2_1000steps_1e7_SFT tags: - trl - dpo - generated_from_trainer model-index: - name: Na_M2_1000steps_1e7rate_03beta_cSFTDPO results: [] --- # Na_M2_1000steps_1e7rate_03beta_cSFTDPO This model is a fine-tuned version of [tsavage68/Na_M2_1000steps_1e7_SFT](https://huggingface.co/tsavage68/Na_M2_1000steps_1e7_SFT) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 0.0000 - Rewards/chosen: 3.5522 - Rewards/rejected: -11.1817 - Rewards/accuracies: 1.0 - Rewards/margins: 14.7339 - Logps/rejected: -117.1956 - Logps/chosen: -36.2917 - Logits/rejected: -2.4993 - Logits/chosen: -2.5149 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 1e-07 - train_batch_size: 2 - eval_batch_size: 1 - seed: 42 - gradient_accumulation_steps: 2 - total_train_batch_size: 4 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 100 - training_steps: 1000 ### Training results | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen | |:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:| | 0.0004 | 0.2667 | 50 | 0.0000 | 2.3360 | -8.0685 | 1.0 | 10.4045 | -106.8185 | -40.3458 | -2.5169 | -2.5309 | | 0.0 | 0.5333 | 100 | 0.0000 | 2.8383 | -9.2541 | 1.0 | 12.0924 | -110.7703 | -38.6713 | -2.5103 | -2.5250 | | 0.0 | 0.8 | 150 | 0.0000 | 3.0920 | -9.7555 | 1.0 | 12.8475 | -112.4418 | -37.8257 | -2.5067 | -2.5217 | | 0.0 | 1.0667 | 200 | 0.0000 | 3.2037 | -10.0769 | 1.0 | 13.2806 | -113.5130 | -37.4533 | -2.5051 | -2.5202 | | 0.0 | 1.3333 | 250 | 0.0000 | 3.2784 | -10.3241 | 1.0 | 13.6025 | -114.3372 | -37.2044 | -2.5046 | -2.5198 | | 0.0 | 1.6 | 300 | 0.0000 | 3.3562 | -10.5498 | 1.0 | 13.9060 | -115.0894 | -36.9450 | -2.5033 | -2.5186 | | 0.0 | 1.8667 | 350 | 0.0000 | 3.4141 | -10.7123 | 1.0 | 14.1265 | -115.6312 | -36.7520 | -2.5019 | -2.5173 | | 0.0 | 2.1333 | 400 | 0.0000 | 3.4694 | -10.8608 | 1.0 | 14.3302 | -116.1261 | -36.5679 | -2.5020 | -2.5174 | | 0.0 | 2.4 | 450 | 0.0000 | 3.4912 | -10.9759 | 1.0 | 14.4671 | -116.5096 | -36.4950 | -2.5011 | -2.5165 | | 0.0 | 2.6667 | 500 | 0.0000 | 3.5172 | -11.0415 | 1.0 | 14.5587 | -116.7282 | -36.4083 | -2.5010 | -2.5165 | | 0.0 | 2.9333 | 550 | 0.0000 | 3.5281 | -11.1219 | 1.0 | 14.6500 | -116.9964 | -36.3719 | -2.4999 | -2.5154 | | 0.0 | 3.2 | 600 | 0.0000 | 3.5544 | -11.1376 | 1.0 | 14.6920 | -117.0486 | -36.2843 | -2.4985 | -2.5140 | | 0.0 | 3.4667 | 650 | 0.0000 | 3.5412 | -11.1686 | 1.0 | 14.7098 | -117.1519 | -36.3284 | -2.4993 | -2.5149 | | 0.0 | 3.7333 | 700 | 0.0000 | 3.5592 | -11.1405 | 1.0 | 14.6997 | -117.0585 | -36.2685 | -2.4988 | -2.5143 | | 0.0 | 4.0 | 750 | 0.0000 | 3.5602 | -11.1575 | 1.0 | 14.7177 | -117.1151 | -36.2652 | -2.4993 | -2.5148 | | 0.0 | 4.2667 | 800 | 0.0000 | 3.5534 | -11.1617 | 1.0 | 14.7151 | -117.1290 | -36.2877 | -2.4996 | -2.5152 | | 0.0 | 4.5333 | 850 | 0.0000 | 3.5623 | -11.1612 | 1.0 | 14.7234 | -117.1272 | -36.2582 | -2.4994 | -2.5150 | | 0.0 | 4.8 | 900 | 0.0000 | 3.5522 | -11.1817 | 1.0 | 14.7339 | -117.1956 | -36.2917 | -2.4993 | -2.5149 | | 0.0 | 5.0667 | 950 | 0.0000 | 3.5522 | -11.1817 | 1.0 | 14.7339 | -117.1956 | -36.2917 | -2.4993 | -2.5149 | | 0.0 | 5.3333 | 1000 | 0.0000 | 3.5522 | -11.1817 | 1.0 | 14.7339 | -117.1956 | -36.2917 | -2.4993 | -2.5149 | ### Framework versions - Transformers 4.44.2 - Pytorch 2.4.0+cu121 - Datasets 2.21.0 - Tokenizers 0.19.1