metadata
library_name: transformers
license: apache-2.0
base_model: tsavage68/Na_M2_1000steps_1e7_SFT
tags:
- trl
- dpo
- generated_from_trainer
model-index:
- name: Na_M2_1000steps_1e7rate_03beta_cSFTDPO
results: []
Na_M2_1000steps_1e7rate_03beta_cSFTDPO
This model is a fine-tuned version of tsavage68/Na_M2_1000steps_1e7_SFT on an unknown dataset. It achieves the following results on the evaluation set:
- Loss: 0.0000
- Rewards/chosen: 3.5522
- Rewards/rejected: -11.1817
- Rewards/accuracies: 1.0
- Rewards/margins: 14.7339
- Logps/rejected: -117.1956
- Logps/chosen: -36.2917
- Logits/rejected: -2.4993
- Logits/chosen: -2.5149
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-07
- train_batch_size: 2
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 100
- training_steps: 1000
Training results
| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.0004 | 0.2667 | 50 | 0.0000 | 2.3360 | -8.0685 | 1.0 | 10.4045 | -106.8185 | -40.3458 | -2.5169 | -2.5309 |
| 0.0 | 0.5333 | 100 | 0.0000 | 2.8383 | -9.2541 | 1.0 | 12.0924 | -110.7703 | -38.6713 | -2.5103 | -2.5250 |
| 0.0 | 0.8 | 150 | 0.0000 | 3.0920 | -9.7555 | 1.0 | 12.8475 | -112.4418 | -37.8257 | -2.5067 | -2.5217 |
| 0.0 | 1.0667 | 200 | 0.0000 | 3.2037 | -10.0769 | 1.0 | 13.2806 | -113.5130 | -37.4533 | -2.5051 | -2.5202 |
| 0.0 | 1.3333 | 250 | 0.0000 | 3.2784 | -10.3241 | 1.0 | 13.6025 | -114.3372 | -37.2044 | -2.5046 | -2.5198 |
| 0.0 | 1.6 | 300 | 0.0000 | 3.3562 | -10.5498 | 1.0 | 13.9060 | -115.0894 | -36.9450 | -2.5033 | -2.5186 |
| 0.0 | 1.8667 | 350 | 0.0000 | 3.4141 | -10.7123 | 1.0 | 14.1265 | -115.6312 | -36.7520 | -2.5019 | -2.5173 |
| 0.0 | 2.1333 | 400 | 0.0000 | 3.4694 | -10.8608 | 1.0 | 14.3302 | -116.1261 | -36.5679 | -2.5020 | -2.5174 |
| 0.0 | 2.4 | 450 | 0.0000 | 3.4912 | -10.9759 | 1.0 | 14.4671 | -116.5096 | -36.4950 | -2.5011 | -2.5165 |
| 0.0 | 2.6667 | 500 | 0.0000 | 3.5172 | -11.0415 | 1.0 | 14.5587 | -116.7282 | -36.4083 | -2.5010 | -2.5165 |
| 0.0 | 2.9333 | 550 | 0.0000 | 3.5281 | -11.1219 | 1.0 | 14.6500 | -116.9964 | -36.3719 | -2.4999 | -2.5154 |
| 0.0 | 3.2 | 600 | 0.0000 | 3.5544 | -11.1376 | 1.0 | 14.6920 | -117.0486 | -36.2843 | -2.4985 | -2.5140 |
| 0.0 | 3.4667 | 650 | 0.0000 | 3.5412 | -11.1686 | 1.0 | 14.7098 | -117.1519 | -36.3284 | -2.4993 | -2.5149 |
| 0.0 | 3.7333 | 700 | 0.0000 | 3.5592 | -11.1405 | 1.0 | 14.6997 | -117.0585 | -36.2685 | -2.4988 | -2.5143 |
| 0.0 | 4.0 | 750 | 0.0000 | 3.5602 | -11.1575 | 1.0 | 14.7177 | -117.1151 | -36.2652 | -2.4993 | -2.5148 |
| 0.0 | 4.2667 | 800 | 0.0000 | 3.5534 | -11.1617 | 1.0 | 14.7151 | -117.1290 | -36.2877 | -2.4996 | -2.5152 |
| 0.0 | 4.5333 | 850 | 0.0000 | 3.5623 | -11.1612 | 1.0 | 14.7234 | -117.1272 | -36.2582 | -2.4994 | -2.5150 |
| 0.0 | 4.8 | 900 | 0.0000 | 3.5522 | -11.1817 | 1.0 | 14.7339 | -117.1956 | -36.2917 | -2.4993 | -2.5149 |
| 0.0 | 5.0667 | 950 | 0.0000 | 3.5522 | -11.1817 | 1.0 | 14.7339 | -117.1956 | -36.2917 | -2.4993 | -2.5149 |
| 0.0 | 5.3333 | 1000 | 0.0000 | 3.5522 | -11.1817 | 1.0 | 14.7339 | -117.1956 | -36.2917 | -2.4993 | -2.5149 |
Framework versions
- Transformers 4.44.2
- Pytorch 2.4.0+cu121
- Datasets 2.21.0
- Tokenizers 0.19.1