--- library_name: transformers license: apache-2.0 base_model: tsavage68/Na_M2_1000steps_1e7_SFT tags: - trl - dpo - generated_from_trainer model-index: - name: Na_M2_1000steps_1e8rate_01beta_cSFTDPO results: [] --- # Na_M2_1000steps_1e8rate_01beta_cSFTDPO This model is a fine-tuned version of [tsavage68/Na_M2_1000steps_1e7_SFT](https://huggingface.co/tsavage68/Na_M2_1000steps_1e7_SFT) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 0.6023 - Rewards/chosen: 0.0529 - Rewards/rejected: -0.1392 - Rewards/accuracies: 1.0 - Rewards/margins: 0.1921 - Logps/rejected: -81.3154 - Logps/chosen: -47.6033 - Logits/rejected: -2.5345 - Logits/chosen: -2.5471 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 1e-08 - train_batch_size: 2 - eval_batch_size: 1 - seed: 42 - gradient_accumulation_steps: 2 - total_train_batch_size: 4 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 100 - training_steps: 1000 ### Training results | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen | |:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:| | 0.6929 | 0.2667 | 50 | 0.6931 | -0.0010 | -0.0014 | 0.5700 | 0.0003 | -79.9371 | -48.1427 | -2.5355 | -2.5481 | | 0.6881 | 0.5333 | 100 | 0.6832 | 0.0062 | -0.0142 | 0.6900 | 0.0204 | -80.0656 | -48.0704 | -2.5357 | -2.5482 | | 0.6652 | 0.8 | 150 | 0.6568 | 0.0223 | -0.0526 | 0.9500 | 0.0748 | -80.4490 | -47.9098 | -2.5356 | -2.5482 | | 0.6475 | 1.0667 | 200 | 0.6389 | 0.0327 | -0.0794 | 1.0 | 0.1121 | -80.7177 | -47.8054 | -2.5355 | -2.5481 | | 0.6224 | 1.3333 | 250 | 0.6217 | 0.0389 | -0.1104 | 1.0 | 0.1492 | -81.0270 | -47.7436 | -2.5352 | -2.5477 | | 0.6068 | 1.6 | 300 | 0.6115 | 0.0553 | -0.1167 | 1.0 | 0.1720 | -81.0905 | -47.5798 | -2.5353 | -2.5478 | | 0.6018 | 1.8667 | 350 | 0.6041 | 0.0523 | -0.1359 | 1.0 | 0.1882 | -81.2823 | -47.6092 | -2.5345 | -2.5471 | | 0.5976 | 2.1333 | 400 | 0.6021 | 0.0543 | -0.1384 | 1.0 | 0.1927 | -81.3072 | -47.5892 | -2.5349 | -2.5474 | | 0.5952 | 2.4 | 450 | 0.5993 | 0.0581 | -0.1408 | 1.0 | 0.1990 | -81.3318 | -47.5512 | -2.5343 | -2.5468 | | 0.6013 | 2.6667 | 500 | 0.6022 | 0.0541 | -0.1384 | 1.0 | 0.1925 | -81.3071 | -47.5913 | -2.5347 | -2.5472 | | 0.5981 | 2.9333 | 550 | 0.6027 | 0.0571 | -0.1340 | 1.0 | 0.1911 | -81.2633 | -47.5610 | -2.5348 | -2.5473 | | 0.6006 | 3.2 | 600 | 0.6009 | 0.0589 | -0.1365 | 1.0 | 0.1954 | -81.2883 | -47.5433 | -2.5347 | -2.5473 | | 0.5961 | 3.4667 | 650 | 0.6036 | 0.0539 | -0.1354 | 1.0 | 0.1893 | -81.2771 | -47.5931 | -2.5350 | -2.5476 | | 0.5896 | 3.7333 | 700 | 0.6024 | 0.0550 | -0.1368 | 1.0 | 0.1918 | -81.2913 | -47.5819 | -2.5345 | -2.5471 | | 0.593 | 4.0 | 750 | 0.6023 | 0.0529 | -0.1392 | 1.0 | 0.1921 | -81.3154 | -47.6033 | -2.5345 | -2.5471 | | 0.603 | 4.2667 | 800 | 0.6023 | 0.0529 | -0.1392 | 1.0 | 0.1921 | -81.3154 | -47.6033 | -2.5345 | -2.5471 | | 0.5989 | 4.5333 | 850 | 0.6023 | 0.0529 | -0.1392 | 1.0 | 0.1921 | -81.3154 | -47.6033 | -2.5345 | -2.5471 | | 0.5879 | 4.8 | 900 | 0.6023 | 0.0529 | -0.1392 | 1.0 | 0.1921 | -81.3154 | -47.6033 | -2.5345 | -2.5471 | | 0.5949 | 5.0667 | 950 | 0.6023 | 0.0529 | -0.1392 | 1.0 | 0.1921 | -81.3154 | -47.6033 | -2.5345 | -2.5471 | | 0.5974 | 5.3333 | 1000 | 0.6023 | 0.0529 | -0.1392 | 1.0 | 0.1921 | -81.3154 | -47.6033 | -2.5345 | -2.5471 | ### Framework versions - Transformers 4.44.2 - Pytorch 2.4.0+cu121 - Datasets 2.21.0 - Tokenizers 0.19.1