metadata
license: apache-2.0
library_name: peft
tags:
- trl
- dpo
- generated_from_trainer
base_model: TheBloke/Mistral-7B-v0.1-GPTQ
model-index:
- name: mistral-dpo
results: []
mistral-dpo
This model is a fine-tuned version of TheBloke/Mistral-7B-v0.1-GPTQ on the None dataset. It achieves the following results on the evaluation set:
- Loss: 0.5603
- Rewards/chosen: -12.5467
- Rewards/rejected: -28.4037
- Rewards/accuracies: 0.8571
- Rewards/margins: 15.8571
- Logps/rejected: -411.7001
- Logps/chosen: -215.4742
- Logits/rejected: -0.7509
- Logits/chosen: -0.7707
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 1
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2
- training_steps: 250
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.6785 | 0.02 | 10 | 0.6291 | -0.0030 | -0.1321 | 0.875 | 0.1291 | -128.9836 | -90.0372 | -2.3988 | -2.3489 |
| 0.5661 | 0.04 | 20 | 0.4421 | 0.0008 | -0.6608 | 0.875 | 0.6616 | -134.2708 | -89.9997 | -2.3613 | -2.3042 |
| 0.3257 | 0.06 | 30 | 0.3584 | -0.7139 | -2.3035 | 0.8393 | 1.5897 | -150.6985 | -97.1463 | -2.2995 | -2.2546 |
| 0.3563 | 0.08 | 40 | 0.5522 | -3.0636 | -6.7067 | 0.8214 | 3.6431 | -194.7305 | -120.6441 | -2.1396 | -2.0849 |
| 0.0013 | 0.1 | 50 | 1.3365 | -8.4317 | -16.1649 | 0.8036 | 7.7332 | -289.3120 | -174.3246 | -1.8243 | -1.7710 |
| 0.0277 | 0.12 | 60 | 2.4224 | -14.8512 | -25.9570 | 0.8214 | 11.1059 | -387.2331 | -238.5192 | -1.5464 | -1.4950 |
| 1.5742 | 0.14 | 70 | 3.1075 | -17.8751 | -29.6755 | 0.8214 | 11.8004 | -424.4176 | -268.7585 | -1.4071 | -1.3681 |
| 14.1036 | 0.16 | 80 | 3.6238 | -20.4205 | -32.7881 | 0.8214 | 12.3675 | -455.5435 | -294.2129 | -1.3237 | -1.2729 |
| 8.531 | 0.18 | 90 | 3.7123 | -21.7863 | -36.0729 | 0.8214 | 14.2866 | -488.3922 | -307.8707 | -1.2975 | -1.2388 |
| 4.6429 | 0.2 | 100 | 2.0394 | -16.6472 | -29.8508 | 0.8393 | 13.2036 | -426.1712 | -256.4797 | -1.3189 | -1.2784 |
| 0.0565 | 0.22 | 110 | 1.6331 | -14.8501 | -27.2015 | 0.8393 | 12.3514 | -399.6779 | -238.5090 | -1.2425 | -1.2118 |
| 0.0056 | 0.24 | 120 | 1.4774 | -15.0784 | -28.0012 | 0.8214 | 12.9228 | -407.6750 | -240.7916 | -1.0819 | -1.0579 |
| 0.0804 | 0.26 | 130 | 1.5398 | -16.7630 | -30.6346 | 0.8393 | 13.8716 | -434.0091 | -257.6378 | -1.0054 | -0.9846 |
| 0.0001 | 0.28 | 140 | 1.5159 | -17.9940 | -33.3459 | 0.8393 | 15.3520 | -461.1225 | -269.9474 | -0.8887 | -0.8844 |
| 0.0 | 0.3 | 150 | 1.5062 | -18.4614 | -34.3481 | 0.8393 | 15.8868 | -471.1445 | -274.6213 | -0.8496 | -0.8503 |
| 0.0 | 0.32 | 160 | 1.5035 | -18.6474 | -34.7165 | 0.8393 | 16.0692 | -474.8286 | -276.4815 | -0.8343 | -0.8367 |
| 4.2123 | 0.34 | 170 | 1.2949 | -17.3471 | -32.6721 | 0.8571 | 15.3250 | -454.3839 | -263.4789 | -0.8672 | -0.8661 |
| 2.13 | 0.36 | 180 | 0.9892 | -15.2178 | -30.1177 | 0.8571 | 14.8999 | -428.8398 | -242.1859 | -0.8992 | -0.9047 |
| 2.0146 | 0.38 | 190 | 0.8365 | -13.9461 | -28.5983 | 0.8571 | 14.6522 | -413.6459 | -229.4683 | -0.9104 | -0.9224 |
| 0.0706 | 0.4 | 200 | 0.7897 | -14.5198 | -29.8989 | 0.8571 | 15.3791 | -426.6525 | -235.2058 | -0.8064 | -0.8224 |
| 5.2517 | 0.42 | 210 | 0.6621 | -13.7049 | -29.2354 | 0.8571 | 15.5305 | -420.0170 | -227.0569 | -0.7981 | -0.8124 |
| 0.0499 | 0.44 | 220 | 0.5752 | -12.8733 | -28.5299 | 0.8571 | 15.6566 | -412.9616 | -218.7403 | -0.7801 | -0.7990 |
| 0.5779 | 0.46 | 230 | 0.5611 | -12.6633 | -28.3836 | 0.8571 | 15.7203 | -411.4988 | -216.6405 | -0.7789 | -0.7975 |
| 0.0322 | 0.48 | 240 | 0.5624 | -12.6348 | -28.4766 | 0.8571 | 15.8418 | -412.4289 | -216.3556 | -0.7696 | -0.7878 |
| 0.1347 | 0.5 | 250 | 0.5603 | -12.5467 | -28.4037 | 0.8571 | 15.8571 | -411.7001 | -215.4742 | -0.7509 | -0.7707 |
Framework versions
- PEFT 0.7.1
- Transformers 4.36.2
- Pytorch 2.0.1+cu118
- Datasets 2.15.0
- Tokenizers 0.15.0