|
--- |
|
license: apache-2.0 |
|
library_name: peft |
|
tags: |
|
- trl |
|
- dpo |
|
- generated_from_trainer |
|
base_model: TheBloke/Mistral-7B-v0.1-GPTQ |
|
model-index: |
|
- name: mistral-dpo |
|
results: [] |
|
--- |
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
# mistral-dpo |
|
|
|
This model is a fine-tuned version of [TheBloke/Mistral-7B-v0.1-GPTQ](https://huggingface.co/TheBloke/Mistral-7B-v0.1-GPTQ) on the None dataset. |
|
It achieves the following results on the evaluation set: |
|
- Loss: 0.5603 |
|
- Rewards/chosen: -12.5467 |
|
- Rewards/rejected: -28.4037 |
|
- Rewards/accuracies: 0.8571 |
|
- Rewards/margins: 15.8571 |
|
- Logps/rejected: -411.7001 |
|
- Logps/chosen: -215.4742 |
|
- Logits/rejected: -0.7509 |
|
- Logits/chosen: -0.7707 |
|
|
|
## Model description |
|
|
|
More information needed |
|
|
|
## Intended uses & limitations |
|
|
|
More information needed |
|
|
|
## Training and evaluation data |
|
|
|
More information needed |
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 0.0002 |
|
- train_batch_size: 1 |
|
- eval_batch_size: 8 |
|
- seed: 42 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- lr_scheduler_warmup_steps: 2 |
|
- training_steps: 250 |
|
- mixed_precision_training: Native AMP |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen | |
|
|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:| |
|
| 0.6785 | 0.02 | 10 | 0.6291 | -0.0030 | -0.1321 | 0.875 | 0.1291 | -128.9836 | -90.0372 | -2.3988 | -2.3489 | |
|
| 0.5661 | 0.04 | 20 | 0.4421 | 0.0008 | -0.6608 | 0.875 | 0.6616 | -134.2708 | -89.9997 | -2.3613 | -2.3042 | |
|
| 0.3257 | 0.06 | 30 | 0.3584 | -0.7139 | -2.3035 | 0.8393 | 1.5897 | -150.6985 | -97.1463 | -2.2995 | -2.2546 | |
|
| 0.3563 | 0.08 | 40 | 0.5522 | -3.0636 | -6.7067 | 0.8214 | 3.6431 | -194.7305 | -120.6441 | -2.1396 | -2.0849 | |
|
| 0.0013 | 0.1 | 50 | 1.3365 | -8.4317 | -16.1649 | 0.8036 | 7.7332 | -289.3120 | -174.3246 | -1.8243 | -1.7710 | |
|
| 0.0277 | 0.12 | 60 | 2.4224 | -14.8512 | -25.9570 | 0.8214 | 11.1059 | -387.2331 | -238.5192 | -1.5464 | -1.4950 | |
|
| 1.5742 | 0.14 | 70 | 3.1075 | -17.8751 | -29.6755 | 0.8214 | 11.8004 | -424.4176 | -268.7585 | -1.4071 | -1.3681 | |
|
| 14.1036 | 0.16 | 80 | 3.6238 | -20.4205 | -32.7881 | 0.8214 | 12.3675 | -455.5435 | -294.2129 | -1.3237 | -1.2729 | |
|
| 8.531 | 0.18 | 90 | 3.7123 | -21.7863 | -36.0729 | 0.8214 | 14.2866 | -488.3922 | -307.8707 | -1.2975 | -1.2388 | |
|
| 4.6429 | 0.2 | 100 | 2.0394 | -16.6472 | -29.8508 | 0.8393 | 13.2036 | -426.1712 | -256.4797 | -1.3189 | -1.2784 | |
|
| 0.0565 | 0.22 | 110 | 1.6331 | -14.8501 | -27.2015 | 0.8393 | 12.3514 | -399.6779 | -238.5090 | -1.2425 | -1.2118 | |
|
| 0.0056 | 0.24 | 120 | 1.4774 | -15.0784 | -28.0012 | 0.8214 | 12.9228 | -407.6750 | -240.7916 | -1.0819 | -1.0579 | |
|
| 0.0804 | 0.26 | 130 | 1.5398 | -16.7630 | -30.6346 | 0.8393 | 13.8716 | -434.0091 | -257.6378 | -1.0054 | -0.9846 | |
|
| 0.0001 | 0.28 | 140 | 1.5159 | -17.9940 | -33.3459 | 0.8393 | 15.3520 | -461.1225 | -269.9474 | -0.8887 | -0.8844 | |
|
| 0.0 | 0.3 | 150 | 1.5062 | -18.4614 | -34.3481 | 0.8393 | 15.8868 | -471.1445 | -274.6213 | -0.8496 | -0.8503 | |
|
| 0.0 | 0.32 | 160 | 1.5035 | -18.6474 | -34.7165 | 0.8393 | 16.0692 | -474.8286 | -276.4815 | -0.8343 | -0.8367 | |
|
| 4.2123 | 0.34 | 170 | 1.2949 | -17.3471 | -32.6721 | 0.8571 | 15.3250 | -454.3839 | -263.4789 | -0.8672 | -0.8661 | |
|
| 2.13 | 0.36 | 180 | 0.9892 | -15.2178 | -30.1177 | 0.8571 | 14.8999 | -428.8398 | -242.1859 | -0.8992 | -0.9047 | |
|
| 2.0146 | 0.38 | 190 | 0.8365 | -13.9461 | -28.5983 | 0.8571 | 14.6522 | -413.6459 | -229.4683 | -0.9104 | -0.9224 | |
|
| 0.0706 | 0.4 | 200 | 0.7897 | -14.5198 | -29.8989 | 0.8571 | 15.3791 | -426.6525 | -235.2058 | -0.8064 | -0.8224 | |
|
| 5.2517 | 0.42 | 210 | 0.6621 | -13.7049 | -29.2354 | 0.8571 | 15.5305 | -420.0170 | -227.0569 | -0.7981 | -0.8124 | |
|
| 0.0499 | 0.44 | 220 | 0.5752 | -12.8733 | -28.5299 | 0.8571 | 15.6566 | -412.9616 | -218.7403 | -0.7801 | -0.7990 | |
|
| 0.5779 | 0.46 | 230 | 0.5611 | -12.6633 | -28.3836 | 0.8571 | 15.7203 | -411.4988 | -216.6405 | -0.7789 | -0.7975 | |
|
| 0.0322 | 0.48 | 240 | 0.5624 | -12.6348 | -28.4766 | 0.8571 | 15.8418 | -412.4289 | -216.3556 | -0.7696 | -0.7878 | |
|
| 0.1347 | 0.5 | 250 | 0.5603 | -12.5467 | -28.4037 | 0.8571 | 15.8571 | -411.7001 | -215.4742 | -0.7509 | -0.7707 | |
|
|
|
|
|
### Framework versions |
|
|
|
- PEFT 0.7.1 |
|
- Transformers 4.36.2 |
|
- Pytorch 2.0.1+cu118 |
|
- Datasets 2.15.0 |
|
- Tokenizers 0.15.0 |
