---
license: apache-2.0
library_name: peft
tags:
- alignment-handbook
- trl
- dpo
- generated_from_trainer
base_model: alignment-handbook/zephyr-7b-sft-full
datasets:
- generation/GPT4
model-index:
- name: zephyr-dpop-qlora-gpt4-5e-7-epoch3
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# zephyr-dpop-qlora-gpt4-5e-7-epoch3

This model is a fine-tuned version of [alignment-handbook/zephyr-7b-sft-full](https://huggingface.co/alignment-handbook/zephyr-7b-sft-full) on the generation/GPT4 dataset.
It achieves the following results on the evaluation set:
- Loss: 1.5678
- Positive Losses: 8.8531
- Dpo Losses: 0.6658
- Rewards/chosen: -0.0421
- Rewards/rejected: -0.1106
- Rewards/accuracies: 0.6151
- Rewards/margins: 0.0684
- Rewards/margins Max: 0.3178
- Rewards/margins Min: -0.1570
- Rewards/margins Std: 0.2114
- Logps/rejected: -270.2393
- Logps/chosen: -289.4331
- Logits/rejected: -2.6601
- Logits/chosen: -2.7033

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-07
- train_batch_size: 2
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- total_train_batch_size: 16
- total_eval_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss | Positive Losses | Dpo Losses | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Rewards/margins Max | Rewards/margins Min | Rewards/margins Std | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|:-------------:|:-----:|:----:|:---------------:|:---------------:|:----------:|:--------------:|:----------------:|:------------------:|:---------------:|:-------------------:|:-------------------:|:-------------------:|:--------------:|:------------:|:---------------:|:-------------:|
| 0.6835        | 0.28  | 100  | 0.6965          | 0.0436          | 0.6917     | 0.0092         | 0.0061           | 0.5833             | 0.0030          | 0.0155              | -0.0076             | 0.0103              | -258.5689      | -284.3059    | -2.8089         | -2.8541       |
| 0.6367        | 0.56  | 200  | 0.7633          | 0.6990          | 0.6863     | 0.0215         | 0.0070           | 0.5873             | 0.0145          | 0.0761              | -0.0391             | 0.0511              | -258.4836      | -283.0695    | -2.7779         | -2.8224       |
| 0.5913        | 0.85  | 300  | 0.9198          | 2.2041          | 0.6810     | 0.0123         | -0.0144          | 0.5714             | 0.0267          | 0.1358              | -0.0683             | 0.0899              | -260.6202      | -283.9922    | -2.7412         | -2.7853       |
| 0.5502        | 1.13  | 400  | 1.0826          | 3.7846          | 0.6770     | 0.0010         | -0.0361          | 0.5754             | 0.0370          | 0.1861              | -0.0963             | 0.1243              | -262.7899      | -285.1261    | -2.7113         | -2.7545       |
| 0.5398        | 1.41  | 500  | 1.1571          | 4.6567          | 0.6734     | 0.0027         | -0.0441          | 0.5833             | 0.0468          | 0.2338              | -0.1166             | 0.1549              | -263.5918      | -284.9548    | -2.6935         | -2.7368       |
| 0.5293        | 1.69  | 600  | 1.2245          | 5.3740          | 0.6703     | 0.0016         | -0.0536          | 0.5913             | 0.0552          | 0.2655              | -0.1284             | 0.1752              | -264.5410      | -285.0616    | -2.6767         | -2.7201       |
| 0.5238        | 1.97  | 700  | 1.3783          | 6.9387          | 0.6683     | -0.0190        | -0.0800          | 0.6032             | 0.0610          | 0.2891              | -0.1425             | 0.1922              | -267.1869      | -287.1237    | -2.6726         | -2.7154       |
| 0.488         | 2.25  | 800  | 1.4896          | 8.0964          | 0.6670     | -0.0328        | -0.0978          | 0.6111             | 0.0650          | 0.3063              | -0.1511             | 0.2037              | -268.9666      | -288.5044    | -2.6644         | -2.7076       |
| 0.5027        | 2.54  | 900  | 1.5575          | 8.7828          | 0.6661     | -0.0416        | -0.1091          | 0.6190             | 0.0675          | 0.3151              | -0.1563             | 0.2099              | -270.0926      | -289.3809    | -2.6629         | -2.7059       |
| 0.4962        | 2.82  | 1000 | 1.5707          | 8.9081          | 0.6660     | -0.0431        | -0.1110          | 0.6151             | 0.0679          | 0.3167              | -0.1568             | 0.2111              | -270.2825      | -289.5273    | -2.6606         | -2.7037       |


### Framework versions

- PEFT 0.7.1
- Transformers 4.39.0.dev0
- Pytorch 2.1.2+cu121
- Datasets 2.14.6
- Tokenizers 0.15.2