metadata

base_model: mistralai/Mistral-7B-Instruct-v0.3
datasets:
  - generator
library_name: peft
license: apache-2.0
tags:
  - trl
  - sft
  - generated_from_trainer
model-index:
  - name: mistral_7b_cosine_lr
    results: []

mistral_7b_cosine_lr

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.3 on the generator dataset. It achieves the following results on the evaluation set:

Loss: 0.3810

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0002
train_batch_size: 2
eval_batch_size: 8
seed: 42
gradient_accumulation_steps: 8
total_train_batch_size: 16
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.03
lr_scheduler_warmup_steps: 15
num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
0.8199	0.0366	10	0.6013
0.5795	0.0732	20	0.5157
0.5089	0.1098	30	0.4825
0.4742	0.1465	40	0.4628
0.468	0.1831	50	0.4502
0.4489	0.2197	60	0.4424
0.4471	0.2563	70	0.4378
0.4529	0.2929	80	0.4365
0.4382	0.3295	90	0.4289
0.4352	0.3661	100	0.4267
0.4331	0.4027	110	0.4214
0.4368	0.4394	120	0.4326
0.4244	0.4760	130	0.4188
0.4231	0.5126	140	0.4143
0.418	0.5492	150	0.4103
0.4118	0.5858	160	0.4066
0.4151	0.6224	170	0.4047
0.4093	0.6590	180	0.4020
0.4066	0.6957	190	0.4004
0.4072	0.7323	200	0.3970
0.4002	0.7689	210	0.3933
0.3968	0.8055	220	0.3919
0.3967	0.8421	230	0.3893
0.3901	0.8787	240	0.3870
0.3895	0.9153	250	0.3852
0.3929	0.9519	260	0.3823
0.3854	0.9886	270	0.3808
0.3524	1.0252	280	0.3833
0.332	1.0618	290	0.3821
0.3283	1.0984	300	0.3810

Framework versions

PEFT 0.13.2
Transformers 4.45.2
Pytorch 2.4.1+cu121
Datasets 3.0.1
Tokenizers 0.20.0