Hugging Face's logo

GaetanMichelet
/
Llama-31-8B_task-1_60-samples_config-3

PEFT
TensorBoard
Safetensors
llama
alignment-handbook
trl
sft
Generated from Trainer
4-bit precision
bitsandbytes
Model card Files Metrics Community
Llama-31-8B_task-1_60-samples_config-3 / README.md
GaetanMichelet's picture
GaetanMichelet
End of training
94c5c0a verified
preview code
|
raw
history blame contribute delete
No virus
3.08 kB
metadata
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
datasets:
  - GaetanMichelet/chat-60_ft_task-1
library_name: peft
license: llama3.1
tags:
  - alignment-handbook
  - trl
  - sft
  - generated_from_trainer
model-index:
  - name: Llama-31-8B_task-1_60-samples_config-3
    results: []

Llama-31-8B_task-1_60-samples_config-3

This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on the GaetanMichelet/chat-60_ft_task-1 dataset. It achieves the following results on the evaluation set:

  • Loss: 1.2924

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 1e-05
  • train_batch_size: 1
  • eval_batch_size: 1
  • seed: 42
  • distributed_type: multi-GPU
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 150

Training results

Training Loss Epoch Step Validation Loss
2.1691 0.8696 5 2.0741
2.0145 1.9130 11 2.0674
2.0918 2.9565 17 2.0492
2.0838 4.0 23 2.0192
2.0314 4.8696 28 1.9792
1.9775 5.9130 34 1.9190
1.8873 6.9565 40 1.8339
1.7547 8.0 46 1.7314
1.6653 8.8696 51 1.6435
1.5709 9.9130 57 1.5691
1.533 10.9565 63 1.5254
1.4035 12.0 69 1.4860
1.4227 12.8696 74 1.4573
1.4167 13.9130 80 1.4216
1.3733 14.9565 86 1.3884
1.2917 16.0 92 1.3621
1.2393 16.8696 97 1.3432
1.1512 17.9130 103 1.3246
1.1361 18.9565 109 1.3081
1.089 20.0 115 1.2985
1.0272 20.8696 120 1.2924
1.0591 21.9130 126 1.2934
0.9601 22.9565 132 1.3023
0.9245 24.0 138 1.3152
0.8188 24.8696 143 1.3258
0.8866 25.9130 149 1.3491
0.7508 26.9565 155 1.3779
0.7961 28.0 161 1.4176

Framework versions

  • PEFT 0.12.0
  • Transformers 4.44.0
  • Pytorch 2.1.2+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1