dhanishetty's picture
dhanishetty/llama_adapters
d610464 verified
metadata
base_model: meta-llama/Meta-Llama-3-8B-Instruct
library_name: peft
license: llama3
tags:
  - trl
  - sft
  - generated_from_trainer
model-index:
  - name: llama-qLoRA
    results: []

Visualize in Weights & Biases

llama-qLoRA

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 0.9017

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-05
  • train_batch_size: 8
  • eval_batch_size: 8
  • seed: 42
  • gradient_accumulation_steps: 2
  • total_train_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: inverse_sqrt
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 3

Training results

Training Loss Epoch Step Validation Loss
1.1337 0.2022 100 1.1567
0.9957 0.4044 200 0.9863
0.9269 0.6067 300 0.9463
0.944 0.8089 400 0.9343
0.8779 1.0111 500 0.9271
0.8902 1.2133 600 0.9218
0.9149 1.4156 700 0.9156
0.8801 1.6178 800 0.9126
0.8752 1.8200 900 0.9093
0.8477 2.0222 1000 0.9068
0.8506 2.2245 1100 0.9067
0.8676 2.4267 1200 0.9048
0.8504 2.6289 1300 0.9028
0.8617 2.8311 1400 0.9017

Framework versions

  • PEFT 0.11.1
  • Transformers 4.42.4
  • Pytorch 2.3.0+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1