--- base_model: meta-llama/Meta-Llama-3-8B library_name: peft license: llama3 tags: - trl - sft - generated_from_trainer model-index: - name: llama-qLoRA results: [] --- <!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. --> [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/dhanishetty-personaluse/huggingface/runs/cdhku7u3) # llama-qLoRA This model is a fine-tuned version of [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 1.9713 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 1e-06 - train_batch_size: 16 - eval_batch_size: 16 - seed: 42 - gradient_accumulation_steps: 2 - total_train_batch_size: 32 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: inverse_sqrt - lr_scheduler_warmup_ratio: 0.05 - num_epochs: 3 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:------:|:----:|:---------------:| | 1.9835 | 0.3350 | 500 | 2.0047 | | 1.97 | 0.6700 | 1000 | 1.9901 | | 1.9764 | 1.0050 | 1500 | 1.9838 | | 1.9714 | 1.3400 | 2000 | 1.9799 | | 1.9547 | 1.6750 | 2500 | 1.9770 | | 1.981 | 2.0101 | 3000 | 1.9747 | | 1.9841 | 2.3451 | 3500 | 1.9729 | | 1.9469 | 2.6801 | 4000 | 1.9713 | ### Framework versions - PEFT 0.12.0 - Transformers 4.42.4 - Pytorch 2.3.1+cu121 - Datasets 2.20.0 - Tokenizers 0.19.1