Llama-3-8B-Instruct-Medical-QLoRA

This model is a adapter for meta-llama/Meta-Llama-3-8B-Instruct, finetuned on a subset of given datasets. It achieves the following results on the evaluation set:

• Loss: 1.1646

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
2.217	0.0591	20	1.5876
1.4821	0.1182	40	1.3649
1.3217	0.1773	60	1.2501
1.2392	0.2363	80	1.2201
1.1963	0.2954	100	1.2075
1.1829	0.3545	120	1.1997
1.2229	0.4136	140	1.1917
1.2016	0.4727	160	1.1868
1.1753	0.5318	180	1.1831
1.216	0.5908	200	1.1790
1.1831	0.6499	220	1.1761
1.1941	0.7090	240	1.1730
1.2566	0.7681	260	1.1702
1.1908	0.8272	280	1.1681
1.1586	0.8863	300	1.1665
1.1956	0.9453	320	1.1646

Framework versions

• PEFT 0.11.0
• Transformers 4.40.2
• Pytorch 2.2.1+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1