llama3-70B-lora-pretrain_v2

This model is a fine-tuned version of meta-llama/Meta-Llama-3-70B-Instruct on the sm_artile dataset. It achieves the following results on the evaluation set:

• Loss: 1.9382

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 2
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• total_eval_batch_size: 2
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 500
• num_epochs: 3.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
2.6995	0.0939	100	2.6305
2.4199	0.1877	200	2.3979
2.2722	0.2816	300	2.2180
2.0762	0.3754	400	2.1251
1.9652	0.4693	500	2.0858
2.1893	0.5631	600	2.0629
2.0153	0.6570	700	2.0473
1.9911	0.7508	800	2.0318
2.1041	0.8447	900	2.0198
2.0488	0.9385	1000	2.0117
1.897	1.0324	1100	2.0018
2.0298	1.1262	1200	1.9952
2.0989	1.2201	1300	1.9890
1.8695	1.3139	1400	1.9838
2.1573	1.4078	1500	1.9764
2.0183	1.5016	1600	1.9713
1.9229	1.5955	1700	1.9672
1.9732	1.6893	1800	1.9617
1.6835	1.7832	1900	1.9574
1.9874	1.8771	2000	1.9539
1.7607	1.9709	2100	1.9512
1.9459	2.0648	2200	1.9480
1.7611	2.1586	2300	1.9463
1.8491	2.2525	2400	1.9441
1.9121	2.3463	2500	1.9427
1.8849	2.4402	2600	1.9413
2.0679	2.5340	2700	1.9400
1.9908	2.6279	2800	1.9394
1.9557	2.7217	2900	1.9388
1.9627	2.8156	3000	1.9384
1.8339	2.9094	3100	1.9383

Framework versions

• PEFT 0.10.0
• Transformers 4.40.0
• Pytorch 2.2.1
• Datasets 2.18.0
• Tokenizers 0.19.1