Llama3.1-8B-QA_CoT-LAW-Instruct-r64

This model is a fine-tuned version of meta-llama/Llama-3.1-8B on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.4632

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3.6e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• num_epochs: 1
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.5851	0.0089	200	0.6076
0.5957	0.0178	400	0.5905
0.6387	0.0267	600	0.5838
0.5579	0.0356	800	0.5772
0.5691	0.0444	1000	0.5704
0.5905	0.0533	1200	0.5665
0.5664	0.0622	1400	0.5619
0.5537	0.0711	1600	0.5583
0.5937	0.08	1800	0.5558
0.5942	0.0889	2000	0.5535
0.4662	0.0978	2200	0.5509
0.5291	0.1067	2400	0.5480
0.5892	0.1156	2600	0.5467
0.535	0.1244	2800	0.5433
0.5375	0.1333	3000	0.5417
0.5217	0.1422	3200	0.5401
0.5701	0.1511	3400	0.5364
0.5183	0.16	3600	0.5353
0.5149	0.1689	3800	0.5345
0.5166	0.1778	4000	0.5321
0.5322	0.1867	4200	0.5304
0.5882	0.1956	4400	0.5286
0.5571	0.2044	4600	0.5266
0.5265	0.2133	4800	0.5258
0.477	0.2222	5000	0.5224
0.5099	0.2311	5200	0.5220
0.5123	0.24	5400	0.5206
0.4467	0.2489	5600	0.5195
0.5461	0.2578	5800	0.5179
0.4798	0.2667	6000	0.5167
0.5436	0.2756	6200	0.5150
0.5304	0.2844	6400	0.5134
0.4854	0.2933	6600	0.5116
0.5618	0.3022	6800	0.5112
0.537	0.3111	7000	0.5099
0.5352	0.32	7200	0.5088
0.4811	0.3289	7400	0.5082
0.452	0.3378	7600	0.5065
0.5921	0.3467	7800	0.5049
0.5043	0.3556	8000	0.5046
0.5269	0.3644	8200	0.5028
0.5481	0.3733	8400	0.5013
0.5285	0.3822	8600	0.5009
0.5726	0.3911	8800	0.4998
0.5535	0.4	9000	0.4984
0.5185	0.4089	9200	0.4976
0.5432	0.4178	9400	0.4963
0.4337	0.4267	9600	0.4953
0.4643	0.4356	9800	0.4948
0.4715	0.4444	10000	0.4932
0.4945	0.4533	10200	0.4925
0.5382	0.4622	10400	0.4916
0.4542	0.4711	10600	0.4908
0.5242	0.48	10800	0.4900
0.5305	0.4889	11000	0.4883
0.5547	0.4978	11200	0.4877
0.4623	0.5067	11400	0.4870
0.501	0.5156	11600	0.4862
0.5399	0.5244	11800	0.4851
0.4667	0.5333	12000	0.4846
0.5373	0.5422	12200	0.4836
0.4633	0.5511	12400	0.4827
0.473	0.56	12600	0.4820
0.5364	0.5689	12800	0.4812
0.4721	0.5778	13000	0.4805
0.5895	0.5867	13200	0.4797
0.5211	0.5956	13400	0.4787
0.4942	0.6044	13600	0.4780
0.4934	0.6133	13800	0.4773
0.5101	0.6222	14000	0.4767
0.4187	0.6311	14200	0.4761
0.5371	0.64	14400	0.4752
0.5041	0.6489	14600	0.4745
0.4861	0.6578	14800	0.4740
0.4723	0.6667	15000	0.4733
0.4193	0.6756	15200	0.4727
0.5096	0.6844	15400	0.4720
0.4864	0.6933	15600	0.4715
0.479	0.7022	15800	0.4711
0.4329	0.7111	16000	0.4704
0.565	0.72	16200	0.4700
0.5131	0.7289	16400	0.4695
0.4722	0.7378	16600	0.4692
0.5044	0.7467	16800	0.4686
0.4701	0.7556	17000	0.4681
0.4476	0.7644	17200	0.4677
0.4116	0.7733	17400	0.4672
0.4345	0.7822	17600	0.4668
0.469	0.7911	17800	0.4665
0.5491	0.8	18000	0.4661
0.531	0.8089	18200	0.4658
0.4313	0.8178	18400	0.4655
0.5244	0.8267	18600	0.4652
0.4248	0.8356	18800	0.4649
0.5095	0.8444	19000	0.4647
0.4436	0.8533	19200	0.4645
0.5154	0.8622	19400	0.4643
0.5023	0.8711	19600	0.4641
0.5312	0.88	19800	0.4639
0.4934	0.8889	20000	0.4638
0.4962	0.8978	20200	0.4637
0.4964	0.9067	20400	0.4636
0.4966	0.9156	20600	0.4635
0.4303	0.9244	20800	0.4634
0.4531	0.9333	21000	0.4633
0.439	0.9422	21200	0.4633
0.4537	0.9511	21400	0.4633
0.4511	0.96	21600	0.4632
0.4713	0.9689	21800	0.4632
0.5265	0.9778	22000	0.4632
0.4636	0.9867	22200	0.4632
0.4287	0.9956	22400	0.4632

Framework versions

• PEFT 0.12.0
• Transformers 4.47.0
• Pytorch 2.5.1+cu124
• Datasets 3.0.0
• Tokenizers 0.21.0