Orcawise/custom_mistral_7b_orcawise_sep_27

results

This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.4476

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 2
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 17
• training_steps: 1792

Training results

Training Loss	Epoch	Step	Validation Loss
2.8156	0.1115	50	1.7682
1.6754	0.2230	100	1.6347
1.5257	0.3344	150	1.5543
1.5435	0.4459	200	1.5301
1.613	0.5574	250	1.5579
1.6178	0.6689	300	1.6006
1.675	0.7804	350	1.5209
1.5046	0.8919	400	1.5203
1.5977	1.0033	450	1.5253
1.5303	1.1148	500	1.4984
1.4748	1.2263	550	1.5073
1.4955	1.3378	600	1.4998
1.5737	1.4493	650	1.5405
1.5662	1.5608	700	1.5147
1.417	1.6722	750	1.5047
1.5732	1.7837	800	1.4768
1.5077	1.8952	850	1.4948
1.5634	2.0067	900	1.4768
1.5219	2.1182	950	1.4752
1.4073	2.2297	1000	1.4776
1.4915	2.3411	1050	1.4799
1.4585	2.4526	1100	1.4868
1.4979	2.5641	1150	1.4703
1.4755	2.6756	1200	1.4617
1.4167	2.7871	1250	1.4576
1.5095	2.8986	1300	1.4651
1.477	3.0100	1350	1.4526
1.4495	3.1215	1400	1.4660
1.5609	3.2330	1450	1.4547
1.4319	3.3445	1500	1.4478
1.3469	3.4560	1550	1.4597
1.4454	3.5674	1600	1.4478
1.4071	3.6789	1650	1.4506
1.3686	3.7904	1700	1.4485
1.4958	3.9019	1750	1.4476

Framework versions

• PEFT 0.13.0
• Transformers 4.44.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.1
• Tokenizers 0.19.1