hf_phi3_lora

This model is a fine-tuned version of microsoft/Phi-3-mini-4k-instruct on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.3171

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 32
• total_train_batch_size: 512
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 1000
• num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss
1.4828	0.1489	500	1.4306
1.4047	0.2978	1000	1.3980
1.3611	0.4468	1500	1.3835
1.3653	0.5957	2000	1.3709
1.3171	0.7446	2500	1.3665
1.3089	0.8935	3000	1.3626
1.312	1.0425	3500	1.3608
1.2771	1.1914	4000	1.3556
1.3031	1.3403	4500	1.3570
1.284	1.4892	5000	1.3508
1.2697	1.6382	5500	1.3477
1.2594	1.7871	6000	1.3453
1.254	1.9360	6500	1.3413
1.2652	2.0849	7000	1.3426
1.2529	2.2338	7500	1.3435
1.2544	2.3828	8000	1.3382
1.2511	2.5317	8500	1.3396
1.2548	2.6806	9000	1.3361
1.2483	2.8295	9500	1.3351
1.2442	2.9785	10000	1.3382
1.2426	3.1274	10500	1.3344
1.2265	3.2763	11000	1.3361
1.2255	3.4252	11500	1.3356
1.2269	3.5742	12000	1.3314
1.2396	3.7231	12500	1.3298
1.2303	3.8720	13000	1.3260
1.2254	4.0209	13500	1.3277
1.2277	4.1698	14000	1.3272
1.2295	4.3188	14500	1.3240
1.2375	4.4677	15000	1.3288
1.2038	4.6166	15500	1.3224
1.2322	4.7655	16000	1.3214
1.2015	4.9145	16500	1.3246
1.208	5.0634	17000	1.3216
1.2248	5.2123	17500	1.3193
1.2155	5.3612	18000	1.3249
1.2194	5.5102	18500	1.3183
1.2185	5.6591	19000	1.3196
1.2119	5.8080	19500	1.3142
1.2171	5.9569	20000	1.3240
1.21	6.1058	20500	1.3235
1.19	6.2548	21000	1.3171

Framework versions

• PEFT 0.9.0
• Transformers 4.43.0.dev0
• Pytorch 2.3.0+cu121
• Datasets 2.18.0
• Tokenizers 0.19.1