TinyLlama-1.1B-Chat-SchemaLinking-v1

This model is a fine-tuned version of TinyLlama/TinyLlama-1.1B-Chat-v1.0 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.0697

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 1
• eval_batch_size: 1
• seed: 14
• gradient_accumulation_steps: 8
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.03
• num_epochs: 5
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.3573	0.2334	250	0.1872
0.1155	0.4668	500	0.1122
0.079	0.7002	750	0.0828
0.0672	0.9336	1000	0.0741
0.0555	1.1670	1250	0.0720
0.055	1.4004	1500	0.0701
0.053	1.6338	1750	0.0681
0.0526	1.8672	2000	0.0664
0.047	2.1006	2250	0.0647
0.0406	2.3340	2500	0.0646
0.0411	2.5674	2750	0.0620
0.0352	2.8010	3000	0.0660
0.0337	3.0344	3250	0.0646
0.0326	3.2678	3500	0.0633
0.0323	3.5012	3750	0.0625
0.0321	3.7346	4000	0.0614
0.0317	3.9680	4250	0.0616
0.0271	4.2014	4500	0.0676
0.0258	4.4348	4750	0.0688
0.0257	4.6682	5000	0.0678
0.0227	4.9019	5250	0.0697

Framework versions

• PEFT 0.13.2
• Transformers 4.45.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.1
• Tokenizers 0.20.1