TinyLlama-1.1B-Chat-v0.6-BALLUFF

This model is a fine-tuned version of TinyLlama/TinyLlama-1.1B-Chat-v0.6 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.2857

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.03
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss
0.4974	1.0	1334	0.4325
0.4279	2.0	2668	0.4047
0.304	3.0	4002	0.3933
0.2024	4.0	5336	0.3896
0.298	5.0	6670	0.3859
0.4684	6.0	8004	0.4056
0.3579	7.0	9338	0.3844
0.3633	8.0	10672	0.3963
0.8385	9.0	12006	0.3809
0.4261	10.0	13340	0.3974
0.6172	11.0	14674	0.3797
0.4697	12.0	16008	0.3747
0.3488	13.0	17342	0.3687
0.5322	14.0	18676	0.3583
0.3615	15.0	20010	0.3571
0.4629	16.0	21344	0.3501
0.2742	17.0	22678	0.3425
0.2556	18.0	24012	0.3340
0.1484	19.0	25346	0.3287
0.3551	20.0	26680	0.3227
0.2442	21.0	28014	0.3129
0.2195	22.0	29348	0.3087
0.1966	23.0	30682	0.3024
0.5957	24.0	32016	0.2962
0.2548	25.0	33350	0.2926
0.2288	26.0	34684	0.2896
0.1871	27.0	36018	0.2872
0.4647	28.0	37352	0.2862
0.1915	29.0	38686	0.2858
0.4057	30.0	40020	0.2857

Framework versions

• PEFT 0.14.0
• Transformers 4.47.1
• Pytorch 2.5.1+cu121
• Datasets 3.2.0
• Tokenizers 0.21.0