collapse_gemma-2-2b_hs2_iter1_sftsd2

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.0646
• Num Input Tokens Seen: 5690264

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 2
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3956	0
1.2529	0.0511	5	1.2627	295904
1.1843	0.1021	10	1.1704	591104
1.1523	0.1532	15	1.1333	883280
1.0979	0.2042	20	1.1076	1177976
1.0923	0.2553	25	1.0980	1470072
1.07	0.3063	30	1.0926	1759320
1.1217	0.3574	35	1.0887	2048280
1.0978	0.4084	40	1.0847	2339776
1.0604	0.4595	45	1.0816	2632712
1.0608	0.5105	50	1.0787	2926200
1.1238	0.5616	55	1.0767	3220536
1.0663	0.6126	60	1.0750	3515696
1.0059	0.6637	65	1.0730	3804824
1.0991	0.7147	70	1.0714	4101032
1.1119	0.7658	75	1.0698	4391096
1.0905	0.8168	80	1.0688	4688752
1.0574	0.8679	85	1.0676	4981792
1.1394	0.9190	90	1.0661	5276840
1.1296	0.9700	95	1.0651	5572144

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1