collapse_gemma-2-2b_hs2_replace_iter9_sftsd2

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.5326
• Num Input Tokens Seen: 4710648

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 2
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3909	0
1.4937	0.0511	5	1.2783	234544
0.9069	0.1022	10	1.3004	476384
0.4737	0.1533	15	1.5336	710232
0.247	0.2043	20	1.7745	954032
0.1058	0.2554	25	1.9661	1202912
0.0603	0.3065	30	2.2302	1448016
0.0357	0.3576	35	2.4185	1689736
0.0277	0.4087	40	2.5194	1931560
0.0235	0.4598	45	2.5744	2178184
0.0226	0.5109	50	2.6315	2420064
0.0261	0.5619	55	2.6343	2661248
0.0216	0.6130	60	2.6350	2903808
0.0229	0.6641	65	2.5899	3141584
0.0207	0.7152	70	2.5668	3382872
0.0217	0.7663	75	2.5357	3625368
0.0205	0.8174	80	2.5397	3874352
0.0209	0.8685	85	2.5497	4121944
0.0211	0.9195	90	2.5299	4365216
0.0206	0.9706	95	2.5237	4613264

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1