collapse_gemma-2-9b_hs2_accumulate_iter1_sftsd1

This model is a fine-tuned version of google/gemma-2-9b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.9313
• Num Input Tokens Seen: 5254884

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 4
• eval_batch_size: 16
• seed: 1
• gradient_accumulation_steps: 32
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.2335	0
1.0811	0.0511	5	1.0631	260128
1.0247	0.1021	10	0.9817	527396
0.9713	0.1532	15	0.9695	803280
1.0094	0.2043	20	0.9637	1074404
0.9265	0.2553	25	0.9583	1348060
1.0149	0.3064	30	0.9544	1614960
0.9107	0.3575	35	0.9504	1884844
0.9349	0.4086	40	0.9473	2154208
0.9956	0.4596	45	0.9446	2424544
0.8864	0.5107	50	0.9431	2690292
0.9664	0.5618	55	0.9416	2962944
0.9601	0.6128	60	0.9398	3234692
0.9302	0.6639	65	0.9377	3510980
0.9355	0.7150	70	0.9365	3790388
0.9319	0.7660	75	0.9356	4069200
1.0081	0.8171	80	0.9351	4338748
0.9418	0.8682	85	0.9336	4606552
0.8993	0.9192	90	0.9321	4877900
0.9327	0.9703	95	0.9321	5147172

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1