collapse_gemma-2-2b_hs2_accumulatesubsample_iter18_sftsd2

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.2146
• Num Input Tokens Seen: 4982224

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 2
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3909	0
1.2659	0.0529	5	1.2807	264312
1.0915	0.1059	10	1.2298	531088
0.9617	0.1588	15	1.2146	797680
0.8165	0.2118	20	1.2558	1063768
0.6736	0.2647	25	1.2705	1333352
0.6868	0.3177	30	1.2647	1594824
0.5659	0.3706	35	1.2549	1858024
0.4964	0.4236	40	1.2572	2122648
0.4903	0.4765	45	1.2329	2390872
0.4303	0.5295	50	1.2455	2656160
0.5471	0.5824	55	1.2231	2921864
0.4603	0.6353	60	1.2356	3187064
0.4218	0.6883	65	1.2259	3449192
0.4204	0.7412	70	1.2386	3712440
0.4478	0.7942	75	1.2208	3978048
0.4628	0.8471	80	1.2195	4239360
0.3803	0.9001	85	1.2154	4498248
0.452	0.9530	90	1.2053	4767016

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1