collapse_gemma-2-2b_hs2_accumulatesubsample_iter10_sftsd2

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.1964
• Num Input Tokens Seen: 4977584

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 2
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3909	0
1.2728	0.0533	5	1.2766	266944
1.0944	0.1067	10	1.2041	528504
1.061	0.16	15	1.1908	790952
0.9567	0.2133	20	1.2032	1063680
0.7259	0.2667	25	1.2107	1330032
0.7803	0.32	30	1.2163	1602016
0.7025	0.3733	35	1.2305	1868424
0.7138	0.4267	40	1.2162	2141336
0.6717	0.48	45	1.2232	2412000
0.5593	0.5333	50	1.2079	2679920
0.5536	0.5867	55	1.2070	2946904
0.562	0.64	60	1.2054	3215776
0.4965	0.6933	65	1.2006	3479152
0.5015	0.7467	70	1.2018	3746592
0.4981	0.8	75	1.1892	4015920
0.5343	0.8533	80	1.1997	4286352
0.4309	0.9067	85	1.2070	4550912
0.5186	0.96	90	1.1959	4816776

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1