collapse_gemma-2-2b_hs2_replace_iter8_sftsd1

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.6101
• Num Input Tokens Seen: 4772632

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 1
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3909	0
1.4409	0.0511	5	1.2767	244408
0.9625	0.1022	10	1.2578	488208
0.5858	0.1533	15	1.4638	734240
0.3514	0.2043	20	1.6624	983736
0.1521	0.2554	25	1.8698	1226824
0.1048	0.3065	30	2.0906	1470240
0.0664	0.3576	35	2.3094	1716144
0.0345	0.4087	40	2.4302	1965688
0.0256	0.4598	45	2.5061	2212032
0.0255	0.5109	50	2.5796	2462992
0.0268	0.5619	55	2.5997	2709392
0.0233	0.6130	60	2.6136	2953608
0.0252	0.6641	65	2.5812	3193224
0.0234	0.7152	70	2.5971	3442080
0.0237	0.7663	75	2.6044	3689840
0.0231	0.8174	80	2.6007	3934368
0.0228	0.8685	85	2.6068	4180664
0.0225	0.9195	90	2.6026	4431504
0.0231	0.9706	95	2.6100	4673432

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1