collapse_gemma-2-2b_hs2_replace_iter10_sftsd2

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.5147
• Num Input Tokens Seen: 4732232

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 2
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3909	0
1.4619	0.0511	5	1.2791	240912
0.8949	0.1022	10	1.3036	481944
0.478	0.1533	15	1.5412	722768
0.2271	0.2043	20	1.7643	969064
0.1021	0.2554	25	1.9727	1214488
0.0647	0.3065	30	2.2166	1459000
0.0316	0.3576	35	2.3927	1700400
0.0246	0.4087	40	2.4791	1945048
0.0259	0.4598	45	2.4862	2193112
0.0236	0.5109	50	2.4822	2435456
0.0205	0.5619	55	2.4775	2674224
0.0206	0.6130	60	2.4809	2918456
0.0233	0.6641	65	2.4910	3154264
0.0206	0.7152	70	2.4973	3394496
0.0216	0.7663	75	2.4903	3640792
0.0213	0.8174	80	2.4945	3895592
0.0199	0.8685	85	2.5025	4138160
0.0223	0.9195	90	2.5100	4383520
0.0217	0.9706	95	2.5158	4628920

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1