collapse_gemma-2-2b_hs2_replace_iter6_sftsd0

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.3592
• Num Input Tokens Seen: 5050360

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 0
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3909	0
1.4425	0.0511	5	1.2792	268048
1.0011	0.1022	10	1.2830	528720
0.5665	0.1533	15	1.5189	789760
0.3288	0.2043	20	1.6677	1055920
0.1659	0.2554	25	1.8473	1316120
0.0756	0.3065	30	1.9989	1571768
0.0561	0.3576	35	2.2048	1827248
0.0501	0.4087	40	2.2714	2080432
0.0308	0.4598	45	2.3411	2341488
0.0309	0.5109	50	2.3422	2597752
0.0273	0.5619	55	2.3220	2858344
0.028	0.6130	60	2.3425	3126248
0.0239	0.6641	65	2.3425	3385352
0.0303	0.7152	70	2.3531	3653912
0.0244	0.7663	75	2.3690	3915120
0.0242	0.8174	80	2.3697	4168000
0.026	0.8685	85	2.3791	4422656
0.0253	0.9195	90	2.3785	4684312
0.0239	0.9706	95	2.3603	4949120

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1