collapse_gemma-2-2b_hs2_accumulatesubsample_iter5_sftsd0

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.1651
• Num Input Tokens Seen: 5121720

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 0
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3909	0
1.4516	0.0549	5	1.2704	281728
1.2569	0.1098	10	1.1896	560496
1.1565	0.1647	15	1.1711	843552
1.0247	0.2196	20	1.1630	1124256
0.999	0.2745	25	1.1730	1405360
0.9404	0.3294	30	1.1667	1687800
0.8346	0.3844	35	1.1909	1973600
0.8733	0.4393	40	1.1780	2246208
0.7992	0.4942	45	1.1868	2527096
0.597	0.5491	50	1.1766	2813840
0.6897	0.6040	55	1.1795	3093112
0.6487	0.6589	60	1.1741	3372936
0.6013	0.7138	65	1.1733	3651336
0.6563	0.7687	70	1.1680	3931512
0.5705	0.8236	75	1.1709	4216528
0.6287	0.8785	80	1.1732	4498448
0.5377	0.9334	85	1.1693	4778952
0.6489	0.9883	90	1.1661	5063320

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1