babylm-default_seed-42_1e-3

This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 3.0140
• Accuracy: 0.4206

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 32
• eval_batch_size: 64
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 256
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 32000
• num_epochs: 20.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
6.1739	0.9998	1788	4.2510	0.3061
4.046	1.9996	3576	3.7290	0.3476
3.6189	2.9999	5365	3.4764	0.3702
3.3937	3.9997	7153	3.3392	0.3835
3.31	4.9995	8941	3.2583	0.3910
3.2013	5.9999	10730	3.2094	0.3957
3.137	6.9997	12518	3.1786	0.3994
3.093	8.0	14307	3.1544	0.4016
3.0609	8.9998	16095	3.1376	0.4034
3.0177	9.9996	17883	3.1239	0.4050
2.996	10.9999	19672	3.1167	0.4059
2.9871	11.9997	21460	3.1099	0.4064
2.9784	12.9995	23248	3.1047	0.4073
2.9731	13.9999	25037	3.1005	0.4079
2.9327	14.9997	26825	3.0990	0.4084
2.9351	16.0	28614	3.0970	0.4088
2.9407	16.9998	30402	3.0905	0.4092
2.9456	17.9996	32190	3.0857	0.4099
2.8908	18.9999	33979	3.0347	0.4161
2.7363	19.9958	35760	3.0140	0.4206

Framework versions

• Transformers 4.45.1
• Pytorch 2.4.1+cu121
• Datasets 2.19.1
• Tokenizers 0.20.0