babylm-unablated_seed-42_1e-3

This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 3.0094
• Accuracy: 0.4211

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 32
• eval_batch_size: 64
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 256
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 32000
• num_epochs: 20.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
6.1639	0.9999	1779	4.2644	0.3043
4.0418	1.9997	3558	3.7367	0.3467
3.6074	2.9996	5337	3.4761	0.3701
3.3836	4.0	7117	3.3380	0.3832
3.2939	4.9999	8896	3.2569	0.3913
3.1911	5.9997	10675	3.2081	0.3959
3.1276	6.9996	12454	3.1743	0.3993
3.0834	8.0	14234	3.1533	0.4014
3.0525	8.9999	16013	3.1344	0.4033
3.0036	9.9997	17792	3.1221	0.4050
2.9901	10.9996	19571	3.1156	0.4056
2.9798	12.0	21351	3.1083	0.4069
2.9729	12.9999	23130	3.1015	0.4074
2.9356	13.9997	24909	3.0997	0.4076
2.9282	14.9996	26688	3.0948	0.4084
2.9332	16.0	28468	3.0928	0.4090
2.9346	16.9999	30247	3.0900	0.4090
2.9394	17.9997	32026	3.0874	0.4097
2.8743	18.9996	33805	3.0350	0.4163
2.729	19.9972	35580	3.0094	0.4211

Framework versions

• Transformers 4.45.1
• Pytorch 2.4.1+cu121
• Datasets 2.19.1
• Tokenizers 0.20.0