opt-babylm2-rewritten-clean-spacy_random-removal-num-adj-earlystop-bpe_seed-42_1e-3

This model was trained from scratch on the kanishka/babylm2-rewritten-clean-spacy_random-removal-num-adj dataset. It achieves the following results on the evaluation set:

• Loss: 2.6863
• Accuracy: 0.4783

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 32
• eval_batch_size: 64
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 256
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 32000
• num_epochs: 20.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
4.1066	0.9997	2240	3.8160	0.3611
3.446	1.9997	4480	3.3024	0.4093
3.1301	2.9997	6720	3.0883	0.4305
2.9677	3.9997	8960	2.9857	0.4410
2.8452	4.9997	11200	2.9248	0.4470
2.7915	5.9997	13440	2.8839	0.4515
2.746	6.9997	15680	2.8591	0.4544
2.7113	7.9997	17920	2.8384	0.4566
2.6925	8.9997	20160	2.8258	0.4583
2.6723	9.9997	22400	2.8152	0.4594
2.657	10.9997	24640	2.8094	0.4603
2.6394	11.9997	26880	2.7989	0.4615
2.6428	12.9997	29120	2.7990	0.4618
2.636	13.9997	31360	2.7911	0.4622
2.6195	14.9997	33600	2.7706	0.4649
2.5716	15.9997	35840	2.7441	0.4685
2.5238	16.9997	38080	2.7221	0.4713
2.4675	17.9997	40320	2.7048	0.4741
2.4042	18.9997	42560	2.6900	0.4769
2.3318	19.9997	44800	2.6863	0.4783

Framework versions

• Transformers 4.48.0
• Pytorch 2.5.1
• Datasets 3.2.0
• Tokenizers 0.21.0