Edit model card

opt-babylm2-clean-20-epochs-earlystop_seed-42_1e-3

This model was trained from scratch on the kanishka/babylm2-clean dataset. It achieves the following results on the evaluation set:

  • Loss: 2.7611
  • Accuracy: 0.4673

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.001
  • train_batch_size: 32
  • eval_batch_size: 64
  • seed: 42
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 256
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 32000
  • num_epochs: 20.0
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Accuracy
4.24 0.9998 2198 3.9528 0.3447
3.5581 2.0 4397 3.4120 0.3948
3.2191 2.9998 6595 3.1806 0.4180
3.0495 4.0 8794 3.0720 0.4287
2.9458 4.9998 10992 3.0054 0.4354
2.8669 6.0 13191 2.9663 0.4399
2.8256 6.9998 15389 2.9382 0.4430
2.79 8.0 17588 2.9199 0.4452
2.7624 8.9998 19786 2.9052 0.4468
2.7361 10.0 21985 2.8915 0.4482
2.7354 10.9998 24183 2.8843 0.4491
2.7225 12.0 26382 2.8777 0.4500
2.7092 12.9998 28580 2.8708 0.4505
2.6987 14.0 30779 2.8688 0.4509
2.6894 14.9998 32977 2.8542 0.4527
2.6561 16.0 35176 2.8258 0.4564
2.6055 16.9998 37374 2.8005 0.4595
2.5464 18.0 39573 2.7814 0.4627
2.4778 18.9998 41771 2.7630 0.4658
2.4036 19.9955 43960 2.7611 0.4673

Framework versions

  • Transformers 4.45.1
  • Pytorch 2.4.1+cu121
  • Datasets 3.0.1
  • Tokenizers 0.20.0
Downloads last month
14
Safetensors
Model size
97.8M params
Tensor type
F32
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Dataset used to train kanishka/opt-babylm2-clean-20-epochs-earlystop_seed-42_1e-3

Evaluation results