---
library_name: transformers
tags:
- generated_from_trainer
metrics:
- accuracy
model-index:
- name: babylm-default_seed-42_1e-3
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# babylm-default_seed-42_1e-3

This model was trained from scratch on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 3.0140
- Accuracy: 0.4206

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.001
- train_batch_size: 32
- eval_batch_size: 64
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 256
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 32000
- num_epochs: 20.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch   | Step  | Validation Loss | Accuracy |
|:-------------:|:-------:|:-----:|:---------------:|:--------:|
| 6.1739        | 0.9998  | 1788  | 4.2510          | 0.3061   |
| 4.046         | 1.9996  | 3576  | 3.7290          | 0.3476   |
| 3.6189        | 2.9999  | 5365  | 3.4764          | 0.3702   |
| 3.3937        | 3.9997  | 7153  | 3.3392          | 0.3835   |
| 3.31          | 4.9995  | 8941  | 3.2583          | 0.3910   |
| 3.2013        | 5.9999  | 10730 | 3.2094          | 0.3957   |
| 3.137         | 6.9997  | 12518 | 3.1786          | 0.3994   |
| 3.093         | 8.0     | 14307 | 3.1544          | 0.4016   |
| 3.0609        | 8.9998  | 16095 | 3.1376          | 0.4034   |
| 3.0177        | 9.9996  | 17883 | 3.1239          | 0.4050   |
| 2.996         | 10.9999 | 19672 | 3.1167          | 0.4059   |
| 2.9871        | 11.9997 | 21460 | 3.1099          | 0.4064   |
| 2.9784        | 12.9995 | 23248 | 3.1047          | 0.4073   |
| 2.9731        | 13.9999 | 25037 | 3.1005          | 0.4079   |
| 2.9327        | 14.9997 | 26825 | 3.0990          | 0.4084   |
| 2.9351        | 16.0    | 28614 | 3.0970          | 0.4088   |
| 2.9407        | 16.9998 | 30402 | 3.0905          | 0.4092   |
| 2.9456        | 17.9996 | 32190 | 3.0857          | 0.4099   |
| 2.8908        | 18.9999 | 33979 | 3.0347          | 0.4161   |
| 2.7363        | 19.9958 | 35760 | 3.0140          | 0.4206   |


### Framework versions

- Transformers 4.45.1
- Pytorch 2.4.1+cu121
- Datasets 2.19.1
- Tokenizers 0.20.0