Model save

README.md

@@ -0,0 +1,80 @@
+---
+library_name: transformers
+tags:
+- generated_from_trainer
+metrics:
+- accuracy
+model-index:
+- name: opt-babylm2-clean-spacy-32k_seed-42_1e-3
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# opt-babylm2-clean-spacy-32k_seed-42_1e-3
+
+This model was trained from scratch on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 3.0357
+- Accuracy: 0.4240
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.001
+- train_batch_size: 32
+- eval_batch_size: 64
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_steps: 32000
+- num_epochs: 20.0
+- mixed_precision_training: Native AMP
+
+### Training results
+
+| Training Loss | Epoch | Step   | Validation Loss | Accuracy |
+|:-------------:|:-----:|:------:|:---------------:|:--------:|
+| 3.2533        | 1.0   | 15580  | 3.3142          | 0.3828   |
+| 3.064         | 2.0   | 31160  | 3.1478          | 0.4002   |
+| 2.9372        | 3.0   | 46740  | 3.0373          | 0.4123   |
+| 2.8458        | 4.0   | 62320  | 2.9892          | 0.4180   |
+| 2.7931        | 5.0   | 77900  | 2.9617          | 0.4212   |
+| 2.7418        | 6.0   | 93480  | 2.9472          | 0.4235   |
+| 2.7019        | 7.0   | 109060 | 2.9387          | 0.4251   |
+| 2.6555        | 8.0   | 124640 | 2.9388          | 0.4259   |
+| 2.6321        | 9.0   | 140220 | 2.9394          | 0.4263   |
+| 2.5968        | 10.0  | 155800 | 2.9411          | 0.4265   |
+| 2.5705        | 11.0  | 171380 | 2.9455          | 0.4267   |
+| 2.5402        | 12.0  | 186960 | 2.9545          | 0.4266   |
+| 2.5136        | 13.0  | 202540 | 2.9615          | 0.4267   |
+| 2.4814        | 14.0  | 218120 | 2.9730          | 0.4261   |
+| 2.4607        | 15.0  | 233700 | 2.9827          | 0.4256   |
+| 2.4394        | 16.0  | 249280 | 2.9935          | 0.4256   |
+| 2.4062        | 17.0  | 264860 | 3.0019          | 0.4253   |
+| 2.3845        | 18.0  | 280440 | 3.0156          | 0.4246   |
+| 2.3632        | 19.0  | 296020 | 3.0269          | 0.4244   |
+| 2.3437        | 20.0  | 311600 | 3.0357          | 0.4240   |
+
+
+### Framework versions
+
+- Transformers 4.45.1
+- Pytorch 2.4.1+cu121
+- Datasets 3.0.1
+- Tokenizers 0.20.0

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 1,
+  "pad_token_id": 0,
+  "transformers_version": "4.45.1"
+}