metadata
license: gemma
base_model: google/gemma-2-2b
tags:
- trl
- sft
- generated_from_trainer
model-index:
- name: collapse_gemma-2-2b_hs2_accumulate_iter3_sftsd0
results: []
collapse_gemma-2-2b_hs2_accumulate_iter3_sftsd0
This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:
- Loss: 1.1114
- Num Input Tokens Seen: 21798600
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 8e-06
- train_batch_size: 8
- eval_batch_size: 16
- seed: 0
- gradient_accumulation_steps: 16
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: constant_with_warmup
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 1
Training results
| Training Loss | Epoch | Step | Validation Loss | Input Tokens Seen |
|---|---|---|---|---|
| No log | 0 | 0 | 1.3956 | 0 |
| 1.5454 | 0.0129 | 5 | 1.3798 | 284096 |
| 1.5595 | 0.0258 | 10 | 1.2917 | 565048 |
| 1.4801 | 0.0388 | 15 | 1.2113 | 857320 |
| 1.2583 | 0.0517 | 20 | 1.1640 | 1143680 |
| 1.2536 | 0.0646 | 25 | 1.1396 | 1426896 |
| 1.1682 | 0.0775 | 30 | 1.1244 | 1704888 |
| 1.1565 | 0.0905 | 35 | 1.1242 | 1985240 |
| 1.0138 | 0.1034 | 40 | 1.1384 | 2269216 |
| 0.9845 | 0.1163 | 45 | 1.1461 | 2554344 |
| 0.91 | 0.1292 | 50 | 1.1554 | 2839272 |
| 0.9047 | 0.1422 | 55 | 1.1678 | 3127496 |
| 0.9137 | 0.1551 | 60 | 1.1697 | 3415328 |
| 0.8846 | 0.1680 | 65 | 1.1704 | 3692024 |
| 0.9215 | 0.1809 | 70 | 1.1719 | 3967168 |
| 0.8233 | 0.1939 | 75 | 1.1850 | 4244568 |
| 0.6717 | 0.2068 | 80 | 1.1881 | 4531936 |
| 0.7733 | 0.2197 | 85 | 1.1770 | 4817232 |
| 0.6835 | 0.2326 | 90 | 1.1663 | 5103112 |
| 0.7503 | 0.2456 | 95 | 1.1860 | 5388248 |
| 0.6998 | 0.2585 | 100 | 1.1702 | 5669656 |
| 0.615 | 0.2714 | 105 | 1.1739 | 5956384 |
| 0.5807 | 0.2843 | 110 | 1.1799 | 6233928 |
| 0.6475 | 0.2973 | 115 | 1.1703 | 6517360 |
| 0.649 | 0.3102 | 120 | 1.1702 | 6802600 |
| 0.6409 | 0.3231 | 125 | 1.1747 | 7086032 |
| 0.6033 | 0.3360 | 130 | 1.1629 | 7364952 |
| 0.4875 | 0.3489 | 135 | 1.1752 | 7650744 |
| 0.6259 | 0.3619 | 140 | 1.1664 | 7933080 |
| 0.5287 | 0.3748 | 145 | 1.1703 | 8220488 |
| 0.4745 | 0.3877 | 150 | 1.1645 | 8501544 |
| 0.4469 | 0.4006 | 155 | 1.1667 | 8781400 |
| 0.5011 | 0.4136 | 160 | 1.1652 | 9056664 |
| 0.4512 | 0.4265 | 165 | 1.1630 | 9337208 |
| 0.5347 | 0.4394 | 170 | 1.1630 | 9620568 |
| 0.5226 | 0.4523 | 175 | 1.1626 | 9896128 |
| 0.4775 | 0.4653 | 180 | 1.1568 | 10176840 |
| 0.5018 | 0.4782 | 185 | 1.1642 | 10461520 |
| 0.508 | 0.4911 | 190 | 1.1530 | 10741632 |
| 0.3972 | 0.5040 | 195 | 1.1550 | 11024096 |
| 0.4409 | 0.5170 | 200 | 1.1539 | 11301736 |
| 0.5384 | 0.5299 | 205 | 1.1477 | 11579816 |
| 0.4633 | 0.5428 | 210 | 1.1501 | 11865648 |
| 0.5198 | 0.5557 | 215 | 1.1410 | 12156088 |
| 0.3293 | 0.5687 | 220 | 1.1480 | 12434448 |
| 0.4762 | 0.5816 | 225 | 1.1375 | 12720344 |
| 0.5467 | 0.5945 | 230 | 1.1424 | 13003704 |
| 0.4776 | 0.6074 | 235 | 1.1361 | 13292824 |
| 0.4567 | 0.6204 | 240 | 1.1398 | 13574560 |
| 0.4565 | 0.6333 | 245 | 1.1371 | 13859632 |
| 0.4899 | 0.6462 | 250 | 1.1369 | 14136888 |
| 0.3492 | 0.6591 | 255 | 1.1327 | 14421200 |
| 0.4968 | 0.6721 | 260 | 1.1315 | 14707344 |
| 0.3487 | 0.6850 | 265 | 1.1329 | 14988680 |
| 0.4001 | 0.6979 | 270 | 1.1258 | 15267688 |
| 0.3161 | 0.7108 | 275 | 1.1308 | 15540888 |
| 0.4089 | 0.7237 | 280 | 1.1262 | 15816840 |
| 0.3835 | 0.7367 | 285 | 1.1289 | 16098568 |
| 0.4023 | 0.7496 | 290 | 1.1270 | 16387224 |
| 0.5333 | 0.7625 | 295 | 1.1243 | 16672848 |
| 0.492 | 0.7754 | 300 | 1.1276 | 16955104 |
| 0.3361 | 0.7884 | 305 | 1.1215 | 17232984 |
| 0.4585 | 0.8013 | 310 | 1.1210 | 17517512 |
| 0.3541 | 0.8142 | 315 | 1.1232 | 17805408 |
| 0.4862 | 0.8271 | 320 | 1.1195 | 18086744 |
| 0.5085 | 0.8401 | 325 | 1.1208 | 18374072 |
| 0.4206 | 0.8530 | 330 | 1.1198 | 18654568 |
| 0.3501 | 0.8659 | 335 | 1.1154 | 18936680 |
| 0.4675 | 0.8788 | 340 | 1.1207 | 19213288 |
| 0.3692 | 0.8918 | 345 | 1.1151 | 19495512 |
| 0.3526 | 0.9047 | 350 | 1.1162 | 19777904 |
| 0.5192 | 0.9176 | 355 | 1.1134 | 20053800 |
| 0.5117 | 0.9305 | 360 | 1.1101 | 20335472 |
| 0.3685 | 0.9435 | 365 | 1.1152 | 20620416 |
| 0.3554 | 0.9564 | 370 | 1.1103 | 20898680 |
| 0.4323 | 0.9693 | 375 | 1.1123 | 21181272 |
| 0.4111 | 0.9822 | 380 | 1.1120 | 21465480 |
| 0.3962 | 0.9952 | 385 | 1.1119 | 21742008 |
Framework versions
- Transformers 4.44.0
- Pytorch 2.4.0+cu121
- Datasets 2.20.0
- Tokenizers 0.19.1