--- license: gemma base_model: google/gemma-2-2b tags: - trl - sft - generated_from_trainer model-index: - name: collapse_gemma-2-2b_hs2_accumulate_iter3_sftsd0 results: [] --- # collapse_gemma-2-2b_hs2_accumulate_iter3_sftsd0 This model is a fine-tuned version of [google/gemma-2-2b](https://huggingface.co/google/gemma-2-2b) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 1.1114 - Num Input Tokens Seen: 21798600 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 8e-06 - train_batch_size: 8 - eval_batch_size: 16 - seed: 0 - gradient_accumulation_steps: 16 - total_train_batch_size: 128 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: constant_with_warmup - lr_scheduler_warmup_ratio: 0.05 - num_epochs: 1 ### Training results | Training Loss | Epoch | Step | Validation Loss | Input Tokens Seen | |:-------------:|:------:|:----:|:---------------:|:-----------------:| | No log | 0 | 0 | 1.3956 | 0 | | 1.5454 | 0.0129 | 5 | 1.3798 | 284096 | | 1.5595 | 0.0258 | 10 | 1.2917 | 565048 | | 1.4801 | 0.0388 | 15 | 1.2113 | 857320 | | 1.2583 | 0.0517 | 20 | 1.1640 | 1143680 | | 1.2536 | 0.0646 | 25 | 1.1396 | 1426896 | | 1.1682 | 0.0775 | 30 | 1.1244 | 1704888 | | 1.1565 | 0.0905 | 35 | 1.1242 | 1985240 | | 1.0138 | 0.1034 | 40 | 1.1384 | 2269216 | | 0.9845 | 0.1163 | 45 | 1.1461 | 2554344 | | 0.91 | 0.1292 | 50 | 1.1554 | 2839272 | | 0.9047 | 0.1422 | 55 | 1.1678 | 3127496 | | 0.9137 | 0.1551 | 60 | 1.1697 | 3415328 | | 0.8846 | 0.1680 | 65 | 1.1704 | 3692024 | | 0.9215 | 0.1809 | 70 | 1.1719 | 3967168 | | 0.8233 | 0.1939 | 75 | 1.1850 | 4244568 | | 0.6717 | 0.2068 | 80 | 1.1881 | 4531936 | | 0.7733 | 0.2197 | 85 | 1.1770 | 4817232 | | 0.6835 | 0.2326 | 90 | 1.1663 | 5103112 | | 0.7503 | 0.2456 | 95 | 1.1860 | 5388248 | | 0.6998 | 0.2585 | 100 | 1.1702 | 5669656 | | 0.615 | 0.2714 | 105 | 1.1739 | 5956384 | | 0.5807 | 0.2843 | 110 | 1.1799 | 6233928 | | 0.6475 | 0.2973 | 115 | 1.1703 | 6517360 | | 0.649 | 0.3102 | 120 | 1.1702 | 6802600 | | 0.6409 | 0.3231 | 125 | 1.1747 | 7086032 | | 0.6033 | 0.3360 | 130 | 1.1629 | 7364952 | | 0.4875 | 0.3489 | 135 | 1.1752 | 7650744 | | 0.6259 | 0.3619 | 140 | 1.1664 | 7933080 | | 0.5287 | 0.3748 | 145 | 1.1703 | 8220488 | | 0.4745 | 0.3877 | 150 | 1.1645 | 8501544 | | 0.4469 | 0.4006 | 155 | 1.1667 | 8781400 | | 0.5011 | 0.4136 | 160 | 1.1652 | 9056664 | | 0.4512 | 0.4265 | 165 | 1.1630 | 9337208 | | 0.5347 | 0.4394 | 170 | 1.1630 | 9620568 | | 0.5226 | 0.4523 | 175 | 1.1626 | 9896128 | | 0.4775 | 0.4653 | 180 | 1.1568 | 10176840 | | 0.5018 | 0.4782 | 185 | 1.1642 | 10461520 | | 0.508 | 0.4911 | 190 | 1.1530 | 10741632 | | 0.3972 | 0.5040 | 195 | 1.1550 | 11024096 | | 0.4409 | 0.5170 | 200 | 1.1539 | 11301736 | | 0.5384 | 0.5299 | 205 | 1.1477 | 11579816 | | 0.4633 | 0.5428 | 210 | 1.1501 | 11865648 | | 0.5198 | 0.5557 | 215 | 1.1410 | 12156088 | | 0.3293 | 0.5687 | 220 | 1.1480 | 12434448 | | 0.4762 | 0.5816 | 225 | 1.1375 | 12720344 | | 0.5467 | 0.5945 | 230 | 1.1424 | 13003704 | | 0.4776 | 0.6074 | 235 | 1.1361 | 13292824 | | 0.4567 | 0.6204 | 240 | 1.1398 | 13574560 | | 0.4565 | 0.6333 | 245 | 1.1371 | 13859632 | | 0.4899 | 0.6462 | 250 | 1.1369 | 14136888 | | 0.3492 | 0.6591 | 255 | 1.1327 | 14421200 | | 0.4968 | 0.6721 | 260 | 1.1315 | 14707344 | | 0.3487 | 0.6850 | 265 | 1.1329 | 14988680 | | 0.4001 | 0.6979 | 270 | 1.1258 | 15267688 | | 0.3161 | 0.7108 | 275 | 1.1308 | 15540888 | | 0.4089 | 0.7237 | 280 | 1.1262 | 15816840 | | 0.3835 | 0.7367 | 285 | 1.1289 | 16098568 | | 0.4023 | 0.7496 | 290 | 1.1270 | 16387224 | | 0.5333 | 0.7625 | 295 | 1.1243 | 16672848 | | 0.492 | 0.7754 | 300 | 1.1276 | 16955104 | | 0.3361 | 0.7884 | 305 | 1.1215 | 17232984 | | 0.4585 | 0.8013 | 310 | 1.1210 | 17517512 | | 0.3541 | 0.8142 | 315 | 1.1232 | 17805408 | | 0.4862 | 0.8271 | 320 | 1.1195 | 18086744 | | 0.5085 | 0.8401 | 325 | 1.1208 | 18374072 | | 0.4206 | 0.8530 | 330 | 1.1198 | 18654568 | | 0.3501 | 0.8659 | 335 | 1.1154 | 18936680 | | 0.4675 | 0.8788 | 340 | 1.1207 | 19213288 | | 0.3692 | 0.8918 | 345 | 1.1151 | 19495512 | | 0.3526 | 0.9047 | 350 | 1.1162 | 19777904 | | 0.5192 | 0.9176 | 355 | 1.1134 | 20053800 | | 0.5117 | 0.9305 | 360 | 1.1101 | 20335472 | | 0.3685 | 0.9435 | 365 | 1.1152 | 20620416 | | 0.3554 | 0.9564 | 370 | 1.1103 | 20898680 | | 0.4323 | 0.9693 | 375 | 1.1123 | 21181272 | | 0.4111 | 0.9822 | 380 | 1.1120 | 21465480 | | 0.3962 | 0.9952 | 385 | 1.1119 | 21742008 | ### Framework versions - Transformers 4.44.0 - Pytorch 2.4.0+cu121 - Datasets 2.20.0 - Tokenizers 0.19.1