--- library_name: transformers license: apache-2.0 base_model: answerdotai/ModernBERT-base tags: - generated_from_trainer metrics: - f1 model-index: - name: ST2_modernbert-base_product_V1 results: [] --- # ST2_modernbert-base_product_V1 This model is a fine-tuned version of [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) on the None dataset. It achieves the following results on the evaluation set: - Loss: 3.5744 - F1: 0.5126 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 5e-05 - train_batch_size: 36 - eval_batch_size: 16 - seed: 42 - optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: linear - num_epochs: 200 ### Training results | Training Loss | Epoch | Step | Validation Loss | F1 | |:-------------:|:-----:|:-----:|:---------------:|:------:| | 6.5711 | 1.0 | 124 | 6.1700 | 0.0106 | | 5.9394 | 2.0 | 248 | 4.8593 | 0.1670 | | 4.6045 | 3.0 | 372 | 3.8976 | 0.3029 | | 2.9945 | 4.0 | 496 | 3.3339 | 0.4090 | | 0.6088 | 5.0 | 620 | 3.0378 | 0.4559 | | 0.2897 | 6.0 | 744 | 3.0109 | 0.4816 | | 0.1962 | 7.0 | 868 | 3.1202 | 0.4717 | | 0.1025 | 8.0 | 992 | 3.0573 | 0.4715 | | 0.0459 | 9.0 | 1116 | 3.1049 | 0.4814 | | 0.0489 | 10.0 | 1240 | 3.1277 | 0.4863 | | 0.0292 | 11.0 | 1364 | 3.0771 | 0.4875 | | 0.0426 | 12.0 | 1488 | 3.1132 | 0.4936 | | 0.0164 | 13.0 | 1612 | 3.0944 | 0.5108 | | 0.0447 | 14.0 | 1736 | 3.1211 | 0.4910 | | 0.0103 | 15.0 | 1860 | 3.0736 | 0.5010 | | 0.0134 | 16.0 | 1984 | 3.1056 | 0.4982 | | 0.0082 | 17.0 | 2108 | 3.0771 | 0.4997 | | 0.0068 | 18.0 | 2232 | 3.0851 | 0.5018 | | 0.0088 | 19.0 | 2356 | 3.0996 | 0.4991 | | 0.0024 | 20.0 | 2480 | 3.0850 | 0.5086 | | 0.0059 | 21.0 | 2604 | 3.0509 | 0.5026 | | 0.0048 | 22.0 | 2728 | 3.1242 | 0.5047 | | 0.004 | 23.0 | 2852 | 3.1142 | 0.4969 | | 0.0041 | 24.0 | 2976 | 3.1523 | 0.5023 | | 0.0041 | 25.0 | 3100 | 3.1448 | 0.4995 | | 0.0028 | 26.0 | 3224 | 3.1528 | 0.5006 | | 0.0057 | 27.0 | 3348 | 3.1483 | 0.5007 | | 0.0029 | 28.0 | 3472 | 3.1724 | 0.5020 | | 0.0033 | 29.0 | 3596 | 3.1834 | 0.4958 | | 0.0034 | 30.0 | 3720 | 3.1474 | 0.4977 | | 0.0038 | 31.0 | 3844 | 3.1788 | 0.4989 | | 0.0037 | 32.0 | 3968 | 3.1785 | 0.4935 | | 0.0034 | 33.0 | 4092 | 3.1674 | 0.4987 | | 0.0027 | 34.0 | 4216 | 3.1771 | 0.4987 | | 0.0041 | 35.0 | 4340 | 3.1944 | 0.4945 | | 0.0019 | 36.0 | 4464 | 3.1890 | 0.5032 | | 0.0028 | 37.0 | 4588 | 3.1763 | 0.4951 | | 0.0022 | 38.0 | 4712 | 3.2159 | 0.4958 | | 0.0037 | 39.0 | 4836 | 3.2109 | 0.5032 | | 0.0026 | 40.0 | 4960 | 3.1828 | 0.4959 | | 0.0028 | 41.0 | 5084 | 3.2274 | 0.4982 | | 0.0025 | 42.0 | 5208 | 3.1930 | 0.4970 | | 0.0036 | 43.0 | 5332 | 3.2186 | 0.4945 | | 0.0032 | 44.0 | 5456 | 3.2385 | 0.5010 | | 0.0016 | 45.0 | 5580 | 3.2317 | 0.5019 | | 0.0034 | 46.0 | 5704 | 3.2135 | 0.5036 | | 0.0035 | 47.0 | 5828 | 3.3843 | 0.4602 | | 0.1754 | 48.0 | 5952 | 3.2086 | 0.4761 | | 0.1549 | 49.0 | 6076 | 3.3204 | 0.4804 | | 0.0232 | 50.0 | 6200 | 3.3169 | 0.4906 | | 0.0173 | 51.0 | 6324 | 3.3614 | 0.4905 | | 0.0149 | 52.0 | 6448 | 3.3885 | 0.4814 | | 0.0085 | 53.0 | 6572 | 3.3473 | 0.4901 | | 0.0049 | 54.0 | 6696 | 3.3235 | 0.4996 | | 0.0027 | 55.0 | 6820 | 3.3272 | 0.4986 | | 0.0013 | 56.0 | 6944 | 3.3385 | 0.5022 | | 0.0039 | 57.0 | 7068 | 3.3398 | 0.5050 | | 0.0025 | 58.0 | 7192 | 3.3412 | 0.5063 | | 0.0025 | 59.0 | 7316 | 3.3466 | 0.5044 | | 0.0022 | 60.0 | 7440 | 3.3510 | 0.5054 | | 0.0018 | 61.0 | 7564 | 3.3570 | 0.5063 | | 0.003 | 62.0 | 7688 | 3.3553 | 0.5047 | | 0.0027 | 63.0 | 7812 | 3.3642 | 0.5053 | | 0.0027 | 64.0 | 7936 | 3.3615 | 0.5061 | | 0.0019 | 65.0 | 8060 | 3.3664 | 0.5053 | | 0.003 | 66.0 | 8184 | 3.3675 | 0.5059 | | 0.0028 | 67.0 | 8308 | 3.3707 | 0.5063 | | 0.0023 | 68.0 | 8432 | 3.3754 | 0.5060 | | 0.0018 | 69.0 | 8556 | 3.3743 | 0.5061 | | 0.0039 | 70.0 | 8680 | 3.3793 | 0.5080 | | 0.0029 | 71.0 | 8804 | 3.3868 | 0.5101 | | 0.0017 | 72.0 | 8928 | 3.3829 | 0.5062 | | 0.0035 | 73.0 | 9052 | 3.3913 | 0.5072 | | 0.0026 | 74.0 | 9176 | 3.3910 | 0.5085 | | 0.0029 | 75.0 | 9300 | 3.3827 | 0.5110 | | 0.0018 | 76.0 | 9424 | 3.3985 | 0.5084 | | 0.0027 | 77.0 | 9548 | 3.3946 | 0.5068 | | 0.0021 | 78.0 | 9672 | 3.3975 | 0.5113 | | 0.0025 | 79.0 | 9796 | 3.3949 | 0.5066 | | 0.0028 | 80.0 | 9920 | 3.4022 | 0.5098 | | 0.0015 | 81.0 | 10044 | 3.4100 | 0.5082 | | 0.0026 | 82.0 | 10168 | 3.3912 | 0.5120 | | 0.0028 | 83.0 | 10292 | 3.4092 | 0.5122 | | 0.0031 | 84.0 | 10416 | 3.3857 | 0.5125 | | 0.002 | 85.0 | 10540 | 3.4220 | 0.5096 | | 0.0013 | 86.0 | 10664 | 3.4071 | 0.5141 | | 0.003 | 87.0 | 10788 | 3.4105 | 0.5148 | | 0.002 | 88.0 | 10912 | 3.4124 | 0.5130 | | 0.0025 | 89.0 | 11036 | 3.4248 | 0.5123 | | 0.0028 | 90.0 | 11160 | 3.4086 | 0.5119 | | 0.0022 | 91.0 | 11284 | 3.4224 | 0.5113 | | 0.0024 | 92.0 | 11408 | 3.4295 | 0.5162 | | 0.0022 | 93.0 | 11532 | 3.4159 | 0.5138 | | 0.0025 | 94.0 | 11656 | 3.4153 | 0.5130 | | 0.0023 | 95.0 | 11780 | 3.4355 | 0.5133 | | 0.0027 | 96.0 | 11904 | 3.4323 | 0.5174 | | 0.0018 | 97.0 | 12028 | 3.3888 | 0.5160 | | 0.0029 | 98.0 | 12152 | 3.4415 | 0.5125 | | 0.0028 | 99.0 | 12276 | 3.4289 | 0.5122 | | 0.0024 | 100.0 | 12400 | 3.4408 | 0.5171 | | 0.002 | 101.0 | 12524 | 3.4226 | 0.5148 | | 0.0025 | 102.0 | 12648 | 3.4544 | 0.5147 | | 0.0019 | 103.0 | 12772 | 3.4467 | 0.5148 | | 0.0024 | 104.0 | 12896 | 3.4552 | 0.5188 | | 0.0026 | 105.0 | 13020 | 3.4581 | 0.5178 | | 0.0023 | 106.0 | 13144 | 3.4570 | 0.5159 | | 0.0019 | 107.0 | 13268 | 3.4456 | 0.5136 | | 0.0019 | 108.0 | 13392 | 3.4553 | 0.5170 | | 0.0024 | 109.0 | 13516 | 3.4750 | 0.5115 | | 0.0015 | 110.0 | 13640 | 3.4556 | 0.5195 | | 0.0027 | 111.0 | 13764 | 3.4916 | 0.5207 | | 0.0023 | 112.0 | 13888 | 3.4637 | 0.5120 | | 0.0014 | 113.0 | 14012 | 3.4714 | 0.5141 | | 0.0026 | 114.0 | 14136 | 3.4919 | 0.5182 | | 0.0024 | 115.0 | 14260 | 3.4987 | 0.5169 | | 0.0016 | 116.0 | 14384 | 3.5065 | 0.5179 | | 0.0023 | 117.0 | 14508 | 3.4585 | 0.5154 | | 0.0019 | 118.0 | 14632 | 3.4927 | 0.5139 | | 0.0014 | 119.0 | 14756 | 3.4963 | 0.5150 | | 0.0031 | 120.0 | 14880 | 3.5130 | 0.5190 | | 0.0021 | 121.0 | 15004 | 3.4772 | 0.5117 | | 0.0021 | 122.0 | 15128 | 3.5224 | 0.5131 | | 0.003 | 123.0 | 15252 | 3.4794 | 0.5165 | | 0.0013 | 124.0 | 15376 | 3.4911 | 0.5099 | | 0.0064 | 125.0 | 15500 | 3.5023 | 0.4909 | | 0.0238 | 126.0 | 15624 | 3.3523 | 0.5030 | | 0.015 | 127.0 | 15748 | 3.4065 | 0.5028 | | 0.0039 | 128.0 | 15872 | 3.3460 | 0.4963 | | 0.0067 | 129.0 | 15996 | 3.3763 | 0.5062 | | 0.0021 | 130.0 | 16120 | 3.3880 | 0.5077 | | 0.0018 | 131.0 | 16244 | 3.3969 | 0.5093 | | 0.0022 | 132.0 | 16368 | 3.4017 | 0.5100 | | 0.0021 | 133.0 | 16492 | 3.4123 | 0.5084 | | 0.002 | 134.0 | 16616 | 3.4158 | 0.5122 | | 0.0019 | 135.0 | 16740 | 3.4215 | 0.5117 | | 0.0017 | 136.0 | 16864 | 3.4257 | 0.5103 | | 0.0023 | 137.0 | 16988 | 3.4289 | 0.5141 | | 0.0018 | 138.0 | 17112 | 3.4344 | 0.5101 | | 0.0023 | 139.0 | 17236 | 3.4371 | 0.5110 | | 0.0014 | 140.0 | 17360 | 3.4411 | 0.5133 | | 0.0019 | 141.0 | 17484 | 3.4437 | 0.5127 | | 0.002 | 142.0 | 17608 | 3.4484 | 0.5138 | | 0.002 | 143.0 | 17732 | 3.4503 | 0.5127 | | 0.0017 | 144.0 | 17856 | 3.4534 | 0.5117 | | 0.0015 | 145.0 | 17980 | 3.4578 | 0.5143 | | 0.0015 | 146.0 | 18104 | 3.4613 | 0.5099 | | 0.002 | 147.0 | 18228 | 3.4645 | 0.5109 | | 0.0012 | 148.0 | 18352 | 3.4679 | 0.5097 | | 0.0026 | 149.0 | 18476 | 3.4691 | 0.5092 | | 0.0016 | 150.0 | 18600 | 3.4734 | 0.5088 | | 0.0017 | 151.0 | 18724 | 3.4754 | 0.5102 | | 0.0023 | 152.0 | 18848 | 3.4798 | 0.5114 | | 0.0024 | 153.0 | 18972 | 3.4822 | 0.5082 | | 0.001 | 154.0 | 19096 | 3.4831 | 0.5104 | | 0.002 | 155.0 | 19220 | 3.4885 | 0.5083 | | 0.0016 | 156.0 | 19344 | 3.4904 | 0.5104 | | 0.0014 | 157.0 | 19468 | 3.4935 | 0.5101 | | 0.0021 | 158.0 | 19592 | 3.4980 | 0.5103 | | 0.0021 | 159.0 | 19716 | 3.4991 | 0.5113 | | 0.0015 | 160.0 | 19840 | 3.5031 | 0.5100 | | 0.0026 | 161.0 | 19964 | 3.5050 | 0.5101 | | 0.0013 | 162.0 | 20088 | 3.5095 | 0.5089 | | 0.0018 | 163.0 | 20212 | 3.5128 | 0.5099 | | 0.0021 | 164.0 | 20336 | 3.5148 | 0.5116 | | 0.0021 | 165.0 | 20460 | 3.5149 | 0.5116 | | 0.0017 | 166.0 | 20584 | 3.5189 | 0.5095 | | 0.0018 | 167.0 | 20708 | 3.5232 | 0.5120 | | 0.0018 | 168.0 | 20832 | 3.5277 | 0.5099 | | 0.0024 | 169.0 | 20956 | 3.5277 | 0.5115 | | 0.0015 | 170.0 | 21080 | 3.5282 | 0.5111 | | 0.0015 | 171.0 | 21204 | 3.5295 | 0.5107 | | 0.0013 | 172.0 | 21328 | 3.5328 | 0.5105 | | 0.0044 | 173.0 | 21452 | 3.5402 | 0.5098 | | 0.0023 | 174.0 | 21576 | 3.5429 | 0.5120 | | 0.0021 | 175.0 | 21700 | 3.5419 | 0.5099 | | 0.0014 | 176.0 | 21824 | 3.5467 | 0.5116 | | 0.0025 | 177.0 | 21948 | 3.5475 | 0.5122 | | 0.0017 | 178.0 | 22072 | 3.5460 | 0.5117 | | 0.0015 | 179.0 | 22196 | 3.5513 | 0.5108 | | 0.0019 | 180.0 | 22320 | 3.5513 | 0.5135 | | 0.0016 | 181.0 | 22444 | 3.5539 | 0.5128 | | 0.0022 | 182.0 | 22568 | 3.5585 | 0.5131 | | 0.0013 | 183.0 | 22692 | 3.5599 | 0.5150 | | 0.0012 | 184.0 | 22816 | 3.5590 | 0.5151 | | 0.0023 | 185.0 | 22940 | 3.5587 | 0.5142 | | 0.002 | 186.0 | 23064 | 3.5601 | 0.5145 | | 0.0011 | 187.0 | 23188 | 3.5630 | 0.5133 | | 0.0019 | 188.0 | 23312 | 3.5662 | 0.5163 | | 0.0021 | 189.0 | 23436 | 3.5643 | 0.5132 | | 0.0012 | 190.0 | 23560 | 3.5684 | 0.5128 | | 0.0021 | 191.0 | 23684 | 3.5681 | 0.5138 | | 0.0019 | 192.0 | 23808 | 3.5700 | 0.5139 | | 0.0018 | 193.0 | 23932 | 3.5721 | 0.5137 | | 0.0019 | 194.0 | 24056 | 3.5742 | 0.5161 | | 0.0017 | 195.0 | 24180 | 3.5719 | 0.5118 | | 0.0016 | 196.0 | 24304 | 3.5718 | 0.5160 | | 0.0018 | 197.0 | 24428 | 3.5737 | 0.5135 | | 0.002 | 198.0 | 24552 | 3.5740 | 0.5144 | | 0.001 | 199.0 | 24676 | 3.5746 | 0.5143 | | 0.0014 | 200.0 | 24800 | 3.5744 | 0.5126 | ### Framework versions - Transformers 4.48.0.dev0 - Pytorch 2.4.1+cu121 - Datasets 3.1.0 - Tokenizers 0.21.0