Hugging Face's logo

nlpguy
/
smolchess

Text Generation
Transformers
Safetensors
llama
Generated from Trainer
text-generation-inference
Inference Endpoints
Model card Files Community
nlpguy commited on
Commit
aeab8b4
•
1 Parent(s): 3637db9

End of training

Browse files
Files changed (10) hide show
  1. README.md +156 -0
  2. config.json +32 -0
  3. generation_config.json +6 -0
  4. merges.txt +0 -0
  5. model.safetensors +3 -0
  6. special_tokens_map.json +43 -0
  7. tokenizer.json +0 -0
  8. tokenizer_config.json +168 -0
  9. training_args.bin +3 -0
  10. vocab.json +0 -0
README.md ADDED
@@ -0,0 +1,156 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ license: apache-2.0
4
+ base_model: HuggingFaceTB/SmolLM2-135M
5
+ tags:
6
+ - generated_from_trainer
7
+ model-index:
8
+ - name: smolchess
9
+ results: []
10
+ ---
11
+
12
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
13
+ should probably proofread and complete it, then remove this comment. -->
14
+
15
+ # smolchess
16
+
17
+ This model is a fine-tuned version of [HuggingFaceTB/SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M) on an unknown dataset.
18
+ It achieves the following results on the evaluation set:
19
+ - Loss: 0.8688
20
+
21
+ ## Model description
22
+
23
+ More information needed
24
+
25
+ ## Intended uses & limitations
26
+
27
+ More information needed
28
+
29
+ ## Training and evaluation data
30
+
31
+ More information needed
32
+
33
+ ## Training procedure
34
+
35
+ ### Training hyperparameters
36
+
37
+ The following hyperparameters were used during training:
38
+ - learning_rate: 3e-05
39
+ - train_batch_size: 8
40
+ - eval_batch_size: 8
41
+ - seed: 42
42
+ - optimizer: Use grokadamw with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
43
+ - lr_scheduler_type: cosine
44
+ - num_epochs: 0.25
45
+
46
+ ### Training results
47
+
48
+ | Training Loss | Epoch | Step | Validation Loss |
49
+ |:-------------:|:------:|:----:|:---------------:|
50
+ | 1.4847 | 0.0025 | 4 | 1.3890 |
51
+ | 1.2333 | 0.0050 | 8 | 1.2242 |
52
+ | 1.2154 | 0.0075 | 12 | 1.1705 |
53
+ | 1.1268 | 0.0100 | 16 | 1.1241 |
54
+ | 1.0556 | 0.0125 | 20 | 1.1055 |
55
+ | 1.0629 | 0.0150 | 24 | 1.0848 |
56
+ | 1.1023 | 0.0176 | 28 | 1.0764 |
57
+ | 1.102 | 0.0201 | 32 | 1.0554 |
58
+ | 1.0798 | 0.0226 | 36 | 1.0567 |
59
+ | 0.9436 | 0.0251 | 40 | 1.0365 |
60
+ | 1.0524 | 0.0276 | 44 | 1.0275 |
61
+ | 1.1201 | 0.0301 | 48 | 1.0198 |
62
+ | 1.0565 | 0.0326 | 52 | 1.0135 |
63
+ | 0.9082 | 0.0351 | 56 | 1.0084 |
64
+ | 1.0544 | 0.0376 | 60 | 0.9970 |
65
+ | 1.0034 | 0.0401 | 64 | 0.9939 |
66
+ | 0.8859 | 0.0426 | 68 | 0.9852 |
67
+ | 1.018 | 0.0451 | 72 | 0.9816 |
68
+ | 0.8901 | 0.0476 | 76 | 0.9761 |
69
+ | 0.8943 | 0.0502 | 80 | 0.9723 |
70
+ | 1.0486 | 0.0527 | 84 | 0.9718 |
71
+ | 1.0102 | 0.0552 | 88 | 0.9680 |
72
+ | 0.9617 | 0.0577 | 92 | 0.9602 |
73
+ | 0.9879 | 0.0602 | 96 | 0.9607 |
74
+ | 0.9482 | 0.0627 | 100 | 0.9523 |
75
+ | 1.0265 | 0.0652 | 104 | 0.9518 |
76
+ | 0.8865 | 0.0677 | 108 | 0.9493 |
77
+ | 1.0046 | 0.0702 | 112 | 0.9448 |
78
+ | 0.9593 | 0.0727 | 116 | 0.9384 |
79
+ | 1.0167 | 0.0752 | 120 | 0.9377 |
80
+ | 0.9041 | 0.0777 | 124 | 0.9345 |
81
+ | 0.8702 | 0.0803 | 128 | 0.9311 |
82
+ | 0.9117 | 0.0828 | 132 | 0.9333 |
83
+ | 0.936 | 0.0853 | 136 | 0.9262 |
84
+ | 0.9341 | 0.0878 | 140 | 0.9237 |
85
+ | 0.913 | 0.0903 | 144 | 0.9219 |
86
+ | 0.9205 | 0.0928 | 148 | 0.9204 |
87
+ | 0.9081 | 0.0953 | 152 | 0.9183 |
88
+ | 0.8826 | 0.0978 | 156 | 0.9162 |
89
+ | 0.9578 | 0.1003 | 160 | 0.9142 |
90
+ | 0.845 | 0.1028 | 164 | 0.9128 |
91
+ | 0.9254 | 0.1053 | 168 | 0.9102 |
92
+ | 0.9622 | 0.1078 | 172 | 0.9096 |
93
+ | 0.7854 | 0.1103 | 176 | 0.9085 |
94
+ | 0.9143 | 0.1129 | 180 | 0.9071 |
95
+ | 0.99 | 0.1154 | 184 | 0.9043 |
96
+ | 0.9855 | 0.1179 | 188 | 0.9038 |
97
+ | 0.9745 | 0.1204 | 192 | 0.9017 |
98
+ | 0.9532 | 0.1229 | 196 | 0.8998 |
99
+ | 0.9464 | 0.1254 | 200 | 0.8989 |
100
+ | 0.8713 | 0.1279 | 204 | 0.8962 |
101
+ | 0.8501 | 0.1304 | 208 | 0.8942 |
102
+ | 0.9065 | 0.1329 | 212 | 0.8936 |
103
+ | 0.8949 | 0.1354 | 216 | 0.8924 |
104
+ | 0.9504 | 0.1379 | 220 | 0.8900 |
105
+ | 0.9059 | 0.1404 | 224 | 0.8900 |
106
+ | 0.909 | 0.1429 | 228 | 0.8881 |
107
+ | 0.9684 | 0.1455 | 232 | 0.8864 |
108
+ | 0.968 | 0.1480 | 236 | 0.8865 |
109
+ | 0.9436 | 0.1505 | 240 | 0.8853 |
110
+ | 0.9166 | 0.1530 | 244 | 0.8841 |
111
+ | 0.977 | 0.1555 | 248 | 0.8825 |
112
+ | 0.9011 | 0.1580 | 252 | 0.8820 |
113
+ | 0.8842 | 0.1605 | 256 | 0.8812 |
114
+ | 0.9399 | 0.1630 | 260 | 0.8806 |
115
+ | 0.9211 | 0.1655 | 264 | 0.8791 |
116
+ | 0.8043 | 0.1680 | 268 | 0.8785 |
117
+ | 0.8406 | 0.1705 | 272 | 0.8778 |
118
+ | 0.8463 | 0.1730 | 276 | 0.8765 |
119
+ | 0.8638 | 0.1755 | 280 | 0.8762 |
120
+ | 0.894 | 0.1781 | 284 | 0.8761 |
121
+ | 0.8925 | 0.1806 | 288 | 0.8753 |
122
+ | 0.9029 | 0.1831 | 292 | 0.8754 |
123
+ | 0.809 | 0.1856 | 296 | 0.8749 |
124
+ | 0.9558 | 0.1881 | 300 | 0.8742 |
125
+ | 0.8286 | 0.1906 | 304 | 0.8736 |
126
+ | 0.8714 | 0.1931 | 308 | 0.8730 |
127
+ | 0.8562 | 0.1956 | 312 | 0.8728 |
128
+ | 0.858 | 0.1981 | 316 | 0.8723 |
129
+ | 0.9027 | 0.2006 | 320 | 0.8719 |
130
+ | 0.9023 | 0.2031 | 324 | 0.8716 |
131
+ | 0.856 | 0.2056 | 328 | 0.8712 |
132
+ | 0.8455 | 0.2082 | 332 | 0.8709 |
133
+ | 0.8886 | 0.2107 | 336 | 0.8705 |
134
+ | 0.8717 | 0.2132 | 340 | 0.8703 |
135
+ | 0.9145 | 0.2157 | 344 | 0.8700 |
136
+ | 0.9618 | 0.2182 | 348 | 0.8698 |
137
+ | 0.9083 | 0.2207 | 352 | 0.8697 |
138
+ | 0.9448 | 0.2232 | 356 | 0.8695 |
139
+ | 0.9188 | 0.2257 | 360 | 0.8693 |
140
+ | 0.8006 | 0.2282 | 364 | 0.8692 |
141
+ | 0.8222 | 0.2307 | 368 | 0.8691 |
142
+ | 0.8936 | 0.2332 | 372 | 0.8690 |
143
+ | 0.9366 | 0.2357 | 376 | 0.8689 |
144
+ | 0.9336 | 0.2382 | 380 | 0.8689 |
145
+ | 0.6878 | 0.2408 | 384 | 0.8689 |
146
+ | 0.9405 | 0.2433 | 388 | 0.8688 |
147
+ | 0.9022 | 0.2458 | 392 | 0.8688 |
148
+ | 0.8499 | 0.2483 | 396 | 0.8688 |
149
+
150
+
151
+ ### Framework versions
152
+
153
+ - Transformers 4.46.1
154
+ - Pytorch 2.4.1+cu121
155
+ - Datasets 3.1.0
156
+ - Tokenizers 0.20.1
config.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "HuggingFaceTB/SmolLM2-135M",
3
+ "architectures": [
4
+ "LlamaForCausalLM"
5
+ ],
6
+ "attention_bias": false,
7
+ "attention_dropout": 0.0,
8
+ "bos_token_id": 0,
9
+ "eos_token_id": 0,
10
+ "head_dim": 64,
11
+ "hidden_act": "silu",
12
+ "hidden_size": 576,
13
+ "initializer_range": 0.041666666666666664,
14
+ "intermediate_size": 1536,
15
+ "is_llama_config": true,
16
+ "max_position_embeddings": 8192,
17
+ "mlp_bias": false,
18
+ "model_type": "llama",
19
+ "num_attention_heads": 9,
20
+ "num_hidden_layers": 30,
21
+ "num_key_value_heads": 3,
22
+ "pretraining_tp": 1,
23
+ "rms_norm_eps": 1e-05,
24
+ "rope_interleaved": false,
25
+ "rope_scaling": null,
26
+ "rope_theta": 100000,
27
+ "tie_word_embeddings": true,
28
+ "torch_dtype": "float32",
29
+ "transformers_version": "4.46.1",
30
+ "use_cache": true,
31
+ "vocab_size": 49152
32
+ }
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 0,
4
+ "eos_token_id": 0,
5
+ "transformers_version": "4.46.1"
6
+ }
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:53da225746815e2e28f8d69433eb05612ff96aef8437b42655d0aa582c57d8b4
3
+ size 538090408
special_tokens_map.json ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<|endoftext|>",
4
+ "<|im_start|>",
5
+ "<|im_end|>",
6
+ "<repo_name>",
7
+ "<reponame>",
8
+ "<file_sep>",
9
+ "<filename>",
10
+ "<gh_stars>",
11
+ "<issue_start>",
12
+ "<issue_comment>",
13
+ "<issue_closed>",
14
+ "<jupyter_start>",
15
+ "<jupyter_text>",
16
+ "<jupyter_code>",
17
+ "<jupyter_output>",
18
+ "<jupyter_script>",
19
+ "<empty_output>"
20
+ ],
21
+ "bos_token": {
22
+ "content": "<|endoftext|>",
23
+ "lstrip": false,
24
+ "normalized": false,
25
+ "rstrip": false,
26
+ "single_word": false
27
+ },
28
+ "eos_token": {
29
+ "content": "<|endoftext|>",
30
+ "lstrip": false,
31
+ "normalized": false,
32
+ "rstrip": false,
33
+ "single_word": false
34
+ },
35
+ "pad_token": "<|endoftext|>",
36
+ "unk_token": {
37
+ "content": "<|endoftext|>",
38
+ "lstrip": false,
39
+ "normalized": false,
40
+ "rstrip": false,
41
+ "single_word": false
42
+ }
43
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,168 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "added_tokens_decoder": {
4
+ "0": {
5
+ "content": "<|endoftext|>",
6
+ "lstrip": false,
7
+ "normalized": false,
8
+ "rstrip": false,
9
+ "single_word": false,
10
+ "special": true
11
+ },
12
+ "1": {
13
+ "content": "<|im_start|>",
14
+ "lstrip": false,
15
+ "normalized": false,
16
+ "rstrip": false,
17
+ "single_word": false,
18
+ "special": true
19
+ },
20
+ "2": {
21
+ "content": "<|im_end|>",
22
+ "lstrip": false,
23
+ "normalized": false,
24
+ "rstrip": false,
25
+ "single_word": false,
26
+ "special": true
27
+ },
28
+ "3": {
29
+ "content": "<repo_name>",
30
+ "lstrip": false,
31
+ "normalized": false,
32
+ "rstrip": false,
33
+ "single_word": false,
34
+ "special": true
35
+ },
36
+ "4": {
37
+ "content": "<reponame>",
38
+ "lstrip": false,
39
+ "normalized": false,
40
+ "rstrip": false,
41
+ "single_word": false,
42
+ "special": true
43
+ },
44
+ "5": {
45
+ "content": "<file_sep>",
46
+ "lstrip": false,
47
+ "normalized": false,
48
+ "rstrip": false,
49
+ "single_word": false,
50
+ "special": true
51
+ },
52
+ "6": {
53
+ "content": "<filename>",
54
+ "lstrip": false,
55
+ "normalized": false,
56
+ "rstrip": false,
57
+ "single_word": false,
58
+ "special": true
59
+ },
60
+ "7": {
61
+ "content": "<gh_stars>",
62
+ "lstrip": false,
63
+ "normalized": false,
64
+ "rstrip": false,
65
+ "single_word": false,
66
+ "special": true
67
+ },
68
+ "8": {
69
+ "content": "<issue_start>",
70
+ "lstrip": false,
71
+ "normalized": false,
72
+ "rstrip": false,
73
+ "single_word": false,
74
+ "special": true
75
+ },
76
+ "9": {
77
+ "content": "<issue_comment>",
78
+ "lstrip": false,
79
+ "normalized": false,
80
+ "rstrip": false,
81
+ "single_word": false,
82
+ "special": true
83
+ },
84
+ "10": {
85
+ "content": "<issue_closed>",
86
+ "lstrip": false,
87
+ "normalized": false,
88
+ "rstrip": false,
89
+ "single_word": false,
90
+ "special": true
91
+ },
92
+ "11": {
93
+ "content": "<jupyter_start>",
94
+ "lstrip": false,
95
+ "normalized": false,
96
+ "rstrip": false,
97
+ "single_word": false,
98
+ "special": true
99
+ },
100
+ "12": {
101
+ "content": "<jupyter_text>",
102
+ "lstrip": false,
103
+ "normalized": false,
104
+ "rstrip": false,
105
+ "single_word": false,
106
+ "special": true
107
+ },
108
+ "13": {
109
+ "content": "<jupyter_code>",
110
+ "lstrip": false,
111
+ "normalized": false,
112
+ "rstrip": false,
113
+ "single_word": false,
114
+ "special": true
115
+ },
116
+ "14": {
117
+ "content": "<jupyter_output>",
118
+ "lstrip": false,
119
+ "normalized": false,
120
+ "rstrip": false,
121
+ "single_word": false,
122
+ "special": true
123
+ },
124
+ "15": {
125
+ "content": "<jupyter_script>",
126
+ "lstrip": false,
127
+ "normalized": false,
128
+ "rstrip": false,
129
+ "single_word": false,
130
+ "special": true
131
+ },
132
+ "16": {
133
+ "content": "<empty_output>",
134
+ "lstrip": false,
135
+ "normalized": false,
136
+ "rstrip": false,
137
+ "single_word": false,
138
+ "special": true
139
+ }
140
+ },
141
+ "additional_special_tokens": [
142
+ "<|endoftext|>",
143
+ "<|im_start|>",
144
+ "<|im_end|>",
145
+ "<repo_name>",
146
+ "<reponame>",
147
+ "<file_sep>",
148
+ "<filename>",
149
+ "<gh_stars>",
150
+ "<issue_start>",
151
+ "<issue_comment>",
152
+ "<issue_closed>",
153
+ "<jupyter_start>",
154
+ "<jupyter_text>",
155
+ "<jupyter_code>",
156
+ "<jupyter_output>",
157
+ "<jupyter_script>",
158
+ "<empty_output>"
159
+ ],
160
+ "bos_token": "<|endoftext|>",
161
+ "clean_up_tokenization_spaces": false,
162
+ "eos_token": "<|endoftext|>",
163
+ "model_max_length": 8192,
164
+ "pad_token": "<|endoftext|>",
165
+ "tokenizer_class": "GPT2Tokenizer",
166
+ "unk_token": "<|endoftext|>",
167
+ "vocab_size": 49152
168
+ }
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a05a181a43009fa19ad7835d4ed918d9714a5dbf93ec6b2f11a63d1bfb1ca083
3
+ size 5176
vocab.json ADDED
The diff for this file is too large to render. See raw diff