Hugging Face's logo

GaetanMichelet
/
Llama-31-8B_task-1_60-samples_config-3

PEFT
TensorBoard
Safetensors
llama
alignment-handbook
trl
sft
Generated from Trainer
4-bit precision
bitsandbytes
Model card Files Metrics Community
GaetanMichelet commited on
Commit
b4e9102
•
1 Parent(s): 4a4b0f6

Model save

Browse files
Files changed (5) hide show
  1. README.md +92 -0
  2. adapter_model.safetensors +1 -1
  3. all_results.json +9 -0
  4. train_results.json +9 -0
  5. trainer_state.json +842 -0
README.md ADDED
@@ -0,0 +1,92 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
3
+ library_name: peft
4
+ license: llama3.1
5
+ tags:
6
+ - trl
7
+ - sft
8
+ - generated_from_trainer
9
+ model-index:
10
+ - name: Llama-31-8B_task-1_60-samples_config-3
11
+ results: []
12
+ ---
13
+
14
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
15
+ should probably proofread and complete it, then remove this comment. -->
16
+
17
+ # Llama-31-8B_task-1_60-samples_config-3
18
+
19
+ This model is a fine-tuned version of [meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) on an unknown dataset.
20
+ It achieves the following results on the evaluation set:
21
+ - Loss: 1.4176
22
+
23
+ ## Model description
24
+
25
+ More information needed
26
+
27
+ ## Intended uses & limitations
28
+
29
+ More information needed
30
+
31
+ ## Training and evaluation data
32
+
33
+ More information needed
34
+
35
+ ## Training procedure
36
+
37
+ ### Training hyperparameters
38
+
39
+ The following hyperparameters were used during training:
40
+ - learning_rate: 1e-05
41
+ - train_batch_size: 1
42
+ - eval_batch_size: 1
43
+ - seed: 42
44
+ - distributed_type: multi-GPU
45
+ - gradient_accumulation_steps: 8
46
+ - total_train_batch_size: 8
47
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
48
+ - lr_scheduler_type: cosine
49
+ - lr_scheduler_warmup_ratio: 0.1
50
+ - num_epochs: 150
51
+
52
+ ### Training results
53
+
54
+ | Training Loss | Epoch | Step | Validation Loss |
55
+ |:-------------:|:-------:|:----:|:---------------:|
56
+ | 2.1691 | 0.8696 | 5 | 2.0741 |
57
+ | 2.0145 | 1.9130 | 11 | 2.0674 |
58
+ | 2.0918 | 2.9565 | 17 | 2.0492 |
59
+ | 2.0838 | 4.0 | 23 | 2.0192 |
60
+ | 2.0314 | 4.8696 | 28 | 1.9792 |
61
+ | 1.9775 | 5.9130 | 34 | 1.9190 |
62
+ | 1.8873 | 6.9565 | 40 | 1.8339 |
63
+ | 1.7547 | 8.0 | 46 | 1.7314 |
64
+ | 1.6653 | 8.8696 | 51 | 1.6435 |
65
+ | 1.5709 | 9.9130 | 57 | 1.5691 |
66
+ | 1.533 | 10.9565 | 63 | 1.5254 |
67
+ | 1.4035 | 12.0 | 69 | 1.4860 |
68
+ | 1.4227 | 12.8696 | 74 | 1.4573 |
69
+ | 1.4167 | 13.9130 | 80 | 1.4216 |
70
+ | 1.3733 | 14.9565 | 86 | 1.3884 |
71
+ | 1.2917 | 16.0 | 92 | 1.3621 |
72
+ | 1.2393 | 16.8696 | 97 | 1.3432 |
73
+ | 1.1512 | 17.9130 | 103 | 1.3246 |
74
+ | 1.1361 | 18.9565 | 109 | 1.3081 |
75
+ | 1.089 | 20.0 | 115 | 1.2985 |
76
+ | 1.0272 | 20.8696 | 120 | 1.2924 |
77
+ | 1.0591 | 21.9130 | 126 | 1.2934 |
78
+ | 0.9601 | 22.9565 | 132 | 1.3023 |
79
+ | 0.9245 | 24.0 | 138 | 1.3152 |
80
+ | 0.8188 | 24.8696 | 143 | 1.3258 |
81
+ | 0.8866 | 25.9130 | 149 | 1.3491 |
82
+ | 0.7508 | 26.9565 | 155 | 1.3779 |
83
+ | 0.7961 | 28.0 | 161 | 1.4176 |
84
+
85
+
86
+ ### Framework versions
87
+
88
+ - PEFT 0.12.0
89
+ - Transformers 4.44.0
90
+ - Pytorch 2.1.2+cu121
91
+ - Datasets 2.20.0
92
+ - Tokenizers 0.19.1
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:08be3e79b957eed8b8146aaf3ba78482486fb6d6bed994ed1710a83d65c8db9b
3
  size 167832240
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9485d2bfe853aaafe6dbdcf8799f9b53a44a4255db3ba2ff210056333d90808b
3
  size 167832240
all_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 28.0,
3
+ "total_flos": 1.927928876105728e+16,
4
+ "train_loss": 1.4128068541147694,
5
+ "train_runtime": 1506.1629,
6
+ "train_samples": 46,
7
+ "train_samples_per_second": 4.581,
8
+ "train_steps_per_second": 0.498
9
+ }
train_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 28.0,
3
+ "total_flos": 1.927928876105728e+16,
4
+ "train_loss": 1.4128068541147694,
5
+ "train_runtime": 1506.1629,
6
+ "train_samples": 46,
7
+ "train_samples_per_second": 4.581,
8
+ "train_steps_per_second": 0.498
9
+ }
trainer_state.json ADDED
@@ -0,0 +1,842 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": 1.2923505306243896,
3
+ "best_model_checkpoint": "data/Llama-31-8B_task-1_60-samples_config-3/checkpoint-120",
4
+ "epoch": 28.0,
5
+ "eval_steps": 500,
6
+ "global_step": 161,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.17391304347826086,
13
+ "grad_norm": 1.9816622734069824,
14
+ "learning_rate": 1.3333333333333336e-07,
15
+ "loss": 2.073,
16
+ "step": 1
17
+ },
18
+ {
19
+ "epoch": 0.34782608695652173,
20
+ "grad_norm": 1.8311290740966797,
21
+ "learning_rate": 2.666666666666667e-07,
22
+ "loss": 2.0852,
23
+ "step": 2
24
+ },
25
+ {
26
+ "epoch": 0.6956521739130435,
27
+ "grad_norm": 1.7322129011154175,
28
+ "learning_rate": 5.333333333333335e-07,
29
+ "loss": 2.1691,
30
+ "step": 4
31
+ },
32
+ {
33
+ "epoch": 0.8695652173913043,
34
+ "eval_loss": 2.0740702152252197,
35
+ "eval_runtime": 4.8331,
36
+ "eval_samples_per_second": 2.483,
37
+ "eval_steps_per_second": 2.483,
38
+ "step": 5
39
+ },
40
+ {
41
+ "epoch": 1.0434782608695652,
42
+ "grad_norm": 1.7766757011413574,
43
+ "learning_rate": 8.000000000000001e-07,
44
+ "loss": 2.106,
45
+ "step": 6
46
+ },
47
+ {
48
+ "epoch": 1.391304347826087,
49
+ "grad_norm": 1.8310916423797607,
50
+ "learning_rate": 1.066666666666667e-06,
51
+ "loss": 2.1909,
52
+ "step": 8
53
+ },
54
+ {
55
+ "epoch": 1.7391304347826086,
56
+ "grad_norm": 1.8460414409637451,
57
+ "learning_rate": 1.3333333333333334e-06,
58
+ "loss": 2.0145,
59
+ "step": 10
60
+ },
61
+ {
62
+ "epoch": 1.9130434782608696,
63
+ "eval_loss": 2.0674192905426025,
64
+ "eval_runtime": 4.8304,
65
+ "eval_samples_per_second": 2.484,
66
+ "eval_steps_per_second": 2.484,
67
+ "step": 11
68
+ },
69
+ {
70
+ "epoch": 2.0869565217391304,
71
+ "grad_norm": 1.7287455797195435,
72
+ "learning_rate": 1.6000000000000001e-06,
73
+ "loss": 2.0286,
74
+ "step": 12
75
+ },
76
+ {
77
+ "epoch": 2.4347826086956523,
78
+ "grad_norm": 2.0367045402526855,
79
+ "learning_rate": 1.8666666666666669e-06,
80
+ "loss": 2.1431,
81
+ "step": 14
82
+ },
83
+ {
84
+ "epoch": 2.782608695652174,
85
+ "grad_norm": 1.8663474321365356,
86
+ "learning_rate": 2.133333333333334e-06,
87
+ "loss": 2.0918,
88
+ "step": 16
89
+ },
90
+ {
91
+ "epoch": 2.9565217391304346,
92
+ "eval_loss": 2.0492312908172607,
93
+ "eval_runtime": 4.8248,
94
+ "eval_samples_per_second": 2.487,
95
+ "eval_steps_per_second": 2.487,
96
+ "step": 17
97
+ },
98
+ {
99
+ "epoch": 3.130434782608696,
100
+ "grad_norm": 1.719813346862793,
101
+ "learning_rate": 2.4000000000000003e-06,
102
+ "loss": 2.0165,
103
+ "step": 18
104
+ },
105
+ {
106
+ "epoch": 3.4782608695652173,
107
+ "grad_norm": 1.7898764610290527,
108
+ "learning_rate": 2.666666666666667e-06,
109
+ "loss": 1.998,
110
+ "step": 20
111
+ },
112
+ {
113
+ "epoch": 3.8260869565217392,
114
+ "grad_norm": 1.9577224254608154,
115
+ "learning_rate": 2.9333333333333338e-06,
116
+ "loss": 2.0838,
117
+ "step": 22
118
+ },
119
+ {
120
+ "epoch": 4.0,
121
+ "eval_loss": 2.0192394256591797,
122
+ "eval_runtime": 4.8214,
123
+ "eval_samples_per_second": 2.489,
124
+ "eval_steps_per_second": 2.489,
125
+ "step": 23
126
+ },
127
+ {
128
+ "epoch": 4.173913043478261,
129
+ "grad_norm": 1.7197067737579346,
130
+ "learning_rate": 3.2000000000000003e-06,
131
+ "loss": 2.0749,
132
+ "step": 24
133
+ },
134
+ {
135
+ "epoch": 4.521739130434782,
136
+ "grad_norm": 1.730695128440857,
137
+ "learning_rate": 3.4666666666666672e-06,
138
+ "loss": 2.0657,
139
+ "step": 26
140
+ },
141
+ {
142
+ "epoch": 4.869565217391305,
143
+ "grad_norm": 1.9637165069580078,
144
+ "learning_rate": 3.7333333333333337e-06,
145
+ "loss": 2.0314,
146
+ "step": 28
147
+ },
148
+ {
149
+ "epoch": 4.869565217391305,
150
+ "eval_loss": 1.9791640043258667,
151
+ "eval_runtime": 4.8308,
152
+ "eval_samples_per_second": 2.484,
153
+ "eval_steps_per_second": 2.484,
154
+ "step": 28
155
+ },
156
+ {
157
+ "epoch": 5.217391304347826,
158
+ "grad_norm": 2.0035128593444824,
159
+ "learning_rate": 4.000000000000001e-06,
160
+ "loss": 1.9946,
161
+ "step": 30
162
+ },
163
+ {
164
+ "epoch": 5.565217391304348,
165
+ "grad_norm": 1.9658902883529663,
166
+ "learning_rate": 4.266666666666668e-06,
167
+ "loss": 1.9726,
168
+ "step": 32
169
+ },
170
+ {
171
+ "epoch": 5.913043478260869,
172
+ "grad_norm": 1.9448286294937134,
173
+ "learning_rate": 4.533333333333334e-06,
174
+ "loss": 1.9775,
175
+ "step": 34
176
+ },
177
+ {
178
+ "epoch": 5.913043478260869,
179
+ "eval_loss": 1.919035792350769,
180
+ "eval_runtime": 4.8258,
181
+ "eval_samples_per_second": 2.487,
182
+ "eval_steps_per_second": 2.487,
183
+ "step": 34
184
+ },
185
+ {
186
+ "epoch": 6.260869565217392,
187
+ "grad_norm": 1.8925299644470215,
188
+ "learning_rate": 4.800000000000001e-06,
189
+ "loss": 1.7775,
190
+ "step": 36
191
+ },
192
+ {
193
+ "epoch": 6.608695652173913,
194
+ "grad_norm": 1.940888524055481,
195
+ "learning_rate": 5.0666666666666676e-06,
196
+ "loss": 1.9153,
197
+ "step": 38
198
+ },
199
+ {
200
+ "epoch": 6.956521739130435,
201
+ "grad_norm": 1.9882278442382812,
202
+ "learning_rate": 5.333333333333334e-06,
203
+ "loss": 1.8873,
204
+ "step": 40
205
+ },
206
+ {
207
+ "epoch": 6.956521739130435,
208
+ "eval_loss": 1.8338693380355835,
209
+ "eval_runtime": 4.8237,
210
+ "eval_samples_per_second": 2.488,
211
+ "eval_steps_per_second": 2.488,
212
+ "step": 40
213
+ },
214
+ {
215
+ "epoch": 7.304347826086957,
216
+ "grad_norm": 1.6699305772781372,
217
+ "learning_rate": 5.600000000000001e-06,
218
+ "loss": 1.831,
219
+ "step": 42
220
+ },
221
+ {
222
+ "epoch": 7.6521739130434785,
223
+ "grad_norm": 1.955969214439392,
224
+ "learning_rate": 5.8666666666666675e-06,
225
+ "loss": 1.7779,
226
+ "step": 44
227
+ },
228
+ {
229
+ "epoch": 8.0,
230
+ "grad_norm": 1.5607645511627197,
231
+ "learning_rate": 6.133333333333334e-06,
232
+ "loss": 1.7547,
233
+ "step": 46
234
+ },
235
+ {
236
+ "epoch": 8.0,
237
+ "eval_loss": 1.7314118146896362,
238
+ "eval_runtime": 4.8242,
239
+ "eval_samples_per_second": 2.487,
240
+ "eval_steps_per_second": 2.487,
241
+ "step": 46
242
+ },
243
+ {
244
+ "epoch": 8.347826086956522,
245
+ "grad_norm": 1.6665287017822266,
246
+ "learning_rate": 6.4000000000000006e-06,
247
+ "loss": 1.725,
248
+ "step": 48
249
+ },
250
+ {
251
+ "epoch": 8.695652173913043,
252
+ "grad_norm": 1.44089674949646,
253
+ "learning_rate": 6.666666666666667e-06,
254
+ "loss": 1.6653,
255
+ "step": 50
256
+ },
257
+ {
258
+ "epoch": 8.869565217391305,
259
+ "eval_loss": 1.6434863805770874,
260
+ "eval_runtime": 4.8273,
261
+ "eval_samples_per_second": 2.486,
262
+ "eval_steps_per_second": 2.486,
263
+ "step": 51
264
+ },
265
+ {
266
+ "epoch": 9.043478260869565,
267
+ "grad_norm": 1.455543041229248,
268
+ "learning_rate": 6.9333333333333344e-06,
269
+ "loss": 1.6278,
270
+ "step": 52
271
+ },
272
+ {
273
+ "epoch": 9.391304347826088,
274
+ "grad_norm": 1.1698004007339478,
275
+ "learning_rate": 7.2000000000000005e-06,
276
+ "loss": 1.576,
277
+ "step": 54
278
+ },
279
+ {
280
+ "epoch": 9.73913043478261,
281
+ "grad_norm": 1.042985200881958,
282
+ "learning_rate": 7.4666666666666675e-06,
283
+ "loss": 1.5709,
284
+ "step": 56
285
+ },
286
+ {
287
+ "epoch": 9.91304347826087,
288
+ "eval_loss": 1.5690679550170898,
289
+ "eval_runtime": 4.8278,
290
+ "eval_samples_per_second": 2.486,
291
+ "eval_steps_per_second": 2.486,
292
+ "step": 57
293
+ },
294
+ {
295
+ "epoch": 10.08695652173913,
296
+ "grad_norm": 0.7936043739318848,
297
+ "learning_rate": 7.733333333333334e-06,
298
+ "loss": 1.5431,
299
+ "step": 58
300
+ },
301
+ {
302
+ "epoch": 10.434782608695652,
303
+ "grad_norm": 0.7456988096237183,
304
+ "learning_rate": 8.000000000000001e-06,
305
+ "loss": 1.4372,
306
+ "step": 60
307
+ },
308
+ {
309
+ "epoch": 10.782608695652174,
310
+ "grad_norm": 0.8631324172019958,
311
+ "learning_rate": 8.266666666666667e-06,
312
+ "loss": 1.533,
313
+ "step": 62
314
+ },
315
+ {
316
+ "epoch": 10.956521739130435,
317
+ "eval_loss": 1.5253533124923706,
318
+ "eval_runtime": 4.8305,
319
+ "eval_samples_per_second": 2.484,
320
+ "eval_steps_per_second": 2.484,
321
+ "step": 63
322
+ },
323
+ {
324
+ "epoch": 11.130434782608695,
325
+ "grad_norm": 0.7893415093421936,
326
+ "learning_rate": 8.533333333333335e-06,
327
+ "loss": 1.5894,
328
+ "step": 64
329
+ },
330
+ {
331
+ "epoch": 11.478260869565217,
332
+ "grad_norm": 0.7381595969200134,
333
+ "learning_rate": 8.8e-06,
334
+ "loss": 1.4961,
335
+ "step": 66
336
+ },
337
+ {
338
+ "epoch": 11.826086956521738,
339
+ "grad_norm": 0.6585392355918884,
340
+ "learning_rate": 9.066666666666667e-06,
341
+ "loss": 1.4035,
342
+ "step": 68
343
+ },
344
+ {
345
+ "epoch": 12.0,
346
+ "eval_loss": 1.485979676246643,
347
+ "eval_runtime": 4.824,
348
+ "eval_samples_per_second": 2.488,
349
+ "eval_steps_per_second": 2.488,
350
+ "step": 69
351
+ },
352
+ {
353
+ "epoch": 12.173913043478262,
354
+ "grad_norm": 0.639266848564148,
355
+ "learning_rate": 9.333333333333334e-06,
356
+ "loss": 1.3733,
357
+ "step": 70
358
+ },
359
+ {
360
+ "epoch": 12.521739130434783,
361
+ "grad_norm": 0.7418985962867737,
362
+ "learning_rate": 9.600000000000001e-06,
363
+ "loss": 1.4263,
364
+ "step": 72
365
+ },
366
+ {
367
+ "epoch": 12.869565217391305,
368
+ "grad_norm": 0.7750474214553833,
369
+ "learning_rate": 9.866666666666668e-06,
370
+ "loss": 1.4227,
371
+ "step": 74
372
+ },
373
+ {
374
+ "epoch": 12.869565217391305,
375
+ "eval_loss": 1.457255244255066,
376
+ "eval_runtime": 4.8296,
377
+ "eval_samples_per_second": 2.485,
378
+ "eval_steps_per_second": 2.485,
379
+ "step": 74
380
+ },
381
+ {
382
+ "epoch": 13.217391304347826,
383
+ "grad_norm": 0.7266564965248108,
384
+ "learning_rate": 9.999945845889795e-06,
385
+ "loss": 1.4963,
386
+ "step": 76
387
+ },
388
+ {
389
+ "epoch": 13.565217391304348,
390
+ "grad_norm": 0.7112168073654175,
391
+ "learning_rate": 9.999512620046523e-06,
392
+ "loss": 1.3112,
393
+ "step": 78
394
+ },
395
+ {
396
+ "epoch": 13.91304347826087,
397
+ "grad_norm": 0.6581439971923828,
398
+ "learning_rate": 9.99864620589731e-06,
399
+ "loss": 1.4167,
400
+ "step": 80
401
+ },
402
+ {
403
+ "epoch": 13.91304347826087,
404
+ "eval_loss": 1.421582818031311,
405
+ "eval_runtime": 4.8288,
406
+ "eval_samples_per_second": 2.485,
407
+ "eval_steps_per_second": 2.485,
408
+ "step": 80
409
+ },
410
+ {
411
+ "epoch": 14.26086956521739,
412
+ "grad_norm": 0.7612527012825012,
413
+ "learning_rate": 9.99734667851357e-06,
414
+ "loss": 1.2943,
415
+ "step": 82
416
+ },
417
+ {
418
+ "epoch": 14.608695652173914,
419
+ "grad_norm": 0.7063285708427429,
420
+ "learning_rate": 9.995614150494293e-06,
421
+ "loss": 1.2531,
422
+ "step": 84
423
+ },
424
+ {
425
+ "epoch": 14.956521739130435,
426
+ "grad_norm": 0.6375510096549988,
427
+ "learning_rate": 9.993448771956285e-06,
428
+ "loss": 1.3733,
429
+ "step": 86
430
+ },
431
+ {
432
+ "epoch": 14.956521739130435,
433
+ "eval_loss": 1.3883886337280273,
434
+ "eval_runtime": 4.8268,
435
+ "eval_samples_per_second": 2.486,
436
+ "eval_steps_per_second": 2.486,
437
+ "step": 86
438
+ },
439
+ {
440
+ "epoch": 15.304347826086957,
441
+ "grad_norm": 0.7057134509086609,
442
+ "learning_rate": 9.99085073052117e-06,
443
+ "loss": 1.2699,
444
+ "step": 88
445
+ },
446
+ {
447
+ "epoch": 15.652173913043478,
448
+ "grad_norm": 0.6965241432189941,
449
+ "learning_rate": 9.987820251299121e-06,
450
+ "loss": 1.2864,
451
+ "step": 90
452
+ },
453
+ {
454
+ "epoch": 16.0,
455
+ "grad_norm": 0.7538113594055176,
456
+ "learning_rate": 9.984357596869369e-06,
457
+ "loss": 1.2917,
458
+ "step": 92
459
+ },
460
+ {
461
+ "epoch": 16.0,
462
+ "eval_loss": 1.362096905708313,
463
+ "eval_runtime": 4.8216,
464
+ "eval_samples_per_second": 2.489,
465
+ "eval_steps_per_second": 2.489,
466
+ "step": 92
467
+ },
468
+ {
469
+ "epoch": 16.347826086956523,
470
+ "grad_norm": 0.7437707781791687,
471
+ "learning_rate": 9.980463067257437e-06,
472
+ "loss": 1.2624,
473
+ "step": 94
474
+ },
475
+ {
476
+ "epoch": 16.695652173913043,
477
+ "grad_norm": 0.7640065550804138,
478
+ "learning_rate": 9.976136999909156e-06,
479
+ "loss": 1.2393,
480
+ "step": 96
481
+ },
482
+ {
483
+ "epoch": 16.869565217391305,
484
+ "eval_loss": 1.3432282209396362,
485
+ "eval_runtime": 4.8287,
486
+ "eval_samples_per_second": 2.485,
487
+ "eval_steps_per_second": 2.485,
488
+ "step": 97
489
+ },
490
+ {
491
+ "epoch": 17.043478260869566,
492
+ "grad_norm": 0.8291388750076294,
493
+ "learning_rate": 9.971379769661422e-06,
494
+ "loss": 1.2362,
495
+ "step": 98
496
+ },
497
+ {
498
+ "epoch": 17.391304347826086,
499
+ "grad_norm": 0.7341519594192505,
500
+ "learning_rate": 9.966191788709716e-06,
501
+ "loss": 1.192,
502
+ "step": 100
503
+ },
504
+ {
505
+ "epoch": 17.73913043478261,
506
+ "grad_norm": 0.8155421614646912,
507
+ "learning_rate": 9.960573506572391e-06,
508
+ "loss": 1.1512,
509
+ "step": 102
510
+ },
511
+ {
512
+ "epoch": 17.91304347826087,
513
+ "eval_loss": 1.324644684791565,
514
+ "eval_runtime": 4.8257,
515
+ "eval_samples_per_second": 2.487,
516
+ "eval_steps_per_second": 2.487,
517
+ "step": 103
518
+ },
519
+ {
520
+ "epoch": 18.08695652173913,
521
+ "grad_norm": 0.8011210560798645,
522
+ "learning_rate": 9.95452541005172e-06,
523
+ "loss": 1.2282,
524
+ "step": 104
525
+ },
526
+ {
527
+ "epoch": 18.434782608695652,
528
+ "grad_norm": 0.7885610461235046,
529
+ "learning_rate": 9.948048023191728e-06,
530
+ "loss": 1.1364,
531
+ "step": 106
532
+ },
533
+ {
534
+ "epoch": 18.782608695652176,
535
+ "grad_norm": 0.7404261231422424,
536
+ "learning_rate": 9.941141907232766e-06,
537
+ "loss": 1.1361,
538
+ "step": 108
539
+ },
540
+ {
541
+ "epoch": 18.956521739130434,
542
+ "eval_loss": 1.3080695867538452,
543
+ "eval_runtime": 4.8288,
544
+ "eval_samples_per_second": 2.485,
545
+ "eval_steps_per_second": 2.485,
546
+ "step": 109
547
+ },
548
+ {
549
+ "epoch": 19.130434782608695,
550
+ "grad_norm": 0.8348779678344727,
551
+ "learning_rate": 9.933807660562898e-06,
552
+ "loss": 1.2224,
553
+ "step": 110
554
+ },
555
+ {
556
+ "epoch": 19.47826086956522,
557
+ "grad_norm": 0.8465795516967773,
558
+ "learning_rate": 9.926045918666045e-06,
559
+ "loss": 1.1334,
560
+ "step": 112
561
+ },
562
+ {
563
+ "epoch": 19.82608695652174,
564
+ "grad_norm": 0.903944194316864,
565
+ "learning_rate": 9.91785735406693e-06,
566
+ "loss": 1.089,
567
+ "step": 114
568
+ },
569
+ {
570
+ "epoch": 20.0,
571
+ "eval_loss": 1.2985278367996216,
572
+ "eval_runtime": 4.8249,
573
+ "eval_samples_per_second": 2.487,
574
+ "eval_steps_per_second": 2.487,
575
+ "step": 115
576
+ },
577
+ {
578
+ "epoch": 20.17391304347826,
579
+ "grad_norm": 0.8637501001358032,
580
+ "learning_rate": 9.909242676272797e-06,
581
+ "loss": 1.0314,
582
+ "step": 116
583
+ },
584
+ {
585
+ "epoch": 20.52173913043478,
586
+ "grad_norm": 0.9288212656974792,
587
+ "learning_rate": 9.90020263171194e-06,
588
+ "loss": 1.1512,
589
+ "step": 118
590
+ },
591
+ {
592
+ "epoch": 20.869565217391305,
593
+ "grad_norm": 0.8621946573257446,
594
+ "learning_rate": 9.890738003669029e-06,
595
+ "loss": 1.0272,
596
+ "step": 120
597
+ },
598
+ {
599
+ "epoch": 20.869565217391305,
600
+ "eval_loss": 1.2923505306243896,
601
+ "eval_runtime": 4.8264,
602
+ "eval_samples_per_second": 2.486,
603
+ "eval_steps_per_second": 2.486,
604
+ "step": 120
605
+ },
606
+ {
607
+ "epoch": 21.217391304347824,
608
+ "grad_norm": 0.9357754588127136,
609
+ "learning_rate": 9.880849612217238e-06,
610
+ "loss": 1.0331,
611
+ "step": 122
612
+ },
613
+ {
614
+ "epoch": 21.565217391304348,
615
+ "grad_norm": 0.9565621614456177,
616
+ "learning_rate": 9.870538314147194e-06,
617
+ "loss": 1.0331,
618
+ "step": 124
619
+ },
620
+ {
621
+ "epoch": 21.91304347826087,
622
+ "grad_norm": 0.9126953482627869,
623
+ "learning_rate": 9.859805002892733e-06,
624
+ "loss": 1.0591,
625
+ "step": 126
626
+ },
627
+ {
628
+ "epoch": 21.91304347826087,
629
+ "eval_loss": 1.2933709621429443,
630
+ "eval_runtime": 4.8294,
631
+ "eval_samples_per_second": 2.485,
632
+ "eval_steps_per_second": 2.485,
633
+ "step": 126
634
+ },
635
+ {
636
+ "epoch": 22.26086956521739,
637
+ "grad_norm": 0.9442431926727295,
638
+ "learning_rate": 9.84865060845349e-06,
639
+ "loss": 0.9782,
640
+ "step": 128
641
+ },
642
+ {
643
+ "epoch": 22.608695652173914,
644
+ "grad_norm": 1.0432406663894653,
645
+ "learning_rate": 9.83707609731432e-06,
646
+ "loss": 1.0154,
647
+ "step": 130
648
+ },
649
+ {
650
+ "epoch": 22.956521739130434,
651
+ "grad_norm": 1.1213728189468384,
652
+ "learning_rate": 9.825082472361558e-06,
653
+ "loss": 0.9601,
654
+ "step": 132
655
+ },
656
+ {
657
+ "epoch": 22.956521739130434,
658
+ "eval_loss": 1.3022547960281372,
659
+ "eval_runtime": 4.8277,
660
+ "eval_samples_per_second": 2.486,
661
+ "eval_steps_per_second": 2.486,
662
+ "step": 132
663
+ },
664
+ {
665
+ "epoch": 23.304347826086957,
666
+ "grad_norm": 1.1582452058792114,
667
+ "learning_rate": 9.812670772796113e-06,
668
+ "loss": 0.9306,
669
+ "step": 134
670
+ },
671
+ {
672
+ "epoch": 23.652173913043477,
673
+ "grad_norm": 1.1142038106918335,
674
+ "learning_rate": 9.799842074043438e-06,
675
+ "loss": 0.952,
676
+ "step": 136
677
+ },
678
+ {
679
+ "epoch": 24.0,
680
+ "grad_norm": 1.2423542737960815,
681
+ "learning_rate": 9.786597487660336e-06,
682
+ "loss": 0.9245,
683
+ "step": 138
684
+ },
685
+ {
686
+ "epoch": 24.0,
687
+ "eval_loss": 1.3152189254760742,
688
+ "eval_runtime": 4.8216,
689
+ "eval_samples_per_second": 2.489,
690
+ "eval_steps_per_second": 2.489,
691
+ "step": 138
692
+ },
693
+ {
694
+ "epoch": 24.347826086956523,
695
+ "grad_norm": 1.235395073890686,
696
+ "learning_rate": 9.77293816123866e-06,
697
+ "loss": 0.9178,
698
+ "step": 140
699
+ },
700
+ {
701
+ "epoch": 24.695652173913043,
702
+ "grad_norm": 1.0877219438552856,
703
+ "learning_rate": 9.75886527830587e-06,
704
+ "loss": 0.8188,
705
+ "step": 142
706
+ },
707
+ {
708
+ "epoch": 24.869565217391305,
709
+ "eval_loss": 1.325753092765808,
710
+ "eval_runtime": 4.8235,
711
+ "eval_samples_per_second": 2.488,
712
+ "eval_steps_per_second": 2.488,
713
+ "step": 143
714
+ },
715
+ {
716
+ "epoch": 25.043478260869566,
717
+ "grad_norm": 1.2893208265304565,
718
+ "learning_rate": 9.744380058222483e-06,
719
+ "loss": 0.9219,
720
+ "step": 144
721
+ },
722
+ {
723
+ "epoch": 25.391304347826086,
724
+ "grad_norm": 1.412602186203003,
725
+ "learning_rate": 9.729483756076436e-06,
726
+ "loss": 0.8844,
727
+ "step": 146
728
+ },
729
+ {
730
+ "epoch": 25.73913043478261,
731
+ "grad_norm": 1.337964415550232,
732
+ "learning_rate": 9.714177662574316e-06,
733
+ "loss": 0.8866,
734
+ "step": 148
735
+ },
736
+ {
737
+ "epoch": 25.91304347826087,
738
+ "eval_loss": 1.3491390943527222,
739
+ "eval_runtime": 4.8253,
740
+ "eval_samples_per_second": 2.487,
741
+ "eval_steps_per_second": 2.487,
742
+ "step": 149
743
+ },
744
+ {
745
+ "epoch": 26.08695652173913,
746
+ "grad_norm": 1.539898157119751,
747
+ "learning_rate": 9.698463103929542e-06,
748
+ "loss": 0.7192,
749
+ "step": 150
750
+ },
751
+ {
752
+ "epoch": 26.434782608695652,
753
+ "grad_norm": 1.4601242542266846,
754
+ "learning_rate": 9.682341441747446e-06,
755
+ "loss": 0.733,
756
+ "step": 152
757
+ },
758
+ {
759
+ "epoch": 26.782608695652176,
760
+ "grad_norm": 1.4517402648925781,
761
+ "learning_rate": 9.665814072907293e-06,
762
+ "loss": 0.7508,
763
+ "step": 154
764
+ },
765
+ {
766
+ "epoch": 26.956521739130434,
767
+ "eval_loss": 1.377853512763977,
768
+ "eval_runtime": 4.8238,
769
+ "eval_samples_per_second": 2.488,
770
+ "eval_steps_per_second": 2.488,
771
+ "step": 155
772
+ },
773
+ {
774
+ "epoch": 27.130434782608695,
775
+ "grad_norm": 1.503257393836975,
776
+ "learning_rate": 9.648882429441258e-06,
777
+ "loss": 0.784,
778
+ "step": 156
779
+ },
780
+ {
781
+ "epoch": 27.47826086956522,
782
+ "grad_norm": 1.569117546081543,
783
+ "learning_rate": 9.63154797841033e-06,
784
+ "loss": 0.7133,
785
+ "step": 158
786
+ },
787
+ {
788
+ "epoch": 27.82608695652174,
789
+ "grad_norm": 2.004321575164795,
790
+ "learning_rate": 9.613812221777212e-06,
791
+ "loss": 0.7961,
792
+ "step": 160
793
+ },
794
+ {
795
+ "epoch": 28.0,
796
+ "eval_loss": 1.4175916910171509,
797
+ "eval_runtime": 4.8249,
798
+ "eval_samples_per_second": 2.487,
799
+ "eval_steps_per_second": 2.487,
800
+ "step": 161
801
+ },
802
+ {
803
+ "epoch": 28.0,
804
+ "step": 161,
805
+ "total_flos": 1.927928876105728e+16,
806
+ "train_loss": 1.4128068541147694,
807
+ "train_runtime": 1506.1629,
808
+ "train_samples_per_second": 4.581,
809
+ "train_steps_per_second": 0.498
810
+ }
811
+ ],
812
+ "logging_steps": 2,
813
+ "max_steps": 750,
814
+ "num_input_tokens_seen": 0,
815
+ "num_train_epochs": 150,
816
+ "save_steps": 25,
817
+ "stateful_callbacks": {
818
+ "EarlyStoppingCallback": {
819
+ "args": {
820
+ "early_stopping_patience": 7,
821
+ "early_stopping_threshold": 0.0
822
+ },
823
+ "attributes": {
824
+ "early_stopping_patience_counter": 0
825
+ }
826
+ },
827
+ "TrainerControl": {
828
+ "args": {
829
+ "should_epoch_stop": false,
830
+ "should_evaluate": false,
831
+ "should_log": false,
832
+ "should_save": true,
833
+ "should_training_stop": true
834
+ },
835
+ "attributes": {}
836
+ }
837
+ },
838
+ "total_flos": 1.927928876105728e+16,
839
+ "train_batch_size": 1,
840
+ "trial_name": null,
841
+ "trial_params": null
842
+ }