File size: 47,661 Bytes
f291f93
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
2021-07-14 20:31:18,076 INFO    MainThread:569706 [internal.py:wandb_internal():88] W&B internal server running at pid: 569706, started at: 2021-07-14 20:31:18.075968
2021-07-14 20:31:18,078 INFO    WriterThread:569706 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/run-3emij6kq.wandb
2021-07-14 20:31:18,078 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 20:31:18,079 DEBUG   SenderThread:569706 [sender.py:send():179] send: header
2021-07-14 20:31:18,079 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: check_version
2021-07-14 20:31:18,118 DEBUG   SenderThread:569706 [sender.py:send():179] send: run
2021-07-14 20:31:18,293 INFO    SenderThread:569706 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files
2021-07-14 20:31:18,293 INFO    SenderThread:569706 [sender.py:_start_run_threads():716] run started: 3emij6kq with start time 1626294677
2021-07-14 20:31:18,293 DEBUG   SenderThread:569706 [sender.py:send():179] send: summary
2021-07-14 20:31:18,293 INFO    SenderThread:569706 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:31:18,293 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 20:31:19,296 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/wandb-summary.json
2021-07-14 20:31:19,373 DEBUG   HandlerThread:569706 [meta.py:__init__():39] meta init
2021-07-14 20:31:19,373 DEBUG   HandlerThread:569706 [meta.py:__init__():53] meta init done
2021-07-14 20:31:19,373 DEBUG   HandlerThread:569706 [meta.py:probe():210] probe
2021-07-14 20:31:19,374 DEBUG   HandlerThread:569706 [meta.py:_setup_git():200] setup git
2021-07-14 20:31:19,403 DEBUG   HandlerThread:569706 [meta.py:_setup_git():207] setup git done
2021-07-14 20:31:19,403 DEBUG   HandlerThread:569706 [meta.py:_save_pip():57] save pip
2021-07-14 20:31:19,404 DEBUG   HandlerThread:569706 [meta.py:_save_pip():71] save pip done
2021-07-14 20:31:19,404 DEBUG   HandlerThread:569706 [meta.py:probe():252] probe done
2021-07-14 20:31:19,407 DEBUG   SenderThread:569706 [sender.py:send():179] send: files
2021-07-14 20:31:19,407 INFO    SenderThread:569706 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 20:31:19,413 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:31:19,413 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:31:19,541 DEBUG   SenderThread:569706 [sender.py:send():179] send: config
2021-07-14 20:31:19,541 DEBUG   SenderThread:569706 [sender.py:send():179] send: config
2021-07-14 20:31:19,542 DEBUG   SenderThread:569706 [sender.py:send():179] send: config
2021-07-14 20:31:19,853 INFO    Thread-11 :569706 [upload_job.py:push():137] Uploaded file /tmp/tmprctrqfj8wandb/151zr105-wandb-metadata.json
2021-07-14 20:31:20,294 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/requirements.txt
2021-07-14 20:31:20,295 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:31:20,295 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/wandb-metadata.json
2021-07-14 20:31:34,300 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:31:34,543 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:31:34,543 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:31:36,300 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:31:40,302 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:31:47,455 DEBUG   SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:31:49,306 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/config.yaml
2021-07-14 20:31:49,691 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:31:49,692 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:32:04,839 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:32:04,839 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:32:17,538 DEBUG   SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:32:19,973 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:32:19,973 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:32:35,105 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:32:35,105 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:32:47,616 DEBUG   SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:32:50,237 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:32:50,237 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:33:05,367 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:33:05,367 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:33:17,693 DEBUG   SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:33:18,340 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:20,341 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:20,549 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:33:20,549 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:33:22,342 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:24,343 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:26,344 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:28,345 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:30,346 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:32,346 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:34,347 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:35,690 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:33:35,691 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:33:36,348 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:38,349 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:40,350 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:42,351 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:44,352 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:46,353 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:47,771 DEBUG   SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:33:48,354 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:50,355 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:50,833 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:33:50,833 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:33:52,356 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:54,356 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:56,357 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:58,358 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:00,359 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:02,360 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:04,361 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:05,963 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:34:05,963 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:34:06,362 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:08,363 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:10,364 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:12,365 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:14,366 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:16,367 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:17,848 DEBUG   SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:34:18,368 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:20,369 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:21,092 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:34:21,093 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:34:22,369 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:24,370 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:26,371 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:28,372 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:30,373 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:32,374 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:34,375 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:36,254 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:34:36,254 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:34:36,376 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:38,377 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:40,377 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:42,379 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:44,380 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:46,381 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:47,925 DEBUG   SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:34:48,381 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:50,382 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:51,390 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:34:51,390 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:34:52,383 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:54,384 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:56,385 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:58,386 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:00,386 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:02,387 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:04,388 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:06,389 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:06,526 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:35:06,527 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:35:08,390 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:10,391 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:12,391 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:14,392 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:16,393 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:18,004 DEBUG   SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:35:18,394 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:20,395 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:21,675 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:35:21,675 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:35:22,396 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:24,397 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:26,398 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:28,398 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:30,399 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:32,400 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:34,401 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:36,402 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:36,809 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:35:36,810 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:35:39,403 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:41,404 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:43,405 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:45,406 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:47,407 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:48,079 DEBUG   SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:35:49,408 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:51,409 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:51,945 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:35:51,945 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:35:53,410 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:07,092 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:36:07,093 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:36:13,798 DEBUG   SenderThread:569706 [sender.py:send():179] send: history
2021-07-14 20:36:13,798 DEBUG   SenderThread:569706 [sender.py:send():179] send: summary
2021-07-14 20:36:13,800 INFO    SenderThread:569706 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:36:14,418 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:14,418 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/wandb-summary.json
2021-07-14 20:36:15,419 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:17,419 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:18,159 DEBUG   SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:36:19,420 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:21,421 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:22,223 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:36:22,223 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:36:23,422 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:25,423 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:27,424 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:29,425 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:31,426 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:33,427 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:35,428 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:37,354 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:36:37,354 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:36:37,429 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:39,430 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:41,431 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:43,431 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:45,432 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:47,433 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:48,232 DEBUG   SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:36:49,434 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:51,435 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:52,488 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:36:52,488 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:36:53,436 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:55,437 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:57,438 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:59,439 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:01,440 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:03,441 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:05,442 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:07,443 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:07,644 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:37:07,644 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:37:09,444 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:11,445 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:13,446 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:15,447 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:17,447 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:18,309 DEBUG   SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:37:19,448 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:21,449 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:22,825 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:37:22,825 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:37:23,450 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:25,451 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:27,452 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:29,453 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:31,454 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:33,455 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:35,456 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:37,457 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:37,956 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:37:37,956 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:37:39,458 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:41,459 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:43,460 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:45,461 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:47,462 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:48,387 DEBUG   SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:37:49,463 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:51,463 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:53,090 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:37:53,091 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:37:53,464 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:55,465 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:57,466 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:59,467 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:01,468 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:03,469 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:05,470 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:07,470 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:08,227 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:38:08,228 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:38:09,471 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:11,472 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:13,473 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:15,474 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:17,474 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:18,464 DEBUG   SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:38:19,475 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:21,476 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:23,361 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:38:23,362 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:38:23,477 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:25,478 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:27,479 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:29,480 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:31,481 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:33,482 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:35,483 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:37,484 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:38,492 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:38:38,492 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:38:39,485 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:41,486 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:43,487 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:45,488 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:47,488 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:48,541 DEBUG   SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:38:53,658 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:38:53,658 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:39:08,787 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:39:08,788 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:39:09,733 DEBUG   SenderThread:569706 [sender.py:send():179] send: history
2021-07-14 20:39:09,733 DEBUG   SenderThread:569706 [sender.py:send():179] send: summary
2021-07-14 20:39:09,734 INFO    SenderThread:569706 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:39:10,498 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/wandb-summary.json
2021-07-14 20:39:11,498 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:39:13,499 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:39:15,500 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:39:17,501 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:39:18,620 DEBUG   SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:39:19,502 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:39:21,503 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:39:23,504 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:39:23,928 DEBUG   HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:39:23,928 DEBUG   SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:39:25,505 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:39:27,119 WARNING MainThread:569706 [internal.py:wandb_internal():147] Internal process interrupt: 1
2021-07-14 20:39:27,506 INFO    Thread-8  :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:39:27,538 WARNING MainThread:569706 [internal.py:wandb_internal():147] Internal process interrupt: 2
2021-07-14 20:39:27,539 ERROR   MainThread:569706 [internal.py:wandb_internal():150] Internal process interrupted.
2021-07-14 20:39:28,203 INFO    SenderThread:569706 [sender.py:finish():945] shutting down sender
2021-07-14 20:39:28,203 INFO    WriterThread:569706 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/run-3emij6kq.wandb
2021-07-14 20:39:28,204 INFO    SenderThread:569706 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 20:39:28,204 INFO    HandlerThread:569706 [handler.py:finish():638] shutting down handler
2021-07-14 20:39:28,507 INFO    SenderThread:569706 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files
2021-07-14 20:39:28,507 INFO    SenderThread:569706 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/requirements.txt requirements.txt
2021-07-14 20:39:28,507 INFO    SenderThread:569706 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log output.log
2021-07-14 20:39:28,507 INFO    SenderThread:569706 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/wandb-metadata.json wandb-metadata.json
2021-07-14 20:39:28,507 INFO    SenderThread:569706 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/config.yaml config.yaml
2021-07-14 20:39:28,508 INFO    SenderThread:569706 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/wandb-summary.json wandb-summary.json
2021-07-14 20:39:28,508 INFO    SenderThread:569706 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 20:39:28,508 INFO    SenderThread:569706 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 20:39:28,823 INFO    MainThread:569706 [internal.py:handle_exit():78] Internal process exited