File size: 32,630 Bytes
f291f93
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
2021-07-14 18:55:37,982 INFO    MainThread:555087 [internal.py:wandb_internal():88] W&B internal server running at pid: 555087, started at: 2021-07-14 18:55:37.982262
2021-07-14 18:55:37,984 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 18:55:37,985 INFO    WriterThread:555087 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/run-379h21h2.wandb
2021-07-14 18:55:37,986 DEBUG   SenderThread:555087 [sender.py:send():179] send: header
2021-07-14 18:55:37,986 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: check_version
2021-07-14 18:55:38,025 DEBUG   SenderThread:555087 [sender.py:send():179] send: run
2021-07-14 18:55:38,206 INFO    SenderThread:555087 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files
2021-07-14 18:55:38,206 INFO    SenderThread:555087 [sender.py:_start_run_threads():716] run started: 379h21h2 with start time 1626288937
2021-07-14 18:55:38,206 DEBUG   SenderThread:555087 [sender.py:send():179] send: summary
2021-07-14 18:55:38,206 INFO    SenderThread:555087 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 18:55:38,207 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 18:55:39,207 INFO    Thread-8  :555087 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/wandb-summary.json
2021-07-14 18:55:39,338 DEBUG   HandlerThread:555087 [meta.py:__init__():39] meta init
2021-07-14 18:55:39,338 DEBUG   HandlerThread:555087 [meta.py:__init__():53] meta init done
2021-07-14 18:55:39,338 DEBUG   HandlerThread:555087 [meta.py:probe():210] probe
2021-07-14 18:55:39,340 DEBUG   HandlerThread:555087 [meta.py:_setup_git():200] setup git
2021-07-14 18:55:39,371 DEBUG   HandlerThread:555087 [meta.py:_setup_git():207] setup git done
2021-07-14 18:55:39,371 DEBUG   HandlerThread:555087 [meta.py:_save_pip():57] save pip
2021-07-14 18:55:39,372 DEBUG   HandlerThread:555087 [meta.py:_save_pip():71] save pip done
2021-07-14 18:55:39,372 DEBUG   HandlerThread:555087 [meta.py:probe():252] probe done
2021-07-14 18:55:39,397 DEBUG   SenderThread:555087 [sender.py:send():179] send: files
2021-07-14 18:55:39,397 INFO    SenderThread:555087 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 18:55:39,404 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:55:39,405 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:55:39,533 DEBUG   SenderThread:555087 [sender.py:send():179] send: config
2021-07-14 18:55:39,534 DEBUG   SenderThread:555087 [sender.py:send():179] send: config
2021-07-14 18:55:39,534 DEBUG   SenderThread:555087 [sender.py:send():179] send: config
2021-07-14 18:55:39,858 INFO    Thread-11 :555087 [upload_job.py:push():137] Uploaded file /tmp/tmpywmfy6u4wandb/302tickz-wandb-metadata.json
2021-07-14 18:55:40,207 INFO    Thread-8  :555087 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/output.log
2021-07-14 18:55:40,207 INFO    Thread-8  :555087 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/wandb-metadata.json
2021-07-14 18:55:40,207 INFO    Thread-8  :555087 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/requirements.txt
2021-07-14 18:55:54,543 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:55:54,543 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:55:58,216 INFO    Thread-8  :555087 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/output.log
2021-07-14 18:56:07,426 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 18:56:09,220 INFO    Thread-8  :555087 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/config.yaml
2021-07-14 18:56:09,675 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:56:09,675 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:56:24,806 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:56:24,806 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:56:37,502 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 18:56:39,942 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:56:39,942 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:56:55,073 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:56:55,073 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:57:07,579 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 18:57:10,210 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:57:10,210 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:57:25,342 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:57:25,343 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:57:37,658 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 18:57:40,474 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:57:40,475 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:57:55,605 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:57:55,606 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:58:07,732 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 18:58:10,737 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:58:10,737 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:58:25,870 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:58:25,870 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:58:37,810 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 18:58:41,005 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:58:41,005 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:58:56,137 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:58:56,138 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:59:07,886 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 18:59:11,274 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:59:11,274 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:59:26,406 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:59:26,406 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:59:37,963 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 18:59:41,543 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:59:41,543 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:59:56,673 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:59:56,674 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:00:08,040 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:00:11,805 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:00:11,806 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:00:26,938 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:00:26,938 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:00:38,115 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:00:42,071 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:00:42,073 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:00:57,206 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:00:57,206 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:01:08,189 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:01:12,338 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:01:12,338 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:01:27,470 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:01:27,470 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:01:38,253 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:01:42,602 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:01:42,602 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:01:57,736 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:01:57,736 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:02:08,323 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:02:12,869 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:02:12,869 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:02:28,002 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:02:28,002 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:02:38,399 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:02:43,167 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:02:43,167 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:02:58,298 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:02:58,299 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:03:08,473 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:03:13,435 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:03:13,435 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:03:28,570 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:03:28,570 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:03:38,550 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:03:43,702 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:03:43,702 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:03:58,838 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:03:58,838 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:04:08,624 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:04:13,976 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:04:13,976 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:04:29,120 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:04:29,121 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:04:38,701 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:04:44,257 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:04:44,258 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:04:59,397 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:04:59,397 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:05:08,777 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:05:14,531 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:05:14,532 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:05:29,687 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:05:29,688 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:05:38,851 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:05:44,819 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:05:44,820 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:05:59,950 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:05:59,951 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:06:08,925 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:06:15,090 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:06:15,091 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:06:30,226 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:06:30,226 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:06:39,001 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:06:45,360 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:06:45,360 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:07:00,496 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:07:00,496 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:07:09,079 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:07:15,632 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:07:15,633 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:07:30,767 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:07:30,767 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:07:39,157 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:07:45,903 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:07:45,903 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:08:01,041 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:08:01,041 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:08:09,235 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:08:16,178 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:08:16,179 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:08:31,336 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:08:31,336 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:08:39,312 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:08:46,469 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:08:46,470 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:09:01,600 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:09:01,601 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:09:09,391 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:09:16,736 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:09:16,736 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:09:31,869 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:09:31,870 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:09:39,454 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:09:47,008 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:09:47,008 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:10:02,139 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:10:02,140 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:10:09,532 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:10:17,277 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:10:17,277 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:10:32,410 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:10:32,410 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:10:39,609 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:10:47,545 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:10:47,545 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:11:01,425 DEBUG   SenderThread:555087 [sender.py:send():179] send: telemetry
2021-07-14 19:11:01,425 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:01,425 DEBUG   SenderThread:555087 [sender.py:send():179] send: exit
2021-07-14 19:11:01,426 INFO    SenderThread:555087 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 19:11:01,426 INFO    SenderThread:555087 [sender.py:send_exit():295] send defer
2021-07-14 19:11:01,427 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:01,427 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:01,427 INFO    HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 19:11:01,428 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:01,428 INFO    SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 19:11:01,428 INFO    SenderThread:555087 [sender.py:transition_state():308] send defer: 1
2021-07-14 19:11:01,428 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:01,428 INFO    HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 19:11:01,460 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:01,460 INFO    SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 19:11:01,460 INFO    SenderThread:555087 [sender.py:transition_state():308] send defer: 2
2021-07-14 19:11:01,460 DEBUG   SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:11:01,461 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:01,461 INFO    HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 19:11:01,461 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:01,461 INFO    SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 19:11:01,461 INFO    SenderThread:555087 [sender.py:transition_state():308] send defer: 3
2021-07-14 19:11:01,461 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:01,462 INFO    HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 19:11:01,462 DEBUG   SenderThread:555087 [sender.py:send():179] send: summary
2021-07-14 19:11:01,462 INFO    SenderThread:555087 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 19:11:01,462 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:01,462 INFO    SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 19:11:01,462 INFO    SenderThread:555087 [sender.py:transition_state():308] send defer: 4
2021-07-14 19:11:01,463 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:01,463 INFO    HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 19:11:01,463 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:01,463 INFO    SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 19:11:01,529 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:01,566 INFO    Thread-8  :555087 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/output.log
2021-07-14 19:11:01,566 INFO    Thread-8  :555087 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/wandb-summary.json
2021-07-14 19:11:01,667 INFO    SenderThread:555087 [sender.py:transition_state():308] send defer: 5
2021-07-14 19:11:01,667 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:01,668 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:01,668 INFO    HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 19:11:01,668 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:01,668 INFO    SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 19:11:01,668 INFO    SenderThread:555087 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 19:11:01,769 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:02,566 INFO    Thread-8  :555087 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/config.yaml
2021-07-14 19:11:02,567 INFO    SenderThread:555087 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files
2021-07-14 19:11:02,567 INFO    SenderThread:555087 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/requirements.txt requirements.txt
2021-07-14 19:11:02,567 INFO    SenderThread:555087 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/output.log output.log
2021-07-14 19:11:02,567 INFO    SenderThread:555087 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/wandb-metadata.json wandb-metadata.json
2021-07-14 19:11:02,567 INFO    SenderThread:555087 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/config.yaml config.yaml
2021-07-14 19:11:02,568 INFO    SenderThread:555087 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/wandb-summary.json wandb-summary.json
2021-07-14 19:11:02,568 INFO    SenderThread:555087 [sender.py:transition_state():308] send defer: 6
2021-07-14 19:11:02,569 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:02,572 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:02,572 INFO    HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 19:11:02,575 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:02,575 INFO    SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 19:11:02,575 INFO    SenderThread:555087 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 19:11:02,671 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:02,671 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:02,773 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:02,774 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:02,876 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:02,876 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:02,977 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:02,978 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:03,031 INFO    Thread-14 :555087 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/config.yaml
2021-07-14 19:11:03,066 INFO    Thread-13 :555087 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/output.log
2021-07-14 19:11:03,074 INFO    Thread-12 :555087 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/requirements.txt
2021-07-14 19:11:03,079 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:03,079 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:03,088 INFO    Thread-15 :555087 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/wandb-summary.json
2021-07-14 19:11:03,181 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:03,181 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:03,283 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:03,283 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:03,288 INFO    Thread-7  :555087 [sender.py:transition_state():308] send defer: 7
2021-07-14 19:11:03,288 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:03,289 INFO    HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 19:11:03,289 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:03,289 INFO    SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 19:11:03,385 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:03,599 INFO    SenderThread:555087 [sender.py:transition_state():308] send defer: 8
2021-07-14 19:11:03,600 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:03,600 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:03,600 INFO    HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 19:11:03,600 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:03,600 INFO    SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 19:11:03,601 INFO    SenderThread:555087 [sender.py:transition_state():308] send defer: 9
2021-07-14 19:11:03,601 DEBUG   SenderThread:555087 [sender.py:send():179] send: final
2021-07-14 19:11:03,601 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:03,601 INFO    HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 19:11:03,601 DEBUG   SenderThread:555087 [sender.py:send():179] send: footer
2021-07-14 19:11:03,602 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:03,602 INFO    SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 19:11:03,702 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:03,702 DEBUG   SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:03,702 INFO    SenderThread:555087 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 19:11:03,704 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 19:11:03,704 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 19:11:03,705 DEBUG   HandlerThread:555087 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 19:11:03,705 INFO    HandlerThread:555087 [handler.py:finish():638] shutting down handler
2021-07-14 19:11:04,602 INFO    WriterThread:555087 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/run-379h21h2.wandb
2021-07-14 19:11:04,703 INFO    SenderThread:555087 [sender.py:finish():945] shutting down sender
2021-07-14 19:11:04,703 INFO    SenderThread:555087 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 19:11:04,703 INFO    SenderThread:555087 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 19:11:04,706 INFO    MainThread:555087 [internal.py:handle_exit():78] Internal process exited