File size: 31,534 Bytes
731244f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
2021-07-14 23:11:47,821 INFO    MainThread:604064 [internal.py:wandb_internal():88] W&B internal server running at pid: 604064, started at: 2021-07-14 23:11:47.821366
2021-07-14 23:11:47,823 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 23:11:47,824 INFO    WriterThread:604064 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/run-gkn68kcy.wandb
2021-07-14 23:11:47,824 DEBUG   SenderThread:604064 [sender.py:send():179] send: header
2021-07-14 23:11:47,825 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: check_version
2021-07-14 23:11:47,862 DEBUG   SenderThread:604064 [sender.py:send():179] send: run
2021-07-14 23:11:48,044 INFO    SenderThread:604064 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files
2021-07-14 23:11:48,044 INFO    SenderThread:604064 [sender.py:_start_run_threads():716] run started: gkn68kcy with start time 1626304307
2021-07-14 23:11:48,044 DEBUG   SenderThread:604064 [sender.py:send():179] send: summary
2021-07-14 23:11:48,044 INFO    SenderThread:604064 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 23:11:48,045 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 23:11:49,047 INFO    Thread-8  :604064 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/wandb-summary.json
2021-07-14 23:11:49,155 DEBUG   HandlerThread:604064 [meta.py:__init__():39] meta init
2021-07-14 23:11:49,155 DEBUG   HandlerThread:604064 [meta.py:__init__():53] meta init done
2021-07-14 23:11:49,155 DEBUG   HandlerThread:604064 [meta.py:probe():210] probe
2021-07-14 23:11:49,156 DEBUG   HandlerThread:604064 [meta.py:_setup_git():200] setup git
2021-07-14 23:11:49,187 DEBUG   HandlerThread:604064 [meta.py:_setup_git():207] setup git done
2021-07-14 23:11:49,188 DEBUG   HandlerThread:604064 [meta.py:_save_pip():57] save pip
2021-07-14 23:11:49,188 DEBUG   HandlerThread:604064 [meta.py:_save_pip():71] save pip done
2021-07-14 23:11:49,188 DEBUG   HandlerThread:604064 [meta.py:probe():252] probe done
2021-07-14 23:11:49,192 DEBUG   SenderThread:604064 [sender.py:send():179] send: files
2021-07-14 23:11:49,192 INFO    SenderThread:604064 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 23:11:49,199 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:11:49,200 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:11:49,336 DEBUG   SenderThread:604064 [sender.py:send():179] send: config
2021-07-14 23:11:49,336 DEBUG   SenderThread:604064 [sender.py:send():179] send: config
2021-07-14 23:11:49,337 DEBUG   SenderThread:604064 [sender.py:send():179] send: config
2021-07-14 23:11:49,685 INFO    Thread-11 :604064 [upload_job.py:push():137] Uploaded file /tmp/tmpnh56hsgtwandb/3jt8aozu-wandb-metadata.json
2021-07-14 23:11:50,046 INFO    Thread-8  :604064 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/output.log
2021-07-14 23:11:50,046 INFO    Thread-8  :604064 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/requirements.txt
2021-07-14 23:11:50,046 INFO    Thread-8  :604064 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/wandb-metadata.json
2021-07-14 23:12:04,051 INFO    Thread-8  :604064 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/output.log
2021-07-14 23:12:04,338 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:12:04,339 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:12:17,233 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:12:19,058 INFO    Thread-8  :604064 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/config.yaml
2021-07-14 23:12:19,470 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:12:19,471 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:12:34,607 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:12:34,607 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:12:47,304 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:12:49,759 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:12:49,760 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:13:04,899 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:13:04,899 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:13:17,375 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:13:20,032 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:13:20,032 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:13:35,167 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:13:35,168 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:13:47,450 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:13:50,301 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:13:50,301 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:14:05,433 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:14:05,434 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:14:17,527 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:14:20,564 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:14:20,564 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:14:35,694 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:14:35,695 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:14:47,605 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:14:50,827 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:14:50,827 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:15:05,965 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:15:05,965 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:15:17,682 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:15:21,099 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:15:21,100 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:15:36,236 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:15:36,237 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:15:47,752 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:15:51,383 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:15:51,384 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:16:06,514 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:16:06,515 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:16:17,832 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:16:21,647 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:16:21,647 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:16:36,777 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:16:36,778 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:16:47,906 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:16:51,911 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:16:51,911 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:17:07,045 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:17:07,045 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:17:17,984 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:17:22,180 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:17:22,180 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:17:37,312 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:17:37,313 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:17:48,061 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:17:52,447 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:17:52,447 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:18:07,584 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:18:07,584 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:18:18,134 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:18:22,718 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:18:22,718 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:18:37,852 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:18:37,852 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:18:48,202 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:18:52,983 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:18:52,984 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:19:08,118 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:19:08,118 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:19:18,280 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:19:23,249 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:19:23,249 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:19:38,406 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:19:38,407 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:19:48,357 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:19:53,538 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:19:53,539 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:20:08,668 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:20:08,668 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:20:18,431 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:20:23,808 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:20:23,809 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:20:38,943 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:20:38,943 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:20:48,507 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:20:54,079 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:20:54,079 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:21:09,222 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:21:09,223 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:21:18,586 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:21:24,353 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:21:24,354 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:21:39,483 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:21:39,484 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:21:48,663 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:21:55,394 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:21:55,394 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:22:10,527 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:22:10,528 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:22:18,741 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:22:25,659 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:22:25,659 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:22:40,790 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:22:40,790 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:22:48,820 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:22:55,922 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:22:55,923 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:23:11,058 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:23:11,059 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:23:18,897 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:23:26,197 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:23:26,198 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:23:41,329 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:23:41,329 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:23:48,974 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:23:56,463 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:23:56,463 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:24:11,593 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:24:11,593 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:24:19,051 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:24:26,724 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:24:26,724 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:24:41,858 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:24:41,858 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:24:49,130 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:24:56,991 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:24:56,991 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:25:12,121 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:25:12,122 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:25:19,207 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:25:27,253 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:25:27,254 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:25:42,385 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:25:42,386 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:25:49,284 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:25:57,527 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:25:57,527 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:26:06,351 INFO    Thread-8  :604064 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/output.log
2021-07-14 23:26:07,081 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:07,082 DEBUG   SenderThread:604064 [sender.py:send():179] send: telemetry
2021-07-14 23:26:07,082 DEBUG   SenderThread:604064 [sender.py:send():179] send: exit
2021-07-14 23:26:07,082 INFO    SenderThread:604064 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 23:26:07,083 INFO    SenderThread:604064 [sender.py:send_exit():295] send defer
2021-07-14 23:26:07,084 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:07,084 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:07,085 INFO    HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 23:26:07,085 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:07,085 INFO    SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 23:26:07,085 INFO    SenderThread:604064 [sender.py:transition_state():308] send defer: 1
2021-07-14 23:26:07,085 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:07,085 INFO    HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 23:26:07,130 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:07,130 INFO    SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 23:26:07,130 INFO    SenderThread:604064 [sender.py:transition_state():308] send defer: 2
2021-07-14 23:26:07,131 DEBUG   SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:26:07,131 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:07,131 INFO    HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 23:26:07,132 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:07,132 INFO    SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 23:26:07,132 INFO    SenderThread:604064 [sender.py:transition_state():308] send defer: 3
2021-07-14 23:26:07,132 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:07,132 INFO    HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 23:26:07,132 DEBUG   SenderThread:604064 [sender.py:send():179] send: summary
2021-07-14 23:26:07,133 INFO    SenderThread:604064 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 23:26:07,133 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:07,133 INFO    SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 23:26:07,133 INFO    SenderThread:604064 [sender.py:transition_state():308] send defer: 4
2021-07-14 23:26:07,134 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:07,134 INFO    HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 23:26:07,134 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:07,134 INFO    SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 23:26:07,228 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:07,305 INFO    SenderThread:604064 [sender.py:transition_state():308] send defer: 5
2021-07-14 23:26:07,305 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:07,306 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:07,306 INFO    HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 23:26:07,306 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:07,306 INFO    SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 23:26:07,306 INFO    SenderThread:604064 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 23:26:07,351 INFO    Thread-8  :604064 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/config.yaml
2021-07-14 23:26:07,351 INFO    SenderThread:604064 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/output.log
2021-07-14 23:26:07,352 INFO    SenderThread:604064 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/wandb-summary.json
2021-07-14 23:26:07,352 INFO    SenderThread:604064 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files
2021-07-14 23:26:07,352 INFO    SenderThread:604064 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/requirements.txt requirements.txt
2021-07-14 23:26:07,352 INFO    SenderThread:604064 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/output.log output.log
2021-07-14 23:26:07,352 INFO    SenderThread:604064 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/wandb-metadata.json wandb-metadata.json
2021-07-14 23:26:07,352 INFO    SenderThread:604064 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/config.yaml config.yaml
2021-07-14 23:26:07,356 INFO    SenderThread:604064 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/wandb-summary.json wandb-summary.json
2021-07-14 23:26:07,359 INFO    SenderThread:604064 [sender.py:transition_state():308] send defer: 6
2021-07-14 23:26:07,360 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:07,360 INFO    HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 23:26:07,360 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:07,361 INFO    SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 23:26:07,361 INFO    SenderThread:604064 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 23:26:07,407 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:07,408 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:07,513 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:07,514 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:07,616 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:07,616 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:07,718 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:07,718 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:07,820 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:07,820 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:07,840 INFO    Thread-14 :604064 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/config.yaml
2021-07-14 23:26:07,841 INFO    Thread-13 :604064 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/output.log
2021-07-14 23:26:07,874 INFO    Thread-12 :604064 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/requirements.txt
2021-07-14 23:26:07,875 INFO    Thread-15 :604064 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/wandb-summary.json
2021-07-14 23:26:07,922 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:07,922 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:08,024 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:08,024 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:08,076 INFO    Thread-7  :604064 [sender.py:transition_state():308] send defer: 7
2021-07-14 23:26:08,077 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:08,077 INFO    HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 23:26:08,077 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:08,077 INFO    SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 23:26:08,126 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:08,360 INFO    SenderThread:604064 [sender.py:transition_state():308] send defer: 8
2021-07-14 23:26:08,360 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:08,361 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:08,361 INFO    HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 23:26:08,361 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:08,361 INFO    SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 23:26:08,361 INFO    SenderThread:604064 [sender.py:transition_state():308] send defer: 9
2021-07-14 23:26:08,362 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:08,362 INFO    HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 23:26:08,362 DEBUG   SenderThread:604064 [sender.py:send():179] send: final
2021-07-14 23:26:08,362 DEBUG   SenderThread:604064 [sender.py:send():179] send: footer
2021-07-14 23:26:08,362 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:08,362 INFO    SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 23:26:08,462 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:08,462 DEBUG   SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:08,463 INFO    SenderThread:604064 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 23:26:08,464 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 23:26:08,465 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 23:26:08,465 DEBUG   HandlerThread:604064 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 23:26:08,465 INFO    HandlerThread:604064 [handler.py:finish():638] shutting down handler
2021-07-14 23:26:09,363 INFO    WriterThread:604064 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/run-gkn68kcy.wandb
2021-07-14 23:26:09,463 INFO    SenderThread:604064 [sender.py:finish():945] shutting down sender
2021-07-14 23:26:09,463 INFO    SenderThread:604064 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 23:26:09,463 INFO    SenderThread:604064 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 23:26:09,466 INFO    MainThread:604064 [internal.py:handle_exit():78] Internal process exited