File size: 26,719 Bytes
f291f93
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
2021-07-14 18:33:05,441 INFO    MainThread:551470 [internal.py:wandb_internal():88] W&B internal server running at pid: 551470, started at: 2021-07-14 18:33:05.441281
2021-07-14 18:33:05,443 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 18:33:05,443 INFO    WriterThread:551470 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/run-3d4rorca.wandb
2021-07-14 18:33:05,444 DEBUG   SenderThread:551470 [sender.py:send():179] send: header
2021-07-14 18:33:05,444 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: check_version
2021-07-14 18:33:05,481 DEBUG   SenderThread:551470 [sender.py:send():179] send: run
2021-07-14 18:33:05,658 INFO    SenderThread:551470 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files
2021-07-14 18:33:05,658 INFO    SenderThread:551470 [sender.py:_start_run_threads():716] run started: 3d4rorca with start time 1626287584
2021-07-14 18:33:05,658 DEBUG   SenderThread:551470 [sender.py:send():179] send: summary
2021-07-14 18:33:05,659 INFO    SenderThread:551470 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 18:33:05,659 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 18:33:06,662 INFO    Thread-8  :551470 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/wandb-summary.json
2021-07-14 18:33:06,772 DEBUG   HandlerThread:551470 [meta.py:__init__():39] meta init
2021-07-14 18:33:06,772 DEBUG   HandlerThread:551470 [meta.py:__init__():53] meta init done
2021-07-14 18:33:06,772 DEBUG   HandlerThread:551470 [meta.py:probe():210] probe
2021-07-14 18:33:06,773 DEBUG   HandlerThread:551470 [meta.py:_setup_git():200] setup git
2021-07-14 18:33:06,799 DEBUG   HandlerThread:551470 [meta.py:_setup_git():207] setup git done
2021-07-14 18:33:06,799 DEBUG   HandlerThread:551470 [meta.py:_save_pip():57] save pip
2021-07-14 18:33:06,799 DEBUG   HandlerThread:551470 [meta.py:_save_pip():71] save pip done
2021-07-14 18:33:06,799 DEBUG   HandlerThread:551470 [meta.py:probe():252] probe done
2021-07-14 18:33:06,802 DEBUG   SenderThread:551470 [sender.py:send():179] send: files
2021-07-14 18:33:06,802 INFO    SenderThread:551470 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 18:33:06,808 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:33:06,808 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:33:06,939 DEBUG   SenderThread:551470 [sender.py:send():179] send: config
2021-07-14 18:33:06,940 DEBUG   SenderThread:551470 [sender.py:send():179] send: config
2021-07-14 18:33:06,940 DEBUG   SenderThread:551470 [sender.py:send():179] send: config
2021-07-14 18:33:07,267 INFO    Thread-11 :551470 [upload_job.py:push():137] Uploaded file /tmp/tmpwkna8gakwandb/1910yhst-wandb-metadata.json
2021-07-14 18:33:07,661 INFO    Thread-8  :551470 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/output.log
2021-07-14 18:33:07,661 INFO    Thread-8  :551470 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/requirements.txt
2021-07-14 18:33:07,661 INFO    Thread-8  :551470 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/wandb-metadata.json
2021-07-14 18:33:21,941 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:33:21,941 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:33:25,667 INFO    Thread-8  :551470 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/output.log
2021-07-14 18:33:34,849 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:33:36,671 INFO    Thread-8  :551470 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/config.yaml
2021-07-14 18:33:37,073 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:33:37,073 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:33:52,207 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:33:52,207 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:34:04,916 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:34:07,343 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:34:07,343 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:34:22,477 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:34:22,477 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:34:34,976 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:34:37,613 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:34:37,613 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:34:52,747 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:34:52,747 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:35:05,044 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:35:07,878 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:35:07,879 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:35:23,013 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:35:23,013 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:35:35,110 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:35:38,147 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:35:38,148 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:35:53,282 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:35:53,282 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:36:05,181 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:36:08,416 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:36:08,417 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:36:23,549 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:36:23,549 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:36:35,251 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:36:38,681 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:36:38,681 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:36:53,817 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:36:53,817 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:37:05,320 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:37:08,952 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:37:08,953 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:37:24,087 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:37:24,087 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:37:35,392 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:37:39,221 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:37:39,222 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:37:54,352 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:37:54,353 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:38:05,464 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:38:09,485 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:38:09,485 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:38:24,626 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:38:24,626 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:38:35,535 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:38:39,761 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:38:39,761 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:38:54,893 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:38:54,893 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:39:05,597 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:39:10,025 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:39:10,026 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:39:25,164 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:39:25,164 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:39:35,672 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:39:40,296 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:39:40,297 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:39:55,430 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:39:55,430 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:40:05,746 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:40:10,563 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:40:10,564 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:40:25,697 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:40:25,698 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:40:35,821 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:40:40,834 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:40:40,834 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:40:55,966 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:40:55,967 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:41:05,888 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:41:11,114 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:41:11,114 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:41:11,814 INFO    Thread-8  :551470 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/output.log
2021-07-14 18:41:26,276 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:41:26,277 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:41:35,966 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:41:41,417 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:41:41,418 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:41:56,548 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:41:56,548 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:42:06,047 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:42:11,683 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:42:11,684 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:42:26,818 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:42:26,818 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:42:36,121 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:42:41,950 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:42:41,951 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:42:51,640 DEBUG   SenderThread:551470 [sender.py:send():179] send: telemetry
2021-07-14 18:42:51,641 DEBUG   SenderThread:551470 [sender.py:send():179] send: exit
2021-07-14 18:42:51,641 INFO    SenderThread:551470 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 18:42:51,641 INFO    SenderThread:551470 [sender.py:send_exit():295] send defer
2021-07-14 18:42:51,641 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:51,642 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:51,642 INFO    HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 18:42:51,642 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:51,642 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:51,642 INFO    SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 18:42:51,642 INFO    SenderThread:551470 [sender.py:transition_state():308] send defer: 1
2021-07-14 18:42:51,643 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:51,643 INFO    HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 18:42:51,659 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:51,660 INFO    SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 18:42:51,660 INFO    SenderThread:551470 [sender.py:transition_state():308] send defer: 2
2021-07-14 18:42:51,660 DEBUG   SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:42:51,660 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:51,660 INFO    HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 18:42:51,660 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:51,660 INFO    SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 18:42:51,660 INFO    SenderThread:551470 [sender.py:transition_state():308] send defer: 3
2021-07-14 18:42:51,661 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:51,661 INFO    HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 18:42:51,661 DEBUG   SenderThread:551470 [sender.py:send():179] send: summary
2021-07-14 18:42:51,661 INFO    SenderThread:551470 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 18:42:51,661 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:51,661 INFO    SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 18:42:51,661 INFO    SenderThread:551470 [sender.py:transition_state():308] send defer: 4
2021-07-14 18:42:51,662 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:51,662 INFO    HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 18:42:51,662 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:51,662 INFO    SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 18:42:51,744 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:51,838 INFO    SenderThread:551470 [sender.py:transition_state():308] send defer: 5
2021-07-14 18:42:51,838 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:51,838 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:51,838 INFO    HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 18:42:51,839 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:51,839 INFO    SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 18:42:51,839 INFO    SenderThread:551470 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 18:42:51,854 INFO    Thread-8  :551470 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/config.yaml
2021-07-14 18:42:51,855 INFO    SenderThread:551470 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/output.log
2021-07-14 18:42:51,855 INFO    SenderThread:551470 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/wandb-summary.json
2021-07-14 18:42:51,855 INFO    SenderThread:551470 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files
2021-07-14 18:42:51,855 INFO    SenderThread:551470 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/requirements.txt requirements.txt
2021-07-14 18:42:51,855 INFO    SenderThread:551470 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/output.log output.log
2021-07-14 18:42:51,856 INFO    SenderThread:551470 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/wandb-metadata.json wandb-metadata.json
2021-07-14 18:42:51,856 INFO    SenderThread:551470 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/config.yaml config.yaml
2021-07-14 18:42:51,856 INFO    SenderThread:551470 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/wandb-summary.json wandb-summary.json
2021-07-14 18:42:51,856 INFO    SenderThread:551470 [sender.py:transition_state():308] send defer: 6
2021-07-14 18:42:51,862 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:51,862 INFO    HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 18:42:51,865 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:51,865 INFO    SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 18:42:51,865 INFO    SenderThread:551470 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 18:42:51,939 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:51,940 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:52,041 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:52,041 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:52,143 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:52,143 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:52,245 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:52,245 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:52,308 INFO    Thread-12 :551470 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/requirements.txt
2021-07-14 18:42:52,325 INFO    Thread-13 :551470 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/output.log
2021-07-14 18:42:52,341 INFO    Thread-15 :551470 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/wandb-summary.json
2021-07-14 18:42:52,346 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:52,346 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:52,361 INFO    Thread-14 :551470 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/config.yaml
2021-07-14 18:42:52,448 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:52,448 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:52,549 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:52,550 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:52,562 INFO    Thread-7  :551470 [sender.py:transition_state():308] send defer: 7
2021-07-14 18:42:52,562 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:52,562 INFO    HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 18:42:52,562 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:52,562 INFO    SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 18:42:52,651 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:52,837 INFO    SenderThread:551470 [sender.py:transition_state():308] send defer: 8
2021-07-14 18:42:52,837 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:52,837 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:52,837 INFO    HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 18:42:52,838 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:52,838 INFO    SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 18:42:52,838 INFO    SenderThread:551470 [sender.py:transition_state():308] send defer: 9
2021-07-14 18:42:52,838 DEBUG   SenderThread:551470 [sender.py:send():179] send: final
2021-07-14 18:42:52,838 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:52,838 INFO    HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 18:42:52,839 DEBUG   SenderThread:551470 [sender.py:send():179] send: footer
2021-07-14 18:42:52,839 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:52,839 INFO    SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 18:42:52,938 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:52,939 DEBUG   SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:52,939 INFO    SenderThread:551470 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 18:42:52,940 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 18:42:52,941 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 18:42:52,941 DEBUG   HandlerThread:551470 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 18:42:52,941 INFO    HandlerThread:551470 [handler.py:finish():638] shutting down handler
2021-07-14 18:42:53,839 INFO    WriterThread:551470 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/run-3d4rorca.wandb
2021-07-14 18:42:53,939 INFO    SenderThread:551470 [sender.py:finish():945] shutting down sender
2021-07-14 18:42:53,939 INFO    SenderThread:551470 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 18:42:53,939 INFO    SenderThread:551470 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 18:42:53,941 INFO    MainThread:551470 [internal.py:handle_exit():78] Internal process exited