File size: 23,414 Bytes
731244f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
2021-07-15 01:00:39,018 INFO    MainThread:626669 [internal.py:wandb_internal():88] W&B internal server running at pid: 626669, started at: 2021-07-15 01:00:39.017812
2021-07-15 01:00:39,020 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: check_version
2021-07-15 01:00:39,020 INFO    WriterThread:626669 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/run-2gdygcu4.wandb
2021-07-15 01:00:39,021 DEBUG   SenderThread:626669 [sender.py:send():179] send: header
2021-07-15 01:00:39,021 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: check_version
2021-07-15 01:00:39,055 DEBUG   SenderThread:626669 [sender.py:send():179] send: run
2021-07-15 01:00:39,219 INFO    SenderThread:626669 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files
2021-07-15 01:00:39,219 INFO    SenderThread:626669 [sender.py:_start_run_threads():716] run started: 2gdygcu4 with start time 1626310838
2021-07-15 01:00:39,220 DEBUG   SenderThread:626669 [sender.py:send():179] send: summary
2021-07-15 01:00:39,220 INFO    SenderThread:626669 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 01:00:39,220 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: run_start
2021-07-15 01:00:40,222 INFO    Thread-8  :626669 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/wandb-summary.json
2021-07-15 01:00:40,314 DEBUG   HandlerThread:626669 [meta.py:__init__():39] meta init
2021-07-15 01:00:40,314 DEBUG   HandlerThread:626669 [meta.py:__init__():53] meta init done
2021-07-15 01:00:40,314 DEBUG   HandlerThread:626669 [meta.py:probe():210] probe
2021-07-15 01:00:40,316 DEBUG   HandlerThread:626669 [meta.py:_setup_git():200] setup git
2021-07-15 01:00:40,345 DEBUG   HandlerThread:626669 [meta.py:_setup_git():207] setup git done
2021-07-15 01:00:40,345 DEBUG   HandlerThread:626669 [meta.py:_save_pip():57] save pip
2021-07-15 01:00:40,346 DEBUG   HandlerThread:626669 [meta.py:_save_pip():71] save pip done
2021-07-15 01:00:40,346 DEBUG   HandlerThread:626669 [meta.py:probe():252] probe done
2021-07-15 01:00:40,349 DEBUG   SenderThread:626669 [sender.py:send():179] send: files
2021-07-15 01:00:40,349 INFO    SenderThread:626669 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-15 01:00:40,354 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:00:40,355 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:00:40,481 DEBUG   SenderThread:626669 [sender.py:send():179] send: config
2021-07-15 01:00:40,482 DEBUG   SenderThread:626669 [sender.py:send():179] send: config
2021-07-15 01:00:40,482 DEBUG   SenderThread:626669 [sender.py:send():179] send: config
2021-07-15 01:00:40,774 INFO    Thread-11 :626669 [upload_job.py:push():137] Uploaded file /tmp/tmpwx_o6qsuwandb/316agsp6-wandb-metadata.json
2021-07-15 01:00:41,221 INFO    Thread-8  :626669 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/requirements.txt
2021-07-15 01:00:41,221 INFO    Thread-8  :626669 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/wandb-metadata.json
2021-07-15 01:00:41,221 INFO    Thread-8  :626669 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/output.log
2021-07-15 01:00:55,227 INFO    Thread-8  :626669 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/output.log
2021-07-15 01:00:55,483 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:00:55,484 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:01:08,399 DEBUG   SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:01:10,233 INFO    Thread-8  :626669 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/config.yaml
2021-07-15 01:01:10,612 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:01:10,613 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:01:25,743 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:01:25,744 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:01:38,482 DEBUG   SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:01:40,875 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:01:40,875 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:01:56,006 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:01:56,006 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:02:08,558 DEBUG   SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:02:11,138 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:02:11,138 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:02:26,268 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:02:26,268 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:02:38,636 DEBUG   SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:02:41,402 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:02:41,402 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:02:56,534 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:02:56,534 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:03:08,712 DEBUG   SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:03:11,667 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:03:11,668 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:03:26,799 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:03:26,800 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:03:38,780 DEBUG   SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:03:41,932 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:03:41,932 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:03:57,065 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:03:57,065 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:04:08,851 DEBUG   SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:04:12,197 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:04:12,197 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:04:27,326 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:04:27,327 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:04:38,928 DEBUG   SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:04:42,462 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:04:42,463 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:04:57,592 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:04:57,593 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:05:08,987 DEBUG   SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:05:12,723 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:05:12,723 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:05:27,851 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:05:27,852 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:05:39,058 DEBUG   SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:05:39,359 INFO    Thread-8  :626669 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/output.log
2021-07-15 01:05:43,032 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:05:43,032 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:05:58,189 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:05:58,190 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:06:09,137 DEBUG   SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:06:13,331 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:06:13,331 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:06:28,461 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:06:28,461 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:06:39,220 DEBUG   SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:06:43,596 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:06:43,596 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:06:58,728 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:06:58,728 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:07:05,299 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:05,299 DEBUG   SenderThread:626669 [sender.py:send():179] send: telemetry
2021-07-15 01:07:05,299 DEBUG   SenderThread:626669 [sender.py:send():179] send: exit
2021-07-15 01:07:05,299 INFO    SenderThread:626669 [sender.py:send_exit():287] handling exit code: 1
2021-07-15 01:07:05,299 INFO    SenderThread:626669 [sender.py:send_exit():295] send defer
2021-07-15 01:07:05,299 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:05,300 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:05,300 INFO    HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-15 01:07:05,300 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:05,300 INFO    SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-15 01:07:05,300 INFO    SenderThread:626669 [sender.py:transition_state():308] send defer: 1
2021-07-15 01:07:05,300 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:05,301 INFO    HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-15 01:07:05,385 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:05,385 INFO    SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-15 01:07:05,385 INFO    SenderThread:626669 [sender.py:transition_state():308] send defer: 2
2021-07-15 01:07:05,385 DEBUG   SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:07:05,386 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:05,386 INFO    HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-15 01:07:05,386 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:05,386 INFO    SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-15 01:07:05,386 INFO    SenderThread:626669 [sender.py:transition_state():308] send defer: 3
2021-07-15 01:07:05,386 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:05,386 INFO    HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-15 01:07:05,386 DEBUG   SenderThread:626669 [sender.py:send():179] send: summary
2021-07-15 01:07:05,387 INFO    SenderThread:626669 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 01:07:05,387 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:05,387 INFO    SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-15 01:07:05,387 INFO    SenderThread:626669 [sender.py:transition_state():308] send defer: 4
2021-07-15 01:07:05,387 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:05,387 INFO    HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-15 01:07:05,387 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:05,387 INFO    SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-15 01:07:05,393 INFO    Thread-8  :626669 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/wandb-summary.json
2021-07-15 01:07:05,393 INFO    Thread-8  :626669 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/output.log
2021-07-15 01:07:05,401 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:05,571 INFO    SenderThread:626669 [sender.py:transition_state():308] send defer: 5
2021-07-15 01:07:05,571 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:05,572 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:05,572 INFO    HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-15 01:07:05,572 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:05,572 INFO    SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-15 01:07:05,572 INFO    SenderThread:626669 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-15 01:07:05,673 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:06,394 INFO    Thread-8  :626669 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/config.yaml
2021-07-15 01:07:06,394 INFO    SenderThread:626669 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files
2021-07-15 01:07:06,395 INFO    SenderThread:626669 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/requirements.txt requirements.txt
2021-07-15 01:07:06,395 INFO    SenderThread:626669 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/output.log output.log
2021-07-15 01:07:06,395 INFO    SenderThread:626669 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/wandb-metadata.json wandb-metadata.json
2021-07-15 01:07:06,395 INFO    SenderThread:626669 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/config.yaml config.yaml
2021-07-15 01:07:06,395 INFO    SenderThread:626669 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/wandb-summary.json wandb-summary.json
2021-07-15 01:07:06,395 INFO    SenderThread:626669 [sender.py:transition_state():308] send defer: 6
2021-07-15 01:07:06,396 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:06,399 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:06,399 INFO    HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-15 01:07:06,399 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:06,400 INFO    SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-15 01:07:06,400 INFO    SenderThread:626669 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 01:07:06,501 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:06,501 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:06,603 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:06,604 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:06,706 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:06,706 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:06,808 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:06,808 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:06,857 INFO    Thread-12 :626669 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/requirements.txt
2021-07-15 01:07:06,884 INFO    Thread-15 :626669 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/wandb-summary.json
2021-07-15 01:07:06,910 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:06,910 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:06,920 INFO    Thread-14 :626669 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/config.yaml
2021-07-15 01:07:06,968 INFO    Thread-13 :626669 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/output.log
2021-07-15 01:07:07,012 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:07,012 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:07,115 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:07,115 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:07,169 INFO    Thread-7  :626669 [sender.py:transition_state():308] send defer: 7
2021-07-15 01:07:07,169 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:07,169 INFO    HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-15 01:07:07,170 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:07,170 INFO    SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-15 01:07:07,217 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:07,450 INFO    SenderThread:626669 [sender.py:transition_state():308] send defer: 8
2021-07-15 01:07:07,450 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:07,451 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:07,451 INFO    HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-15 01:07:07,451 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:07,451 INFO    SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-15 01:07:07,451 INFO    SenderThread:626669 [sender.py:transition_state():308] send defer: 9
2021-07-15 01:07:07,452 DEBUG   SenderThread:626669 [sender.py:send():179] send: final
2021-07-15 01:07:07,452 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:07,452 INFO    HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-15 01:07:07,452 DEBUG   SenderThread:626669 [sender.py:send():179] send: footer
2021-07-15 01:07:07,452 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:07,452 INFO    SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-15 01:07:07,552 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:07,552 DEBUG   SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:07,553 INFO    SenderThread:626669 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 01:07:07,554 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: get_summary
2021-07-15 01:07:07,555 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-15 01:07:07,555 DEBUG   HandlerThread:626669 [handler.py:handle_request():124] handle_request: shutdown
2021-07-15 01:07:07,555 INFO    HandlerThread:626669 [handler.py:finish():638] shutting down handler
2021-07-15 01:07:08,453 INFO    WriterThread:626669 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/run-2gdygcu4.wandb
2021-07-15 01:07:08,553 INFO    SenderThread:626669 [sender.py:finish():945] shutting down sender
2021-07-15 01:07:08,553 INFO    SenderThread:626669 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 01:07:08,553 INFO    SenderThread:626669 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 01:07:08,555 INFO    MainThread:626669 [internal.py:handle_exit():78] Internal process exited