File size: 24,236 Bytes
731244f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
2021-07-15 00:52:29,330 INFO    MainThread:624595 [internal.py:wandb_internal():88] W&B internal server running at pid: 624595, started at: 2021-07-15 00:52:29.329937
2021-07-15 00:52:29,332 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: check_version
2021-07-15 00:52:29,333 INFO    WriterThread:624595 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/run-361cvuow.wandb
2021-07-15 00:52:29,333 DEBUG   SenderThread:624595 [sender.py:send():179] send: header
2021-07-15 00:52:29,333 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: check_version
2021-07-15 00:52:29,368 DEBUG   SenderThread:624595 [sender.py:send():179] send: run
2021-07-15 00:52:29,550 INFO    SenderThread:624595 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files
2021-07-15 00:52:29,550 INFO    SenderThread:624595 [sender.py:_start_run_threads():716] run started: 361cvuow with start time 1626310348
2021-07-15 00:52:29,550 DEBUG   SenderThread:624595 [sender.py:send():179] send: summary
2021-07-15 00:52:29,551 INFO    SenderThread:624595 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 00:52:29,551 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: run_start
2021-07-15 00:52:30,553 INFO    Thread-8  :624595 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/wandb-summary.json
2021-07-15 00:52:30,640 DEBUG   HandlerThread:624595 [meta.py:__init__():39] meta init
2021-07-15 00:52:30,640 DEBUG   HandlerThread:624595 [meta.py:__init__():53] meta init done
2021-07-15 00:52:30,640 DEBUG   HandlerThread:624595 [meta.py:probe():210] probe
2021-07-15 00:52:30,641 DEBUG   HandlerThread:624595 [meta.py:_setup_git():200] setup git
2021-07-15 00:52:30,670 DEBUG   HandlerThread:624595 [meta.py:_setup_git():207] setup git done
2021-07-15 00:52:30,670 DEBUG   HandlerThread:624595 [meta.py:_save_pip():57] save pip
2021-07-15 00:52:30,671 DEBUG   HandlerThread:624595 [meta.py:_save_pip():71] save pip done
2021-07-15 00:52:30,671 DEBUG   HandlerThread:624595 [meta.py:probe():252] probe done
2021-07-15 00:52:30,674 DEBUG   SenderThread:624595 [sender.py:send():179] send: files
2021-07-15 00:52:30,674 INFO    SenderThread:624595 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-15 00:52:30,680 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:52:30,680 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:52:30,812 DEBUG   SenderThread:624595 [sender.py:send():179] send: config
2021-07-15 00:52:30,812 DEBUG   SenderThread:624595 [sender.py:send():179] send: config
2021-07-15 00:52:30,813 DEBUG   SenderThread:624595 [sender.py:send():179] send: config
2021-07-15 00:52:31,126 INFO    Thread-11 :624595 [upload_job.py:push():137] Uploaded file /tmp/tmp7sk_pd_0wandb/2r26ralk-wandb-metadata.json
2021-07-15 00:52:31,551 INFO    Thread-8  :624595 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/wandb-metadata.json
2021-07-15 00:52:31,552 INFO    Thread-8  :624595 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/requirements.txt
2021-07-15 00:52:31,552 INFO    Thread-8  :624595 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log
2021-07-15 00:52:45,557 INFO    Thread-8  :624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log
2021-07-15 00:52:45,814 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:52:45,814 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:52:58,723 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:53:00,564 INFO    Thread-8  :624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/config.yaml
2021-07-15 00:53:00,947 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:53:00,947 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:53:16,080 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:53:16,080 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:53:28,803 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:53:31,216 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:53:31,216 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:53:46,347 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:53:46,347 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:53:58,879 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:54:01,496 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:54:01,496 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:54:16,630 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:54:16,630 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:54:28,953 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:54:31,760 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:54:31,761 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:54:46,892 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:54:46,893 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:54:59,029 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:55:02,026 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:55:02,027 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:55:17,159 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:55:17,160 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:55:29,107 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:55:32,292 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:55:32,292 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:55:47,422 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:55:47,422 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:55:59,186 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:56:02,555 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:56:02,556 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:56:17,688 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:56:17,688 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:56:29,261 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:56:32,824 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:56:32,824 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:56:47,955 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:56:47,955 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:56:59,339 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:57:03,086 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:57:03,086 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:57:18,219 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:57:18,219 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:57:29,416 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:57:33,350 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:57:33,350 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:57:48,482 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:57:48,483 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:57:59,490 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:58:03,614 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:58:03,614 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:58:17,674 INFO    Thread-8  :624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log
2021-07-15 00:58:18,763 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:58:18,763 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:58:29,567 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:58:33,909 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:58:33,910 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:58:49,058 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:58:49,059 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:58:59,648 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:59:04,188 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:59:04,188 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:59:19,321 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:59:19,322 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:59:29,729 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:59:34,455 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:59:34,455 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:59:41,706 INFO    Thread-8  :624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log
2021-07-15 00:59:43,394 DEBUG   SenderThread:624595 [sender.py:send():179] send: telemetry
2021-07-15 00:59:43,394 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:43,395 DEBUG   SenderThread:624595 [sender.py:send():179] send: exit
2021-07-15 00:59:43,395 INFO    SenderThread:624595 [sender.py:send_exit():287] handling exit code: 1
2021-07-15 00:59:43,395 INFO    SenderThread:624595 [sender.py:send_exit():295] send defer
2021-07-15 00:59:43,395 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:43,396 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,396 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-15 00:59:43,396 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,396 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-15 00:59:43,396 INFO    SenderThread:624595 [sender.py:transition_state():308] send defer: 1
2021-07-15 00:59:43,396 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,396 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-15 00:59:43,467 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,467 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-15 00:59:43,467 INFO    SenderThread:624595 [sender.py:transition_state():308] send defer: 2
2021-07-15 00:59:43,467 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:59:43,467 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,467 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-15 00:59:43,468 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,468 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-15 00:59:43,468 INFO    SenderThread:624595 [sender.py:transition_state():308] send defer: 3
2021-07-15 00:59:43,468 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,468 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-15 00:59:43,468 DEBUG   SenderThread:624595 [sender.py:send():179] send: summary
2021-07-15 00:59:43,468 INFO    SenderThread:624595 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 00:59:43,468 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,469 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-15 00:59:43,469 INFO    SenderThread:624595 [sender.py:transition_state():308] send defer: 4
2021-07-15 00:59:43,469 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,469 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-15 00:59:43,469 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,469 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-15 00:59:43,498 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:43,675 INFO    SenderThread:624595 [sender.py:transition_state():308] send defer: 5
2021-07-15 00:59:43,675 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:43,676 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,676 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-15 00:59:43,676 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,676 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-15 00:59:43,676 INFO    SenderThread:624595 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-15 00:59:43,707 INFO    SenderThread:624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/config.yaml
2021-07-15 00:59:43,708 INFO    SenderThread:624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log
2021-07-15 00:59:43,708 INFO    SenderThread:624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/wandb-summary.json
2021-07-15 00:59:43,708 INFO    SenderThread:624595 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files
2021-07-15 00:59:43,708 INFO    SenderThread:624595 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/requirements.txt requirements.txt
2021-07-15 00:59:43,708 INFO    SenderThread:624595 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log output.log
2021-07-15 00:59:43,708 INFO    SenderThread:624595 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/wandb-metadata.json wandb-metadata.json
2021-07-15 00:59:43,708 INFO    SenderThread:624595 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/config.yaml config.yaml
2021-07-15 00:59:43,708 INFO    SenderThread:624595 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/wandb-summary.json wandb-summary.json
2021-07-15 00:59:43,709 INFO    SenderThread:624595 [sender.py:transition_state():308] send defer: 6
2021-07-15 00:59:43,709 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,715 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-15 00:59:43,720 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,720 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-15 00:59:43,721 INFO    SenderThread:624595 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 00:59:43,778 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:43,778 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:43,880 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:43,880 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:43,982 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:43,982 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,084 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:44,084 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,146 INFO    Thread-13 :624595 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log
2021-07-15 00:59:44,161 INFO    Thread-15 :624595 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/wandb-summary.json
2021-07-15 00:59:44,186 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:44,187 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,194 INFO    Thread-12 :624595 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/requirements.txt
2021-07-15 00:59:44,198 INFO    Thread-14 :624595 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/config.yaml
2021-07-15 00:59:44,289 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:44,289 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,391 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:44,391 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,399 INFO    Thread-7  :624595 [sender.py:transition_state():308] send defer: 7
2021-07-15 00:59:44,399 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:44,399 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-15 00:59:44,399 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:44,400 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-15 00:59:44,493 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:44,686 INFO    SenderThread:624595 [sender.py:transition_state():308] send defer: 8
2021-07-15 00:59:44,687 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,687 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:44,687 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-15 00:59:44,687 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:44,688 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-15 00:59:44,688 INFO    SenderThread:624595 [sender.py:transition_state():308] send defer: 9
2021-07-15 00:59:44,688 DEBUG   SenderThread:624595 [sender.py:send():179] send: final
2021-07-15 00:59:44,688 DEBUG   SenderThread:624595 [sender.py:send():179] send: footer
2021-07-15 00:59:44,688 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:44,689 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-15 00:59:44,689 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:44,689 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-15 00:59:44,789 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:44,789 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,789 INFO    SenderThread:624595 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 00:59:44,791 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: get_summary
2021-07-15 00:59:44,791 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-15 00:59:44,792 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: shutdown
2021-07-15 00:59:44,792 INFO    HandlerThread:624595 [handler.py:finish():638] shutting down handler
2021-07-15 00:59:45,689 INFO    WriterThread:624595 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/run-361cvuow.wandb
2021-07-15 00:59:45,789 INFO    SenderThread:624595 [sender.py:finish():945] shutting down sender
2021-07-15 00:59:45,790 INFO    SenderThread:624595 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 00:59:45,790 INFO    SenderThread:624595 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 00:59:45,792 INFO    MainThread:624595 [internal.py:handle_exit():78] Internal process exited