File size: 20,144 Bytes
f291f93
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
2021-07-13 11:20:39,477 INFO    MainThread:350932 [internal.py:wandb_internal():88] W&B internal server running at pid: 350932, started at: 2021-07-13 11:20:39.476738
2021-07-13 11:20:39,479 INFO    WriterThread:350932 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/run-qbdrt0c0.wandb
2021-07-13 11:20:39,479 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: check_version
2021-07-13 11:20:39,480 DEBUG   SenderThread:350932 [sender.py:send():179] send: header
2021-07-13 11:20:39,480 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: check_version
2021-07-13 11:20:39,519 DEBUG   SenderThread:350932 [sender.py:send():179] send: run
2021-07-13 11:20:39,698 INFO    SenderThread:350932 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files
2021-07-13 11:20:39,698 INFO    SenderThread:350932 [sender.py:_start_run_threads():716] run started: qbdrt0c0 with start time 1626175238
2021-07-13 11:20:39,698 DEBUG   SenderThread:350932 [sender.py:send():179] send: summary
2021-07-13 11:20:39,698 INFO    SenderThread:350932 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 11:20:39,699 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: run_start
2021-07-13 11:20:40,702 INFO    Thread-8  :350932 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/wandb-summary.json
2021-07-13 11:20:40,844 DEBUG   HandlerThread:350932 [meta.py:__init__():39] meta init
2021-07-13 11:20:40,845 DEBUG   HandlerThread:350932 [meta.py:__init__():53] meta init done
2021-07-13 11:20:40,845 DEBUG   HandlerThread:350932 [meta.py:probe():210] probe
2021-07-13 11:20:40,846 DEBUG   HandlerThread:350932 [meta.py:_setup_git():200] setup git
2021-07-13 11:20:40,874 DEBUG   HandlerThread:350932 [meta.py:_setup_git():207] setup git done
2021-07-13 11:20:40,874 DEBUG   HandlerThread:350932 [meta.py:_save_pip():57] save pip
2021-07-13 11:20:40,875 DEBUG   HandlerThread:350932 [meta.py:_save_pip():71] save pip done
2021-07-13 11:20:40,875 DEBUG   HandlerThread:350932 [meta.py:probe():252] probe done
2021-07-13 11:20:40,878 DEBUG   SenderThread:350932 [sender.py:send():179] send: files
2021-07-13 11:20:40,878 INFO    SenderThread:350932 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-13 11:20:40,885 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:20:40,886 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:20:41,013 DEBUG   SenderThread:350932 [sender.py:send():179] send: config
2021-07-13 11:20:41,014 DEBUG   SenderThread:350932 [sender.py:send():179] send: config
2021-07-13 11:20:41,014 DEBUG   SenderThread:350932 [sender.py:send():179] send: config
2021-07-13 11:20:41,368 INFO    Thread-11 :350932 [upload_job.py:push():137] Uploaded file /tmp/tmp_o2vfehlwandb/wusrh9hx-wandb-metadata.json
2021-07-13 11:20:41,700 INFO    Thread-8  :350932 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/output.log
2021-07-13 11:20:41,700 INFO    Thread-8  :350932 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/wandb-metadata.json
2021-07-13 11:20:41,700 INFO    Thread-8  :350932 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/requirements.txt
2021-07-13 11:20:55,705 INFO    Thread-8  :350932 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/output.log
2021-07-13 11:20:56,015 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:20:56,016 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:20:57,706 INFO    Thread-8  :350932 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/output.log
2021-07-13 11:21:08,929 DEBUG   SenderThread:350932 [sender.py:send():179] send: stats
2021-07-13 11:21:10,712 INFO    Thread-8  :350932 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/config.yaml
2021-07-13 11:21:11,150 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:21:11,150 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:21:26,282 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:21:26,282 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:21:39,007 DEBUG   SenderThread:350932 [sender.py:send():179] send: stats
2021-07-13 11:21:41,413 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:21:41,414 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:21:56,566 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:21:56,566 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:22:09,085 DEBUG   SenderThread:350932 [sender.py:send():179] send: stats
2021-07-13 11:22:11,700 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:22:11,700 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:22:26,842 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:22:26,842 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:22:27,737 INFO    Thread-8  :350932 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/output.log
2021-07-13 11:22:39,163 DEBUG   SenderThread:350932 [sender.py:send():179] send: stats
2021-07-13 11:22:41,997 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:22:41,997 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:22:57,140 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:22:57,140 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:23:09,248 DEBUG   SenderThread:350932 [sender.py:send():179] send: stats
2021-07-13 11:23:12,271 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:23:12,272 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:23:27,405 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:23:27,405 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:23:39,327 DEBUG   SenderThread:350932 [sender.py:send():179] send: stats
2021-07-13 11:23:42,540 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:23:42,540 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:23:47,766 INFO    Thread-8  :350932 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/output.log
2021-07-13 11:23:49,172 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:49,172 DEBUG   SenderThread:350932 [sender.py:send():179] send: telemetry
2021-07-13 11:23:49,172 DEBUG   SenderThread:350932 [sender.py:send():179] send: exit
2021-07-13 11:23:49,172 INFO    SenderThread:350932 [sender.py:send_exit():287] handling exit code: 1
2021-07-13 11:23:49,173 INFO    SenderThread:350932 [sender.py:send_exit():295] send defer
2021-07-13 11:23:49,173 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:49,173 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:49,174 INFO    HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-13 11:23:49,174 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:49,174 INFO    SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-13 11:23:49,174 INFO    SenderThread:350932 [sender.py:transition_state():308] send defer: 1
2021-07-13 11:23:49,175 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:49,175 INFO    HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-13 11:23:49,253 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:49,254 INFO    SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-13 11:23:49,254 INFO    SenderThread:350932 [sender.py:transition_state():308] send defer: 2
2021-07-13 11:23:49,254 DEBUG   SenderThread:350932 [sender.py:send():179] send: stats
2021-07-13 11:23:49,254 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:49,254 INFO    HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-13 11:23:49,255 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:49,255 INFO    SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-13 11:23:49,255 INFO    SenderThread:350932 [sender.py:transition_state():308] send defer: 3
2021-07-13 11:23:49,255 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:49,255 INFO    HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-13 11:23:49,255 DEBUG   SenderThread:350932 [sender.py:send():179] send: summary
2021-07-13 11:23:49,256 INFO    SenderThread:350932 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 11:23:49,256 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:49,256 INFO    SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-13 11:23:49,256 INFO    SenderThread:350932 [sender.py:transition_state():308] send defer: 4
2021-07-13 11:23:49,256 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:49,256 INFO    HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-13 11:23:49,256 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:49,256 INFO    SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-13 11:23:49,275 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:49,445 INFO    SenderThread:350932 [sender.py:transition_state():308] send defer: 5
2021-07-13 11:23:49,445 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:49,445 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:49,445 INFO    HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-13 11:23:49,446 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:49,446 INFO    SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-13 11:23:49,446 INFO    SenderThread:350932 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-13 11:23:49,547 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:49,766 INFO    Thread-8  :350932 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/config.yaml
2021-07-13 11:23:49,767 INFO    SenderThread:350932 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/output.log
2021-07-13 11:23:49,767 INFO    SenderThread:350932 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/wandb-summary.json
2021-07-13 11:23:49,767 INFO    SenderThread:350932 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files
2021-07-13 11:23:49,767 INFO    SenderThread:350932 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/requirements.txt requirements.txt
2021-07-13 11:23:49,767 INFO    SenderThread:350932 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/output.log output.log
2021-07-13 11:23:49,767 INFO    SenderThread:350932 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/wandb-metadata.json wandb-metadata.json
2021-07-13 11:23:49,767 INFO    SenderThread:350932 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/config.yaml config.yaml
2021-07-13 11:23:49,767 INFO    SenderThread:350932 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/wandb-summary.json wandb-summary.json
2021-07-13 11:23:49,768 INFO    SenderThread:350932 [sender.py:transition_state():308] send defer: 6
2021-07-13 11:23:49,768 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:49,771 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:49,771 INFO    HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-13 11:23:49,771 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:49,772 INFO    SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-13 11:23:49,772 INFO    SenderThread:350932 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 11:23:49,873 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:49,873 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:49,975 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:49,976 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:50,078 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:50,078 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:50,180 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:50,180 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:50,215 INFO    Thread-15 :350932 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/wandb-summary.json
2021-07-13 11:23:50,216 INFO    Thread-14 :350932 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/config.yaml
2021-07-13 11:23:50,221 INFO    Thread-12 :350932 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/requirements.txt
2021-07-13 11:23:50,225 INFO    Thread-13 :350932 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/output.log
2021-07-13 11:23:50,282 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:50,282 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:50,384 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:50,384 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:50,425 INFO    Thread-7  :350932 [sender.py:transition_state():308] send defer: 7
2021-07-13 11:23:50,426 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:50,426 INFO    HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-13 11:23:50,426 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:50,426 INFO    SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-13 11:23:50,486 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:50,554 INFO    SenderThread:350932 [sender.py:transition_state():308] send defer: 8
2021-07-13 11:23:50,554 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:50,555 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:50,555 INFO    HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-13 11:23:50,555 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:50,555 INFO    SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-13 11:23:50,555 INFO    SenderThread:350932 [sender.py:transition_state():308] send defer: 9
2021-07-13 11:23:50,556 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:50,556 INFO    HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-13 11:23:50,556 DEBUG   SenderThread:350932 [sender.py:send():179] send: final
2021-07-13 11:23:50,556 DEBUG   SenderThread:350932 [sender.py:send():179] send: footer
2021-07-13 11:23:50,556 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:50,556 INFO    SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-13 11:23:50,656 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:50,656 DEBUG   SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:50,657 INFO    SenderThread:350932 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 11:23:50,658 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: get_summary
2021-07-13 11:23:50,659 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-13 11:23:50,659 DEBUG   HandlerThread:350932 [handler.py:handle_request():124] handle_request: shutdown
2021-07-13 11:23:50,660 INFO    HandlerThread:350932 [handler.py:finish():638] shutting down handler
2021-07-13 11:23:51,557 INFO    WriterThread:350932 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/run-qbdrt0c0.wandb
2021-07-13 11:23:51,657 INFO    SenderThread:350932 [sender.py:finish():945] shutting down sender
2021-07-13 11:23:51,657 INFO    SenderThread:350932 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 11:23:51,657 INFO    SenderThread:350932 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 11:23:51,659 INFO    MainThread:350932 [internal.py:handle_exit():78] Internal process exited