File size: 20,580 Bytes
f291f93
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
2021-07-13 11:11:12,514 INFO    MainThread:348072 [internal.py:wandb_internal():88] W&B internal server running at pid: 348072, started at: 2021-07-13 11:11:12.514575
2021-07-13 11:11:12,516 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: check_version
2021-07-13 11:11:12,517 INFO    WriterThread:348072 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/run-3cwslju0.wandb
2021-07-13 11:11:12,518 DEBUG   SenderThread:348072 [sender.py:send():179] send: header
2021-07-13 11:11:12,518 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: check_version
2021-07-13 11:11:12,553 DEBUG   SenderThread:348072 [sender.py:send():179] send: run
2021-07-13 11:11:12,741 INFO    SenderThread:348072 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files
2021-07-13 11:11:12,741 INFO    SenderThread:348072 [sender.py:_start_run_threads():716] run started: 3cwslju0 with start time 1626174671
2021-07-13 11:11:12,741 DEBUG   SenderThread:348072 [sender.py:send():179] send: summary
2021-07-13 11:11:12,742 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: run_start
2021-07-13 11:11:12,743 INFO    SenderThread:348072 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 11:11:13,745 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/wandb-summary.json
2021-07-13 11:11:14,000 DEBUG   HandlerThread:348072 [meta.py:__init__():39] meta init
2021-07-13 11:11:14,000 DEBUG   HandlerThread:348072 [meta.py:__init__():53] meta init done
2021-07-13 11:11:14,000 DEBUG   HandlerThread:348072 [meta.py:probe():210] probe
2021-07-13 11:11:14,001 DEBUG   HandlerThread:348072 [meta.py:_setup_git():200] setup git
2021-07-13 11:11:14,031 DEBUG   HandlerThread:348072 [meta.py:_setup_git():207] setup git done
2021-07-13 11:11:14,031 DEBUG   HandlerThread:348072 [meta.py:_save_pip():57] save pip
2021-07-13 11:11:14,031 DEBUG   HandlerThread:348072 [meta.py:_save_pip():71] save pip done
2021-07-13 11:11:14,031 DEBUG   HandlerThread:348072 [meta.py:probe():252] probe done
2021-07-13 11:11:14,035 DEBUG   SenderThread:348072 [sender.py:send():179] send: files
2021-07-13 11:11:14,035 INFO    SenderThread:348072 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-13 11:11:14,043 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:11:14,043 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:11:14,170 DEBUG   SenderThread:348072 [sender.py:send():179] send: config
2021-07-13 11:11:14,170 DEBUG   SenderThread:348072 [sender.py:send():179] send: config
2021-07-13 11:11:14,170 DEBUG   SenderThread:348072 [sender.py:send():179] send: config
2021-07-13 11:11:14,515 INFO    Thread-11 :348072 [upload_job.py:push():137] Uploaded file /tmp/tmp64u985tqwandb/1u3tchat-wandb-metadata.json
2021-07-13 11:11:14,745 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/wandb-metadata.json
2021-07-13 11:11:14,745 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:11:14,745 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/requirements.txt
2021-07-13 11:11:28,753 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:11:29,172 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:11:29,172 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:11:30,754 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:11:42,078 DEBUG   SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:11:43,760 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/config.yaml
2021-07-13 11:11:44,303 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:11:44,304 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:11:59,435 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:11:59,436 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:12:12,149 DEBUG   SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:12:14,568 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:12:14,569 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:12:29,703 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:12:29,704 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:12:42,213 DEBUG   SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:12:44,838 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:12:44,839 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:12:59,973 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:12:59,974 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:13:06,802 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:13:12,281 DEBUG   SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:13:15,129 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:13:15,129 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:13:30,277 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:13:30,277 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:13:42,360 DEBUG   SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:13:45,408 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:13:45,409 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:14:00,541 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:14:00,555 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:14:12,440 DEBUG   SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:14:15,687 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:14:15,687 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:14:26,833 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:14:27,502 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:27,503 DEBUG   SenderThread:348072 [sender.py:send():179] send: telemetry
2021-07-13 11:14:27,503 DEBUG   SenderThread:348072 [sender.py:send():179] send: exit
2021-07-13 11:14:27,503 INFO    SenderThread:348072 [sender.py:send_exit():287] handling exit code: 1
2021-07-13 11:14:27,503 INFO    SenderThread:348072 [sender.py:send_exit():295] send defer
2021-07-13 11:14:27,504 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:27,504 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:27,504 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-13 11:14:27,504 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:27,505 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-13 11:14:27,505 INFO    SenderThread:348072 [sender.py:transition_state():308] send defer: 1
2021-07-13 11:14:27,505 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:27,505 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-13 11:14:27,574 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:27,574 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-13 11:14:27,574 INFO    SenderThread:348072 [sender.py:transition_state():308] send defer: 2
2021-07-13 11:14:27,575 DEBUG   SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:14:27,575 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:27,575 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-13 11:14:27,576 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:27,576 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-13 11:14:27,576 INFO    SenderThread:348072 [sender.py:transition_state():308] send defer: 3
2021-07-13 11:14:27,576 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:27,576 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-13 11:14:27,576 DEBUG   SenderThread:348072 [sender.py:send():179] send: summary
2021-07-13 11:14:27,606 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:27,704 INFO    SenderThread:348072 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 11:14:27,705 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:27,705 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-13 11:14:27,705 INFO    SenderThread:348072 [sender.py:transition_state():308] send defer: 4
2021-07-13 11:14:27,705 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:27,706 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:27,706 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-13 11:14:27,706 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:27,706 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-13 11:14:27,807 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:27,834 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:14:27,834 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/wandb-summary.json
2021-07-13 11:14:27,901 INFO    SenderThread:348072 [sender.py:transition_state():308] send defer: 5
2021-07-13 11:14:27,901 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:27,902 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:27,902 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-13 11:14:27,902 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:27,902 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-13 11:14:27,902 INFO    SenderThread:348072 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-13 11:14:28,004 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:28,834 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/config.yaml
2021-07-13 11:14:28,836 INFO    SenderThread:348072 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files
2021-07-13 11:14:28,836 INFO    SenderThread:348072 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/requirements.txt requirements.txt
2021-07-13 11:14:28,836 INFO    SenderThread:348072 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log output.log
2021-07-13 11:14:28,836 INFO    SenderThread:348072 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/wandb-metadata.json wandb-metadata.json
2021-07-13 11:14:28,836 INFO    SenderThread:348072 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/config.yaml config.yaml
2021-07-13 11:14:28,836 INFO    SenderThread:348072 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/wandb-summary.json wandb-summary.json
2021-07-13 11:14:28,837 INFO    SenderThread:348072 [sender.py:transition_state():308] send defer: 6
2021-07-13 11:14:28,837 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:28,840 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:28,840 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-13 11:14:28,843 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:28,844 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-13 11:14:28,844 INFO    SenderThread:348072 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 11:14:28,943 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:28,943 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,046 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,046 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,148 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,149 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,251 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,251 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,300 INFO    Thread-14 :348072 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/config.yaml
2021-07-13 11:14:29,309 INFO    Thread-12 :348072 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/requirements.txt
2021-07-13 11:14:29,332 INFO    Thread-15 :348072 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/wandb-summary.json
2021-07-13 11:14:29,353 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,354 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,401 INFO    Thread-13 :348072 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:14:29,456 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,479 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,581 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,582 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,679 INFO    Thread-7  :348072 [sender.py:transition_state():308] send defer: 7
2021-07-13 11:14:29,680 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:29,680 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-13 11:14:29,680 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:29,680 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-13 11:14:29,684 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,949 INFO    SenderThread:348072 [sender.py:transition_state():308] send defer: 8
2021-07-13 11:14:29,949 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,950 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:29,950 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-13 11:14:29,950 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:29,950 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-13 11:14:29,950 INFO    SenderThread:348072 [sender.py:transition_state():308] send defer: 9
2021-07-13 11:14:29,951 DEBUG   SenderThread:348072 [sender.py:send():179] send: final
2021-07-13 11:14:29,951 DEBUG   SenderThread:348072 [sender.py:send():179] send: footer
2021-07-13 11:14:29,951 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:29,951 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-13 11:14:29,951 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:29,951 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-13 11:14:30,051 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:30,052 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:30,052 INFO    SenderThread:348072 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 11:14:30,054 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: get_summary
2021-07-13 11:14:30,055 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-13 11:14:30,055 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: shutdown
2021-07-13 11:14:30,056 INFO    HandlerThread:348072 [handler.py:finish():638] shutting down handler
2021-07-13 11:14:30,951 INFO    WriterThread:348072 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/run-3cwslju0.wandb
2021-07-13 11:14:31,052 INFO    SenderThread:348072 [sender.py:finish():945] shutting down sender
2021-07-13 11:14:31,052 INFO    SenderThread:348072 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 11:14:31,052 INFO    SenderThread:348072 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 11:14:31,054 INFO    MainThread:348072 [internal.py:handle_exit():78] Internal process exited