File size: 25,306 Bytes
731244f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 |
2021-07-15 00:28:10,840 INFO MainThread:620043 [internal.py:wandb_internal():88] W&B internal server running at pid: 620043, started at: 2021-07-15 00:28:10.839711
2021-07-15 00:28:10,842 INFO WriterThread:620043 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/run-22keyfxg.wandb
2021-07-15 00:28:10,842 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: check_version
2021-07-15 00:28:10,843 DEBUG SenderThread:620043 [sender.py:send():179] send: header
2021-07-15 00:28:10,843 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: check_version
2021-07-15 00:28:10,882 DEBUG SenderThread:620043 [sender.py:send():179] send: run
2021-07-15 00:28:11,065 INFO SenderThread:620043 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files
2021-07-15 00:28:11,065 INFO SenderThread:620043 [sender.py:_start_run_threads():716] run started: 22keyfxg with start time 1626308890
2021-07-15 00:28:11,065 DEBUG SenderThread:620043 [sender.py:send():179] send: summary
2021-07-15 00:28:11,065 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: run_start
2021-07-15 00:28:11,066 INFO SenderThread:620043 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 00:28:12,069 INFO Thread-8 :620043 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/wandb-summary.json
2021-07-15 00:28:12,177 DEBUG HandlerThread:620043 [meta.py:__init__():39] meta init
2021-07-15 00:28:12,177 DEBUG HandlerThread:620043 [meta.py:__init__():53] meta init done
2021-07-15 00:28:12,177 DEBUG HandlerThread:620043 [meta.py:probe():210] probe
2021-07-15 00:28:12,178 DEBUG HandlerThread:620043 [meta.py:_setup_git():200] setup git
2021-07-15 00:28:12,205 DEBUG HandlerThread:620043 [meta.py:_setup_git():207] setup git done
2021-07-15 00:28:12,205 DEBUG HandlerThread:620043 [meta.py:_save_pip():57] save pip
2021-07-15 00:28:12,206 DEBUG HandlerThread:620043 [meta.py:_save_pip():71] save pip done
2021-07-15 00:28:12,206 DEBUG HandlerThread:620043 [meta.py:probe():252] probe done
2021-07-15 00:28:12,209 DEBUG SenderThread:620043 [sender.py:send():179] send: files
2021-07-15 00:28:12,209 INFO SenderThread:620043 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-15 00:28:12,217 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:28:12,217 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:28:12,363 DEBUG SenderThread:620043 [sender.py:send():179] send: config
2021-07-15 00:28:12,363 DEBUG SenderThread:620043 [sender.py:send():179] send: config
2021-07-15 00:28:12,364 DEBUG SenderThread:620043 [sender.py:send():179] send: config
2021-07-15 00:28:12,702 INFO Thread-11 :620043 [upload_job.py:push():137] Uploaded file /tmp/tmpkt2gzisdwandb/opsxvzfj-wandb-metadata.json
2021-07-15 00:28:13,068 INFO Thread-8 :620043 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/requirements.txt
2021-07-15 00:28:13,068 INFO Thread-8 :620043 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/output.log
2021-07-15 00:28:13,069 INFO Thread-8 :620043 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/wandb-metadata.json
2021-07-15 00:28:27,074 INFO Thread-8 :620043 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/output.log
2021-07-15 00:28:27,365 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:28:27,366 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:28:40,252 DEBUG SenderThread:620043 [sender.py:send():179] send: stats
2021-07-15 00:28:42,080 INFO Thread-8 :620043 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/config.yaml
2021-07-15 00:28:42,496 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:28:42,497 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:28:57,628 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:28:57,628 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:29:10,324 DEBUG SenderThread:620043 [sender.py:send():179] send: stats
2021-07-15 00:29:12,761 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:29:12,762 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:29:27,895 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:29:27,896 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:29:40,392 DEBUG SenderThread:620043 [sender.py:send():179] send: stats
2021-07-15 00:29:43,029 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:29:43,029 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:29:58,161 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:29:58,161 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:30:10,458 DEBUG SenderThread:620043 [sender.py:send():179] send: stats
2021-07-15 00:30:13,295 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:30:13,296 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:30:28,431 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:30:28,431 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:30:40,524 DEBUG SenderThread:620043 [sender.py:send():179] send: stats
2021-07-15 00:30:43,564 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:30:43,564 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:30:58,695 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:30:58,695 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:31:10,591 DEBUG SenderThread:620043 [sender.py:send():179] send: stats
2021-07-15 00:31:13,829 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:31:13,829 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:31:28,959 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:31:28,960 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:31:40,658 DEBUG SenderThread:620043 [sender.py:send():179] send: stats
2021-07-15 00:31:44,094 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:31:44,095 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:31:59,226 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:31:59,227 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:32:10,721 DEBUG SenderThread:620043 [sender.py:send():179] send: stats
2021-07-15 00:32:14,364 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:32:14,364 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:32:29,496 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:32:29,497 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:32:40,792 DEBUG SenderThread:620043 [sender.py:send():179] send: stats
2021-07-15 00:32:44,631 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:32:44,632 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:32:59,762 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:32:59,762 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:33:10,860 DEBUG SenderThread:620043 [sender.py:send():179] send: stats
2021-07-15 00:33:14,895 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:33:14,895 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:33:30,027 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:33:30,027 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:33:40,933 DEBUG SenderThread:620043 [sender.py:send():179] send: stats
2021-07-15 00:33:45,162 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:33:45,162 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:34:00,292 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:34:00,293 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:34:11,005 DEBUG SenderThread:620043 [sender.py:send():179] send: stats
2021-07-15 00:34:15,424 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:34:15,425 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:34:30,556 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:34:30,556 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:34:41,080 DEBUG SenderThread:620043 [sender.py:send():179] send: stats
2021-07-15 00:34:45,687 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:34:45,688 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:34:57,238 INFO Thread-8 :620043 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/output.log
2021-07-15 00:35:00,957 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:35:00,957 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:35:11,161 DEBUG SenderThread:620043 [sender.py:send():179] send: stats
2021-07-15 00:35:16,118 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:35:16,119 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:35:31,255 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:35:31,255 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:35:41,236 DEBUG SenderThread:620043 [sender.py:send():179] send: stats
2021-07-15 00:35:46,389 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:35:46,390 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:36:01,522 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:36:01,522 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:36:11,313 DEBUG SenderThread:620043 [sender.py:send():179] send: stats
2021-07-15 00:36:16,658 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:36:16,658 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:36:19,273 INFO Thread-8 :620043 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/output.log
2021-07-15 00:36:20,774 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:36:20,775 DEBUG SenderThread:620043 [sender.py:send():179] send: telemetry
2021-07-15 00:36:20,775 DEBUG SenderThread:620043 [sender.py:send():179] send: exit
2021-07-15 00:36:20,775 INFO SenderThread:620043 [sender.py:send_exit():287] handling exit code: 1
2021-07-15 00:36:20,775 INFO SenderThread:620043 [sender.py:send_exit():295] send defer
2021-07-15 00:36:20,776 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:36:20,776 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:36:20,776 INFO HandlerThread:620043 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-15 00:36:20,777 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: defer
2021-07-15 00:36:20,777 INFO SenderThread:620043 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-15 00:36:20,777 INFO SenderThread:620043 [sender.py:transition_state():308] send defer: 1
2021-07-15 00:36:20,777 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:36:20,777 INFO HandlerThread:620043 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-15 00:36:20,837 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: defer
2021-07-15 00:36:20,837 INFO SenderThread:620043 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-15 00:36:20,837 INFO SenderThread:620043 [sender.py:transition_state():308] send defer: 2
2021-07-15 00:36:20,837 DEBUG SenderThread:620043 [sender.py:send():179] send: stats
2021-07-15 00:36:20,843 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:36:20,843 INFO HandlerThread:620043 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-15 00:36:20,843 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: defer
2021-07-15 00:36:20,844 INFO SenderThread:620043 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-15 00:36:20,844 INFO SenderThread:620043 [sender.py:transition_state():308] send defer: 3
2021-07-15 00:36:20,844 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:36:20,844 INFO HandlerThread:620043 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-15 00:36:20,844 DEBUG SenderThread:620043 [sender.py:send():179] send: summary
2021-07-15 00:36:20,845 INFO SenderThread:620043 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 00:36:20,846 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: defer
2021-07-15 00:36:20,846 INFO SenderThread:620043 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-15 00:36:20,846 INFO SenderThread:620043 [sender.py:transition_state():308] send defer: 4
2021-07-15 00:36:20,847 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:36:20,847 INFO HandlerThread:620043 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-15 00:36:20,847 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: defer
2021-07-15 00:36:20,847 INFO SenderThread:620043 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-15 00:36:20,878 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:36:21,021 INFO SenderThread:620043 [sender.py:transition_state():308] send defer: 5
2021-07-15 00:36:21,021 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:36:21,022 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:36:21,022 INFO HandlerThread:620043 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-15 00:36:21,022 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: defer
2021-07-15 00:36:21,022 INFO SenderThread:620043 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-15 00:36:21,023 INFO SenderThread:620043 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-15 00:36:21,124 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:36:21,275 INFO SenderThread:620043 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/output.log
2021-07-15 00:36:21,275 INFO SenderThread:620043 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/config.yaml
2021-07-15 00:36:21,275 INFO SenderThread:620043 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/wandb-summary.json
2021-07-15 00:36:21,275 INFO SenderThread:620043 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files
2021-07-15 00:36:21,276 INFO SenderThread:620043 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/requirements.txt requirements.txt
2021-07-15 00:36:21,276 INFO SenderThread:620043 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/output.log output.log
2021-07-15 00:36:21,276 INFO SenderThread:620043 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/wandb-metadata.json wandb-metadata.json
2021-07-15 00:36:21,276 INFO SenderThread:620043 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/config.yaml config.yaml
2021-07-15 00:36:21,280 INFO SenderThread:620043 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/wandb-summary.json wandb-summary.json
2021-07-15 00:36:21,281 INFO SenderThread:620043 [sender.py:transition_state():308] send defer: 6
2021-07-15 00:36:21,281 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:36:21,284 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:36:21,284 INFO HandlerThread:620043 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-15 00:36:21,288 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: defer
2021-07-15 00:36:21,288 INFO SenderThread:620043 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-15 00:36:21,288 INFO SenderThread:620043 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 00:36:21,386 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:36:21,386 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:36:21,488 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:36:21,488 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:36:21,590 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:36:21,590 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:36:21,691 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:36:21,692 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:36:21,711 INFO Thread-12 :620043 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/requirements.txt
2021-07-15 00:36:21,719 INFO Thread-13 :620043 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/output.log
2021-07-15 00:36:21,731 INFO Thread-14 :620043 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/config.yaml
2021-07-15 00:36:21,759 INFO Thread-15 :620043 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/files/wandb-summary.json
2021-07-15 00:36:21,793 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:36:21,793 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:36:21,895 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:36:21,895 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:36:21,959 INFO Thread-7 :620043 [sender.py:transition_state():308] send defer: 7
2021-07-15 00:36:21,960 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:36:21,960 INFO HandlerThread:620043 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-15 00:36:21,960 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: defer
2021-07-15 00:36:21,960 INFO SenderThread:620043 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-15 00:36:21,997 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:36:22,240 INFO SenderThread:620043 [sender.py:transition_state():308] send defer: 8
2021-07-15 00:36:22,241 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:36:22,241 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:36:22,241 INFO HandlerThread:620043 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-15 00:36:22,242 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: defer
2021-07-15 00:36:22,242 INFO SenderThread:620043 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-15 00:36:22,242 INFO SenderThread:620043 [sender.py:transition_state():308] send defer: 9
2021-07-15 00:36:22,242 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:36:22,242 INFO HandlerThread:620043 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-15 00:36:22,243 DEBUG SenderThread:620043 [sender.py:send():179] send: final
2021-07-15 00:36:22,243 DEBUG SenderThread:620043 [sender.py:send():179] send: footer
2021-07-15 00:36:22,243 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: defer
2021-07-15 00:36:22,243 INFO SenderThread:620043 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-15 00:36:22,342 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:36:22,343 DEBUG SenderThread:620043 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:36:22,343 INFO SenderThread:620043 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 00:36:22,344 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: get_summary
2021-07-15 00:36:22,345 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-15 00:36:22,345 DEBUG HandlerThread:620043 [handler.py:handle_request():124] handle_request: shutdown
2021-07-15 00:36:22,345 INFO HandlerThread:620043 [handler.py:finish():638] shutting down handler
2021-07-15 00:36:23,243 INFO WriterThread:620043 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210715_002810-22keyfxg/run-22keyfxg.wandb
2021-07-15 00:36:23,343 INFO SenderThread:620043 [sender.py:finish():945] shutting down sender
2021-07-15 00:36:23,344 INFO SenderThread:620043 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 00:36:23,344 INFO SenderThread:620043 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 00:36:23,346 INFO MainThread:620043 [internal.py:handle_exit():78] Internal process exited
|