File size: 26,598 Bytes
9915204 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 |
2021-07-14 22:29:21,187 INFO MainThread:596546 [internal.py:wandb_internal():88] W&B internal server running at pid: 596546, started at: 2021-07-14 22:29:21.187444
2021-07-14 22:29:21,189 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 22:29:21,189 INFO WriterThread:596546 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/run-2p7mu4rm.wandb
2021-07-14 22:29:21,191 DEBUG SenderThread:596546 [sender.py:send():179] send: header
2021-07-14 22:29:21,191 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: check_version
2021-07-14 22:29:21,226 DEBUG SenderThread:596546 [sender.py:send():179] send: run
2021-07-14 22:29:21,391 INFO SenderThread:596546 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files
2021-07-14 22:29:21,391 INFO SenderThread:596546 [sender.py:_start_run_threads():716] run started: 2p7mu4rm with start time 1626301760
2021-07-14 22:29:21,391 DEBUG SenderThread:596546 [sender.py:send():179] send: summary
2021-07-14 22:29:21,391 INFO SenderThread:596546 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 22:29:21,392 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 22:29:22,397 INFO Thread-8 :596546 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/wandb-summary.json
2021-07-14 22:29:22,511 DEBUG HandlerThread:596546 [meta.py:__init__():39] meta init
2021-07-14 22:29:22,511 DEBUG HandlerThread:596546 [meta.py:__init__():53] meta init done
2021-07-14 22:29:22,511 DEBUG HandlerThread:596546 [meta.py:probe():210] probe
2021-07-14 22:29:22,513 DEBUG HandlerThread:596546 [meta.py:_setup_git():200] setup git
2021-07-14 22:29:22,541 DEBUG HandlerThread:596546 [meta.py:_setup_git():207] setup git done
2021-07-14 22:29:22,541 DEBUG HandlerThread:596546 [meta.py:_save_pip():57] save pip
2021-07-14 22:29:22,542 DEBUG HandlerThread:596546 [meta.py:_save_pip():71] save pip done
2021-07-14 22:29:22,542 DEBUG HandlerThread:596546 [meta.py:probe():252] probe done
2021-07-14 22:29:22,545 DEBUG SenderThread:596546 [sender.py:send():179] send: files
2021-07-14 22:29:22,545 INFO SenderThread:596546 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 22:29:22,551 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:29:22,551 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:29:22,678 DEBUG SenderThread:596546 [sender.py:send():179] send: config
2021-07-14 22:29:22,678 DEBUG SenderThread:596546 [sender.py:send():179] send: config
2021-07-14 22:29:22,679 DEBUG SenderThread:596546 [sender.py:send():179] send: config
2021-07-14 22:29:22,981 INFO Thread-11 :596546 [upload_job.py:push():137] Uploaded file /tmp/tmpkw6g32phwandb/2nns5d67-wandb-metadata.json
2021-07-14 22:29:23,396 INFO Thread-8 :596546 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/requirements.txt
2021-07-14 22:29:23,396 INFO Thread-8 :596546 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/wandb-metadata.json
2021-07-14 22:29:23,397 INFO Thread-8 :596546 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/output.log
2021-07-14 22:29:37,401 INFO Thread-8 :596546 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/output.log
2021-07-14 22:29:37,681 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:29:37,681 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:29:50,595 DEBUG SenderThread:596546 [sender.py:send():179] send: stats
2021-07-14 22:29:52,407 INFO Thread-8 :596546 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/config.yaml
2021-07-14 22:29:52,815 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:29:52,815 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:30:07,946 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:30:07,947 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:30:20,679 DEBUG SenderThread:596546 [sender.py:send():179] send: stats
2021-07-14 22:30:23,081 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:30:23,081 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:30:38,211 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:30:38,212 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:30:50,744 DEBUG SenderThread:596546 [sender.py:send():179] send: stats
2021-07-14 22:30:53,343 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:30:53,344 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:31:08,475 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:31:08,476 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:31:20,817 DEBUG SenderThread:596546 [sender.py:send():179] send: stats
2021-07-14 22:31:23,611 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:31:23,611 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:31:38,742 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:31:38,742 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:31:50,892 DEBUG SenderThread:596546 [sender.py:send():179] send: stats
2021-07-14 22:31:53,876 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:31:53,876 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:32:09,009 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:32:09,010 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:32:20,968 DEBUG SenderThread:596546 [sender.py:send():179] send: stats
2021-07-14 22:32:24,154 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:32:24,154 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:32:39,289 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:32:39,289 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:32:51,042 DEBUG SenderThread:596546 [sender.py:send():179] send: stats
2021-07-14 22:32:54,420 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:32:54,420 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:33:09,552 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:33:09,552 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:33:21,119 DEBUG SenderThread:596546 [sender.py:send():179] send: stats
2021-07-14 22:33:24,688 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:33:24,689 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:33:39,824 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:33:39,825 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:33:51,197 DEBUG SenderThread:596546 [sender.py:send():179] send: stats
2021-07-14 22:33:54,955 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:33:54,955 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:34:10,085 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:34:10,086 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:34:21,275 DEBUG SenderThread:596546 [sender.py:send():179] send: stats
2021-07-14 22:34:25,221 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:34:25,221 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:34:40,360 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:34:40,360 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:34:51,349 DEBUG SenderThread:596546 [sender.py:send():179] send: stats
2021-07-14 22:34:55,491 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:34:55,491 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:35:10,620 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:35:10,621 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:35:21,421 DEBUG SenderThread:596546 [sender.py:send():179] send: stats
2021-07-14 22:35:25,755 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:35:25,755 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:35:40,915 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:35:40,916 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:35:51,496 DEBUG SenderThread:596546 [sender.py:send():179] send: stats
2021-07-14 22:35:56,049 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:35:56,049 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:36:11,183 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:36:11,184 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:36:21,575 DEBUG SenderThread:596546 [sender.py:send():179] send: stats
2021-07-14 22:36:26,315 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:36:26,316 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:36:41,448 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:36:41,448 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:36:51,652 DEBUG SenderThread:596546 [sender.py:send():179] send: stats
2021-07-14 22:36:56,580 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:36:56,581 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:37:11,712 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:37:11,712 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:37:21,566 INFO Thread-8 :596546 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/output.log
2021-07-14 22:37:21,727 DEBUG SenderThread:596546 [sender.py:send():179] send: stats
2021-07-14 22:37:27,049 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:37:27,050 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:37:42,194 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:37:42,194 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:37:51,805 DEBUG SenderThread:596546 [sender.py:send():179] send: stats
2021-07-14 22:37:57,327 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:37:57,327 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:38:12,463 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:38:12,464 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:38:21,882 DEBUG SenderThread:596546 [sender.py:send():179] send: stats
2021-07-14 22:38:27,596 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:38:27,596 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:38:42,728 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:38:42,728 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:38:45,598 INFO Thread-8 :596546 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/output.log
2021-07-14 22:38:45,823 DEBUG SenderThread:596546 [sender.py:send():179] send: telemetry
2021-07-14 22:38:45,823 DEBUG SenderThread:596546 [sender.py:send():179] send: exit
2021-07-14 22:38:45,823 INFO SenderThread:596546 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 22:38:45,824 INFO SenderThread:596546 [sender.py:send_exit():295] send defer
2021-07-14 22:38:45,824 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:38:45,825 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:38:45,825 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:38:45,825 INFO HandlerThread:596546 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 22:38:45,825 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: defer
2021-07-14 22:38:45,825 INFO SenderThread:596546 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 22:38:45,825 INFO SenderThread:596546 [sender.py:transition_state():308] send defer: 1
2021-07-14 22:38:45,826 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:38:45,826 INFO HandlerThread:596546 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 22:38:45,857 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: defer
2021-07-14 22:38:45,857 INFO SenderThread:596546 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 22:38:45,857 INFO SenderThread:596546 [sender.py:transition_state():308] send defer: 2
2021-07-14 22:38:45,857 DEBUG SenderThread:596546 [sender.py:send():179] send: stats
2021-07-14 22:38:45,857 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:38:45,858 INFO HandlerThread:596546 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 22:38:45,858 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: defer
2021-07-14 22:38:45,858 INFO SenderThread:596546 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 22:38:45,858 INFO SenderThread:596546 [sender.py:transition_state():308] send defer: 3
2021-07-14 22:38:45,859 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:38:45,859 INFO HandlerThread:596546 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 22:38:45,859 DEBUG SenderThread:596546 [sender.py:send():179] send: summary
2021-07-14 22:38:45,859 INFO SenderThread:596546 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 22:38:45,859 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: defer
2021-07-14 22:38:45,859 INFO SenderThread:596546 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 22:38:45,859 INFO SenderThread:596546 [sender.py:transition_state():308] send defer: 4
2021-07-14 22:38:45,860 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:38:45,860 INFO HandlerThread:596546 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 22:38:45,860 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: defer
2021-07-14 22:38:45,860 INFO SenderThread:596546 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 22:38:45,927 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:38:46,024 INFO SenderThread:596546 [sender.py:transition_state():308] send defer: 5
2021-07-14 22:38:46,024 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:38:46,024 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:38:46,024 INFO HandlerThread:596546 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 22:38:46,025 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: defer
2021-07-14 22:38:46,025 INFO SenderThread:596546 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 22:38:46,025 INFO SenderThread:596546 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 22:38:46,126 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:38:46,598 INFO Thread-8 :596546 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/wandb-summary.json
2021-07-14 22:38:46,599 INFO SenderThread:596546 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/config.yaml
2021-07-14 22:38:46,599 INFO SenderThread:596546 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/output.log
2021-07-14 22:38:46,599 INFO SenderThread:596546 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files
2021-07-14 22:38:46,599 INFO SenderThread:596546 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/requirements.txt requirements.txt
2021-07-14 22:38:46,600 INFO SenderThread:596546 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/output.log output.log
2021-07-14 22:38:46,600 INFO SenderThread:596546 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/wandb-metadata.json wandb-metadata.json
2021-07-14 22:38:46,600 INFO SenderThread:596546 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/config.yaml config.yaml
2021-07-14 22:38:46,600 INFO SenderThread:596546 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/wandb-summary.json wandb-summary.json
2021-07-14 22:38:46,603 INFO SenderThread:596546 [sender.py:transition_state():308] send defer: 6
2021-07-14 22:38:46,604 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:38:46,607 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:38:46,607 INFO HandlerThread:596546 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 22:38:46,608 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: defer
2021-07-14 22:38:46,610 INFO SenderThread:596546 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 22:38:46,610 INFO SenderThread:596546 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 22:38:46,708 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:38:46,709 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:38:46,811 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:38:46,811 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:38:46,913 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:38:46,913 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:38:47,015 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:38:47,015 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:38:47,054 INFO Thread-14 :596546 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/config.yaml
2021-07-14 22:38:47,063 INFO Thread-12 :596546 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/requirements.txt
2021-07-14 22:38:47,074 INFO Thread-13 :596546 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/output.log
2021-07-14 22:38:47,095 INFO Thread-15 :596546 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/files/wandb-summary.json
2021-07-14 22:38:47,117 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:38:47,117 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:38:47,219 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:38:47,219 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:38:47,295 INFO Thread-7 :596546 [sender.py:transition_state():308] send defer: 7
2021-07-14 22:38:47,295 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:38:47,296 INFO HandlerThread:596546 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 22:38:47,296 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: defer
2021-07-14 22:38:47,296 INFO SenderThread:596546 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 22:38:47,321 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:38:47,939 INFO SenderThread:596546 [sender.py:transition_state():308] send defer: 8
2021-07-14 22:38:47,940 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:38:47,940 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:38:47,940 INFO HandlerThread:596546 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 22:38:47,940 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: defer
2021-07-14 22:38:47,941 INFO SenderThread:596546 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 22:38:47,941 INFO SenderThread:596546 [sender.py:transition_state():308] send defer: 9
2021-07-14 22:38:47,941 DEBUG SenderThread:596546 [sender.py:send():179] send: final
2021-07-14 22:38:47,941 DEBUG SenderThread:596546 [sender.py:send():179] send: footer
2021-07-14 22:38:47,942 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:38:47,942 INFO HandlerThread:596546 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 22:38:47,942 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: defer
2021-07-14 22:38:47,942 INFO SenderThread:596546 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 22:38:48,042 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:38:48,042 DEBUG SenderThread:596546 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:38:48,042 INFO SenderThread:596546 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 22:38:48,044 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 22:38:48,044 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 22:38:48,045 DEBUG HandlerThread:596546 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 22:38:48,045 INFO HandlerThread:596546 [handler.py:finish():638] shutting down handler
2021-07-14 22:38:48,942 INFO WriterThread:596546 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_222920-2p7mu4rm/run-2p7mu4rm.wandb
2021-07-14 22:38:49,042 INFO SenderThread:596546 [sender.py:finish():945] shutting down sender
2021-07-14 22:38:49,043 INFO SenderThread:596546 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 22:38:49,043 INFO SenderThread:596546 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 22:38:49,045 INFO MainThread:596546 [internal.py:handle_exit():78] Internal process exited
|