File size: 26,788 Bytes
9915204 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 |
2021-07-14 22:19:21,314 INFO MainThread:593294 [internal.py:wandb_internal():88] W&B internal server running at pid: 593294, started at: 2021-07-14 22:19:21.314432
2021-07-14 22:19:21,317 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 22:19:21,317 INFO WriterThread:593294 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/run-s091gfok.wandb
2021-07-14 22:19:21,318 DEBUG SenderThread:593294 [sender.py:send():179] send: header
2021-07-14 22:19:21,318 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: check_version
2021-07-14 22:19:21,357 DEBUG SenderThread:593294 [sender.py:send():179] send: run
2021-07-14 22:19:21,536 INFO SenderThread:593294 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files
2021-07-14 22:19:21,536 INFO SenderThread:593294 [sender.py:_start_run_threads():716] run started: s091gfok with start time 1626301160
2021-07-14 22:19:21,536 DEBUG SenderThread:593294 [sender.py:send():179] send: summary
2021-07-14 22:19:21,537 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 22:19:21,537 INFO SenderThread:593294 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 22:19:22,539 INFO Thread-8 :593294 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/wandb-summary.json
2021-07-14 22:19:22,632 DEBUG HandlerThread:593294 [meta.py:__init__():39] meta init
2021-07-14 22:19:22,632 DEBUG HandlerThread:593294 [meta.py:__init__():53] meta init done
2021-07-14 22:19:22,632 DEBUG HandlerThread:593294 [meta.py:probe():210] probe
2021-07-14 22:19:22,634 DEBUG HandlerThread:593294 [meta.py:_setup_git():200] setup git
2021-07-14 22:19:22,663 DEBUG HandlerThread:593294 [meta.py:_setup_git():207] setup git done
2021-07-14 22:19:22,663 DEBUG HandlerThread:593294 [meta.py:_save_pip():57] save pip
2021-07-14 22:19:22,664 DEBUG HandlerThread:593294 [meta.py:_save_pip():71] save pip done
2021-07-14 22:19:22,664 DEBUG HandlerThread:593294 [meta.py:probe():252] probe done
2021-07-14 22:19:22,667 DEBUG SenderThread:593294 [sender.py:send():179] send: files
2021-07-14 22:19:22,667 INFO SenderThread:593294 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 22:19:22,674 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:19:22,674 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:19:22,802 DEBUG SenderThread:593294 [sender.py:send():179] send: config
2021-07-14 22:19:22,803 DEBUG SenderThread:593294 [sender.py:send():179] send: config
2021-07-14 22:19:22,803 DEBUG SenderThread:593294 [sender.py:send():179] send: config
2021-07-14 22:19:23,119 INFO Thread-11 :593294 [upload_job.py:push():137] Uploaded file /tmp/tmpn0n6xzzmwandb/2vhpic31-wandb-metadata.json
2021-07-14 22:19:23,537 INFO Thread-8 :593294 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/wandb-metadata.json
2021-07-14 22:19:23,537 INFO Thread-8 :593294 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/requirements.txt
2021-07-14 22:19:23,538 INFO Thread-8 :593294 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log
2021-07-14 22:19:37,543 INFO Thread-8 :593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log
2021-07-14 22:19:37,804 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:19:37,804 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:19:39,545 INFO Thread-8 :593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log
2021-07-14 22:19:50,715 DEBUG SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:19:52,550 INFO Thread-8 :593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/config.yaml
2021-07-14 22:19:52,936 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:19:52,936 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:20:08,079 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:20:08,080 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:20:20,789 DEBUG SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:20:23,215 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:20:23,215 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:20:38,362 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:20:38,363 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:20:50,861 DEBUG SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:20:53,496 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:20:53,496 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:21:08,625 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:21:08,625 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:21:20,932 DEBUG SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:21:23,756 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:21:23,757 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:21:38,885 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:21:38,886 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:21:50,997 DEBUG SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:21:54,016 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:21:54,016 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:22:09,146 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:22:09,147 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:22:21,114 DEBUG SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:22:24,279 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:22:24,279 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:22:39,412 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:22:39,413 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:22:51,192 DEBUG SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:22:54,548 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:22:54,548 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:23:09,678 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:23:09,678 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:23:21,267 DEBUG SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:23:24,814 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:23:24,814 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:23:39,949 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:23:39,949 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:23:51,337 DEBUG SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:23:55,081 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:23:55,082 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:24:10,212 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:24:10,212 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:24:21,405 DEBUG SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:24:25,345 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:24:25,346 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:24:40,483 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:24:40,483 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:24:51,475 DEBUG SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:24:55,615 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:24:55,615 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:25:10,746 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:25:10,746 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:25:21,548 DEBUG SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:25:25,876 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:25:25,876 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:25:41,015 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:25:41,016 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:25:51,619 DEBUG SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:25:56,148 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:25:56,148 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:26:11,280 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:26:11,280 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:26:21,695 DEBUG SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:26:26,412 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:26:26,413 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:26:41,546 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:26:41,547 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:26:51,772 DEBUG SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:26:56,683 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:26:56,683 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:27:11,816 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:27:11,816 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:27:21,849 DEBUG SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:27:26,950 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:27:26,950 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:27:29,710 INFO Thread-8 :593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log
2021-07-14 22:27:42,097 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:27:42,097 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:27:51,925 DEBUG SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:27:57,249 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:27:57,250 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:28:12,383 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:28:12,384 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:28:22,007 DEBUG SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:28:27,521 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:28:27,522 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:28:42,658 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:28:42,658 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:28:49,741 INFO Thread-8 :593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log
2021-07-14 22:28:51,116 DEBUG SenderThread:593294 [sender.py:send():179] send: telemetry
2021-07-14 22:28:51,116 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:51,116 DEBUG SenderThread:593294 [sender.py:send():179] send: exit
2021-07-14 22:28:51,116 INFO SenderThread:593294 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 22:28:51,117 INFO SenderThread:593294 [sender.py:send_exit():295] send defer
2021-07-14 22:28:51,117 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:51,118 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:51,118 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 22:28:51,118 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:51,118 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 22:28:51,118 INFO SenderThread:593294 [sender.py:transition_state():308] send defer: 1
2021-07-14 22:28:51,118 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:51,118 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 22:28:51,182 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:51,182 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 22:28:51,183 INFO SenderThread:593294 [sender.py:transition_state():308] send defer: 2
2021-07-14 22:28:51,183 DEBUG SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:28:51,183 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:51,183 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 22:28:51,184 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:51,184 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 22:28:51,184 INFO SenderThread:593294 [sender.py:transition_state():308] send defer: 3
2021-07-14 22:28:51,184 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:51,184 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 22:28:51,184 DEBUG SenderThread:593294 [sender.py:send():179] send: summary
2021-07-14 22:28:51,185 INFO SenderThread:593294 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 22:28:51,185 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:51,185 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 22:28:51,185 INFO SenderThread:593294 [sender.py:transition_state():308] send defer: 4
2021-07-14 22:28:51,185 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:51,185 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 22:28:51,185 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:51,185 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 22:28:51,220 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:51,361 INFO SenderThread:593294 [sender.py:transition_state():308] send defer: 5
2021-07-14 22:28:51,361 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:51,361 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:51,362 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 22:28:51,362 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:51,362 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 22:28:51,362 INFO SenderThread:593294 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 22:28:51,463 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:51,742 INFO Thread-8 :593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/wandb-summary.json
2021-07-14 22:28:51,743 INFO SenderThread:593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/config.yaml
2021-07-14 22:28:51,743 INFO SenderThread:593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log
2021-07-14 22:28:51,743 INFO SenderThread:593294 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files
2021-07-14 22:28:51,743 INFO SenderThread:593294 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/requirements.txt requirements.txt
2021-07-14 22:28:51,744 INFO SenderThread:593294 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log output.log
2021-07-14 22:28:51,744 INFO SenderThread:593294 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/wandb-metadata.json wandb-metadata.json
2021-07-14 22:28:51,744 INFO SenderThread:593294 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/config.yaml config.yaml
2021-07-14 22:28:51,744 INFO SenderThread:593294 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/wandb-summary.json wandb-summary.json
2021-07-14 22:28:51,750 INFO SenderThread:593294 [sender.py:transition_state():308] send defer: 6
2021-07-14 22:28:51,750 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:51,751 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:51,751 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 22:28:51,754 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:51,754 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 22:28:51,754 INFO SenderThread:593294 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 22:28:51,856 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:51,856 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:51,958 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:51,958 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:52,060 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:52,061 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:52,162 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:52,163 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:52,191 INFO Thread-13 :593294 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log
2021-07-14 22:28:52,198 INFO Thread-12 :593294 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/requirements.txt
2021-07-14 22:28:52,200 INFO Thread-14 :593294 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/config.yaml
2021-07-14 22:28:52,212 INFO Thread-15 :593294 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/wandb-summary.json
2021-07-14 22:28:52,264 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:52,265 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:52,366 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:52,366 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:52,413 INFO Thread-7 :593294 [sender.py:transition_state():308] send defer: 7
2021-07-14 22:28:52,413 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:52,413 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 22:28:52,413 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:52,414 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 22:28:52,468 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:52,536 INFO SenderThread:593294 [sender.py:transition_state():308] send defer: 8
2021-07-14 22:28:52,536 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:52,537 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:52,537 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 22:28:52,537 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:52,537 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 22:28:52,537 INFO SenderThread:593294 [sender.py:transition_state():308] send defer: 9
2021-07-14 22:28:52,538 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:52,538 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 22:28:52,538 DEBUG SenderThread:593294 [sender.py:send():179] send: final
2021-07-14 22:28:52,538 DEBUG SenderThread:593294 [sender.py:send():179] send: footer
2021-07-14 22:28:52,538 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:52,538 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 22:28:52,638 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:52,638 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:52,638 INFO SenderThread:593294 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 22:28:52,640 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 22:28:52,640 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 22:28:52,641 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 22:28:52,641 INFO HandlerThread:593294 [handler.py:finish():638] shutting down handler
2021-07-14 22:28:53,538 INFO WriterThread:593294 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/run-s091gfok.wandb
2021-07-14 22:28:53,639 INFO SenderThread:593294 [sender.py:finish():945] shutting down sender
2021-07-14 22:28:53,639 INFO SenderThread:593294 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 22:28:53,639 INFO SenderThread:593294 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 22:28:53,641 INFO MainThread:593294 [internal.py:handle_exit():78] Internal process exited
|