File size: 27,129 Bytes
f291f93 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 |
2021-07-14 18:19:55,251 INFO MainThread:549018 [internal.py:wandb_internal():88] W&B internal server running at pid: 549018, started at: 2021-07-14 18:19:55.251471
2021-07-14 18:19:55,253 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 18:19:55,254 INFO WriterThread:549018 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/run-351hrfn2.wandb
2021-07-14 18:19:55,255 DEBUG SenderThread:549018 [sender.py:send():179] send: header
2021-07-14 18:19:55,255 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: check_version
2021-07-14 18:19:55,291 DEBUG SenderThread:549018 [sender.py:send():179] send: run
2021-07-14 18:19:55,471 INFO SenderThread:549018 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files
2021-07-14 18:19:55,471 INFO SenderThread:549018 [sender.py:_start_run_threads():716] run started: 351hrfn2 with start time 1626286794
2021-07-14 18:19:55,471 DEBUG SenderThread:549018 [sender.py:send():179] send: summary
2021-07-14 18:19:55,471 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 18:19:55,472 INFO SenderThread:549018 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 18:19:56,473 INFO Thread-8 :549018 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/wandb-summary.json
2021-07-14 18:19:56,606 DEBUG HandlerThread:549018 [meta.py:__init__():39] meta init
2021-07-14 18:19:56,606 DEBUG HandlerThread:549018 [meta.py:__init__():53] meta init done
2021-07-14 18:19:56,606 DEBUG HandlerThread:549018 [meta.py:probe():210] probe
2021-07-14 18:19:56,607 DEBUG HandlerThread:549018 [meta.py:_setup_git():200] setup git
2021-07-14 18:19:56,636 DEBUG HandlerThread:549018 [meta.py:_setup_git():207] setup git done
2021-07-14 18:19:56,636 DEBUG HandlerThread:549018 [meta.py:_save_pip():57] save pip
2021-07-14 18:19:56,636 DEBUG HandlerThread:549018 [meta.py:_save_pip():71] save pip done
2021-07-14 18:19:56,636 DEBUG HandlerThread:549018 [meta.py:probe():252] probe done
2021-07-14 18:19:56,640 DEBUG SenderThread:549018 [sender.py:send():179] send: files
2021-07-14 18:19:56,640 INFO SenderThread:549018 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 18:19:56,648 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:19:56,648 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:19:56,776 DEBUG SenderThread:549018 [sender.py:send():179] send: config
2021-07-14 18:19:56,777 DEBUG SenderThread:549018 [sender.py:send():179] send: config
2021-07-14 18:19:56,777 DEBUG SenderThread:549018 [sender.py:send():179] send: config
2021-07-14 18:19:57,091 INFO Thread-11 :549018 [upload_job.py:push():137] Uploaded file /tmp/tmpup88esztwandb/nma268qr-wandb-metadata.json
2021-07-14 18:19:57,473 INFO Thread-8 :549018 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/wandb-metadata.json
2021-07-14 18:19:57,473 INFO Thread-8 :549018 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/output.log
2021-07-14 18:19:57,473 INFO Thread-8 :549018 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/requirements.txt
2021-07-14 18:20:11,779 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:20:11,779 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:20:15,481 INFO Thread-8 :549018 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/output.log
2021-07-14 18:20:24,687 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:20:26,485 INFO Thread-8 :549018 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/config.yaml
2021-07-14 18:20:26,913 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:20:26,913 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:20:42,047 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:20:42,048 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:20:54,758 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:20:57,181 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:20:57,182 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:21:12,314 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:21:12,315 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:21:24,825 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:21:27,445 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:21:27,446 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:21:42,580 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:21:42,580 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:21:54,891 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:21:57,716 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:21:57,716 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:22:12,849 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:22:12,849 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:22:24,963 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:22:27,983 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:22:27,984 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:22:43,121 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:22:43,121 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:22:55,039 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:22:58,254 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:22:58,254 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:23:13,385 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:23:13,385 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:23:25,114 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:23:28,517 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:23:28,517 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:23:43,648 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:23:43,649 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:23:55,185 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:23:58,785 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:23:58,785 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:24:13,918 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:24:13,918 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:24:25,261 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:24:29,053 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:24:29,053 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:24:44,185 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:24:44,185 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:24:55,335 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:24:59,318 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:24:59,318 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:25:14,450 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:25:14,450 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:25:25,409 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:25:29,584 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:25:29,584 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:25:44,713 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:25:44,713 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:25:55,479 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:25:59,845 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:25:59,845 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:26:14,976 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:26:14,977 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:26:25,544 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:26:30,114 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:26:30,114 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:26:45,247 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:26:45,247 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:26:55,606 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:27:00,379 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:27:00,380 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:27:15,542 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:27:15,542 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:27:25,670 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:27:30,683 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:27:30,684 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:27:45,816 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:27:45,817 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:27:55,730 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:28:00,949 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:28:00,949 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:28:05,678 INFO Thread-8 :549018 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/output.log
2021-07-14 18:28:16,097 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:28:16,097 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:28:25,799 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:28:31,288 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:28:31,288 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:28:46,425 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:28:46,425 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:28:55,882 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:29:01,559 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:29:01,559 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:29:16,690 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:29:16,690 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:29:25,961 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:29:31,823 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:29:31,823 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:29:43,722 INFO Thread-8 :549018 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/output.log
2021-07-14 18:29:44,729 DEBUG SenderThread:549018 [sender.py:send():179] send: telemetry
2021-07-14 18:29:44,730 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:44,730 DEBUG SenderThread:549018 [sender.py:send():179] send: exit
2021-07-14 18:29:44,730 INFO SenderThread:549018 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 18:29:44,731 INFO SenderThread:549018 [sender.py:send_exit():295] send defer
2021-07-14 18:29:44,731 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:44,731 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:44,731 INFO HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 18:29:44,732 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:44,732 INFO SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 18:29:44,732 INFO SenderThread:549018 [sender.py:transition_state():308] send defer: 1
2021-07-14 18:29:44,732 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:44,732 INFO HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 18:29:44,807 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:44,807 INFO SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 18:29:44,807 INFO SenderThread:549018 [sender.py:transition_state():308] send defer: 2
2021-07-14 18:29:44,807 DEBUG SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:29:44,807 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:44,807 INFO HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 18:29:44,808 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:44,808 INFO SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 18:29:44,808 INFO SenderThread:549018 [sender.py:transition_state():308] send defer: 3
2021-07-14 18:29:44,808 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:44,808 INFO HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 18:29:44,808 DEBUG SenderThread:549018 [sender.py:send():179] send: summary
2021-07-14 18:29:44,808 INFO SenderThread:549018 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 18:29:44,809 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:44,809 INFO SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 18:29:44,809 INFO SenderThread:549018 [sender.py:transition_state():308] send defer: 4
2021-07-14 18:29:44,809 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:44,809 INFO HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 18:29:44,809 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:44,809 INFO SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 18:29:44,833 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:44,983 INFO SenderThread:549018 [sender.py:transition_state():308] send defer: 5
2021-07-14 18:29:44,983 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:44,984 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:44,984 INFO HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 18:29:44,984 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:44,984 INFO SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 18:29:44,984 INFO SenderThread:549018 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 18:29:45,085 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:45,722 INFO Thread-8 :549018 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/output.log
2021-07-14 18:29:45,723 INFO SenderThread:549018 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/wandb-summary.json
2021-07-14 18:29:45,723 INFO SenderThread:549018 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/config.yaml
2021-07-14 18:29:45,724 INFO SenderThread:549018 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files
2021-07-14 18:29:45,724 INFO SenderThread:549018 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/requirements.txt requirements.txt
2021-07-14 18:29:45,724 INFO SenderThread:549018 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/output.log output.log
2021-07-14 18:29:45,725 INFO SenderThread:549018 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/wandb-metadata.json wandb-metadata.json
2021-07-14 18:29:45,725 INFO SenderThread:549018 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/config.yaml config.yaml
2021-07-14 18:29:45,725 INFO SenderThread:549018 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/wandb-summary.json wandb-summary.json
2021-07-14 18:29:45,731 INFO SenderThread:549018 [sender.py:transition_state():308] send defer: 6
2021-07-14 18:29:45,731 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:45,735 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:45,735 INFO HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 18:29:45,738 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:45,738 INFO SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 18:29:45,738 INFO SenderThread:549018 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 18:29:45,833 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:45,834 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:45,935 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:45,935 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:46,037 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:46,037 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:46,139 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:46,140 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:46,204 INFO Thread-15 :549018 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/wandb-summary.json
2021-07-14 18:29:46,205 INFO Thread-12 :549018 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/requirements.txt
2021-07-14 18:29:46,218 INFO Thread-14 :549018 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/config.yaml
2021-07-14 18:29:46,241 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:46,242 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:46,287 INFO Thread-13 :549018 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/output.log
2021-07-14 18:29:46,343 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:46,344 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:46,446 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:46,446 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:46,488 INFO Thread-7 :549018 [sender.py:transition_state():308] send defer: 7
2021-07-14 18:29:46,489 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:46,489 INFO HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 18:29:46,489 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:46,489 INFO SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 18:29:46,548 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:46,780 INFO SenderThread:549018 [sender.py:transition_state():308] send defer: 8
2021-07-14 18:29:46,780 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:46,780 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:46,780 INFO HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 18:29:46,780 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:46,781 INFO SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 18:29:46,781 INFO SenderThread:549018 [sender.py:transition_state():308] send defer: 9
2021-07-14 18:29:46,781 DEBUG SenderThread:549018 [sender.py:send():179] send: final
2021-07-14 18:29:46,781 DEBUG SenderThread:549018 [sender.py:send():179] send: footer
2021-07-14 18:29:46,781 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:46,782 INFO HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 18:29:46,782 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:46,782 INFO SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 18:29:46,882 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:46,882 DEBUG SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:46,882 INFO SenderThread:549018 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 18:29:46,883 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 18:29:46,884 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 18:29:46,884 DEBUG HandlerThread:549018 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 18:29:46,885 INFO HandlerThread:549018 [handler.py:finish():638] shutting down handler
2021-07-14 18:29:47,782 INFO WriterThread:549018 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/run-351hrfn2.wandb
2021-07-14 18:29:47,882 INFO SenderThread:549018 [sender.py:finish():945] shutting down sender
2021-07-14 18:29:47,882 INFO SenderThread:549018 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 18:29:47,882 INFO SenderThread:549018 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 18:29:47,885 INFO MainThread:549018 [internal.py:handle_exit():78] Internal process exited
|