File size: 26,719 Bytes
f291f93 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 |
2021-07-14 18:33:05,441 INFO MainThread:551470 [internal.py:wandb_internal():88] W&B internal server running at pid: 551470, started at: 2021-07-14 18:33:05.441281
2021-07-14 18:33:05,443 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 18:33:05,443 INFO WriterThread:551470 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/run-3d4rorca.wandb
2021-07-14 18:33:05,444 DEBUG SenderThread:551470 [sender.py:send():179] send: header
2021-07-14 18:33:05,444 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: check_version
2021-07-14 18:33:05,481 DEBUG SenderThread:551470 [sender.py:send():179] send: run
2021-07-14 18:33:05,658 INFO SenderThread:551470 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files
2021-07-14 18:33:05,658 INFO SenderThread:551470 [sender.py:_start_run_threads():716] run started: 3d4rorca with start time 1626287584
2021-07-14 18:33:05,658 DEBUG SenderThread:551470 [sender.py:send():179] send: summary
2021-07-14 18:33:05,659 INFO SenderThread:551470 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 18:33:05,659 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 18:33:06,662 INFO Thread-8 :551470 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/wandb-summary.json
2021-07-14 18:33:06,772 DEBUG HandlerThread:551470 [meta.py:__init__():39] meta init
2021-07-14 18:33:06,772 DEBUG HandlerThread:551470 [meta.py:__init__():53] meta init done
2021-07-14 18:33:06,772 DEBUG HandlerThread:551470 [meta.py:probe():210] probe
2021-07-14 18:33:06,773 DEBUG HandlerThread:551470 [meta.py:_setup_git():200] setup git
2021-07-14 18:33:06,799 DEBUG HandlerThread:551470 [meta.py:_setup_git():207] setup git done
2021-07-14 18:33:06,799 DEBUG HandlerThread:551470 [meta.py:_save_pip():57] save pip
2021-07-14 18:33:06,799 DEBUG HandlerThread:551470 [meta.py:_save_pip():71] save pip done
2021-07-14 18:33:06,799 DEBUG HandlerThread:551470 [meta.py:probe():252] probe done
2021-07-14 18:33:06,802 DEBUG SenderThread:551470 [sender.py:send():179] send: files
2021-07-14 18:33:06,802 INFO SenderThread:551470 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 18:33:06,808 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:33:06,808 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:33:06,939 DEBUG SenderThread:551470 [sender.py:send():179] send: config
2021-07-14 18:33:06,940 DEBUG SenderThread:551470 [sender.py:send():179] send: config
2021-07-14 18:33:06,940 DEBUG SenderThread:551470 [sender.py:send():179] send: config
2021-07-14 18:33:07,267 INFO Thread-11 :551470 [upload_job.py:push():137] Uploaded file /tmp/tmpwkna8gakwandb/1910yhst-wandb-metadata.json
2021-07-14 18:33:07,661 INFO Thread-8 :551470 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/output.log
2021-07-14 18:33:07,661 INFO Thread-8 :551470 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/requirements.txt
2021-07-14 18:33:07,661 INFO Thread-8 :551470 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/wandb-metadata.json
2021-07-14 18:33:21,941 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:33:21,941 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:33:25,667 INFO Thread-8 :551470 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/output.log
2021-07-14 18:33:34,849 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:33:36,671 INFO Thread-8 :551470 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/config.yaml
2021-07-14 18:33:37,073 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:33:37,073 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:33:52,207 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:33:52,207 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:34:04,916 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:34:07,343 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:34:07,343 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:34:22,477 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:34:22,477 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:34:34,976 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:34:37,613 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:34:37,613 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:34:52,747 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:34:52,747 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:35:05,044 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:35:07,878 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:35:07,879 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:35:23,013 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:35:23,013 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:35:35,110 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:35:38,147 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:35:38,148 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:35:53,282 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:35:53,282 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:36:05,181 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:36:08,416 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:36:08,417 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:36:23,549 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:36:23,549 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:36:35,251 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:36:38,681 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:36:38,681 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:36:53,817 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:36:53,817 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:37:05,320 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:37:08,952 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:37:08,953 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:37:24,087 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:37:24,087 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:37:35,392 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:37:39,221 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:37:39,222 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:37:54,352 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:37:54,353 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:38:05,464 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:38:09,485 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:38:09,485 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:38:24,626 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:38:24,626 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:38:35,535 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:38:39,761 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:38:39,761 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:38:54,893 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:38:54,893 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:39:05,597 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:39:10,025 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:39:10,026 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:39:25,164 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:39:25,164 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:39:35,672 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:39:40,296 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:39:40,297 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:39:55,430 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:39:55,430 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:40:05,746 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:40:10,563 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:40:10,564 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:40:25,697 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:40:25,698 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:40:35,821 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:40:40,834 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:40:40,834 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:40:55,966 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:40:55,967 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:41:05,888 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:41:11,114 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:41:11,114 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:41:11,814 INFO Thread-8 :551470 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/output.log
2021-07-14 18:41:26,276 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:41:26,277 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:41:35,966 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:41:41,417 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:41:41,418 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:41:56,548 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:41:56,548 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:42:06,047 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:42:11,683 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:42:11,684 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:42:26,818 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:42:26,818 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:42:36,121 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:42:41,950 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:42:41,951 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:42:51,640 DEBUG SenderThread:551470 [sender.py:send():179] send: telemetry
2021-07-14 18:42:51,641 DEBUG SenderThread:551470 [sender.py:send():179] send: exit
2021-07-14 18:42:51,641 INFO SenderThread:551470 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 18:42:51,641 INFO SenderThread:551470 [sender.py:send_exit():295] send defer
2021-07-14 18:42:51,641 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:51,642 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:51,642 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 18:42:51,642 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:51,642 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:51,642 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 18:42:51,642 INFO SenderThread:551470 [sender.py:transition_state():308] send defer: 1
2021-07-14 18:42:51,643 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:51,643 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 18:42:51,659 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:51,660 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 18:42:51,660 INFO SenderThread:551470 [sender.py:transition_state():308] send defer: 2
2021-07-14 18:42:51,660 DEBUG SenderThread:551470 [sender.py:send():179] send: stats
2021-07-14 18:42:51,660 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:51,660 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 18:42:51,660 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:51,660 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 18:42:51,660 INFO SenderThread:551470 [sender.py:transition_state():308] send defer: 3
2021-07-14 18:42:51,661 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:51,661 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 18:42:51,661 DEBUG SenderThread:551470 [sender.py:send():179] send: summary
2021-07-14 18:42:51,661 INFO SenderThread:551470 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 18:42:51,661 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:51,661 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 18:42:51,661 INFO SenderThread:551470 [sender.py:transition_state():308] send defer: 4
2021-07-14 18:42:51,662 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:51,662 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 18:42:51,662 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:51,662 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 18:42:51,744 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:51,838 INFO SenderThread:551470 [sender.py:transition_state():308] send defer: 5
2021-07-14 18:42:51,838 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:51,838 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:51,838 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 18:42:51,839 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:51,839 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 18:42:51,839 INFO SenderThread:551470 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 18:42:51,854 INFO Thread-8 :551470 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/config.yaml
2021-07-14 18:42:51,855 INFO SenderThread:551470 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/output.log
2021-07-14 18:42:51,855 INFO SenderThread:551470 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/wandb-summary.json
2021-07-14 18:42:51,855 INFO SenderThread:551470 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files
2021-07-14 18:42:51,855 INFO SenderThread:551470 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/requirements.txt requirements.txt
2021-07-14 18:42:51,855 INFO SenderThread:551470 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/output.log output.log
2021-07-14 18:42:51,856 INFO SenderThread:551470 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/wandb-metadata.json wandb-metadata.json
2021-07-14 18:42:51,856 INFO SenderThread:551470 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/config.yaml config.yaml
2021-07-14 18:42:51,856 INFO SenderThread:551470 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/wandb-summary.json wandb-summary.json
2021-07-14 18:42:51,856 INFO SenderThread:551470 [sender.py:transition_state():308] send defer: 6
2021-07-14 18:42:51,862 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:51,862 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 18:42:51,865 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:51,865 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 18:42:51,865 INFO SenderThread:551470 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 18:42:51,939 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:51,940 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:52,041 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:52,041 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:52,143 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:52,143 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:52,245 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:52,245 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:52,308 INFO Thread-12 :551470 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/requirements.txt
2021-07-14 18:42:52,325 INFO Thread-13 :551470 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/output.log
2021-07-14 18:42:52,341 INFO Thread-15 :551470 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/wandb-summary.json
2021-07-14 18:42:52,346 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:52,346 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:52,361 INFO Thread-14 :551470 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/config.yaml
2021-07-14 18:42:52,448 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:52,448 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:52,549 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:52,550 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:52,562 INFO Thread-7 :551470 [sender.py:transition_state():308] send defer: 7
2021-07-14 18:42:52,562 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:52,562 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 18:42:52,562 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:52,562 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 18:42:52,651 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:52,837 INFO SenderThread:551470 [sender.py:transition_state():308] send defer: 8
2021-07-14 18:42:52,837 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:52,837 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:52,837 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 18:42:52,838 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:52,838 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 18:42:52,838 INFO SenderThread:551470 [sender.py:transition_state():308] send defer: 9
2021-07-14 18:42:52,838 DEBUG SenderThread:551470 [sender.py:send():179] send: final
2021-07-14 18:42:52,838 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:42:52,838 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 18:42:52,839 DEBUG SenderThread:551470 [sender.py:send():179] send: footer
2021-07-14 18:42:52,839 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer
2021-07-14 18:42:52,839 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 18:42:52,938 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:42:52,939 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:42:52,939 INFO SenderThread:551470 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 18:42:52,940 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 18:42:52,941 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 18:42:52,941 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 18:42:52,941 INFO HandlerThread:551470 [handler.py:finish():638] shutting down handler
2021-07-14 18:42:53,839 INFO WriterThread:551470 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/run-3d4rorca.wandb
2021-07-14 18:42:53,939 INFO SenderThread:551470 [sender.py:finish():945] shutting down sender
2021-07-14 18:42:53,939 INFO SenderThread:551470 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 18:42:53,939 INFO SenderThread:551470 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 18:42:53,941 INFO MainThread:551470 [internal.py:handle_exit():78] Internal process exited
|