File size: 20,144 Bytes
f291f93 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 |
2021-07-13 11:20:39,477 INFO MainThread:350932 [internal.py:wandb_internal():88] W&B internal server running at pid: 350932, started at: 2021-07-13 11:20:39.476738
2021-07-13 11:20:39,479 INFO WriterThread:350932 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/run-qbdrt0c0.wandb
2021-07-13 11:20:39,479 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: check_version
2021-07-13 11:20:39,480 DEBUG SenderThread:350932 [sender.py:send():179] send: header
2021-07-13 11:20:39,480 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: check_version
2021-07-13 11:20:39,519 DEBUG SenderThread:350932 [sender.py:send():179] send: run
2021-07-13 11:20:39,698 INFO SenderThread:350932 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files
2021-07-13 11:20:39,698 INFO SenderThread:350932 [sender.py:_start_run_threads():716] run started: qbdrt0c0 with start time 1626175238
2021-07-13 11:20:39,698 DEBUG SenderThread:350932 [sender.py:send():179] send: summary
2021-07-13 11:20:39,698 INFO SenderThread:350932 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 11:20:39,699 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: run_start
2021-07-13 11:20:40,702 INFO Thread-8 :350932 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/wandb-summary.json
2021-07-13 11:20:40,844 DEBUG HandlerThread:350932 [meta.py:__init__():39] meta init
2021-07-13 11:20:40,845 DEBUG HandlerThread:350932 [meta.py:__init__():53] meta init done
2021-07-13 11:20:40,845 DEBUG HandlerThread:350932 [meta.py:probe():210] probe
2021-07-13 11:20:40,846 DEBUG HandlerThread:350932 [meta.py:_setup_git():200] setup git
2021-07-13 11:20:40,874 DEBUG HandlerThread:350932 [meta.py:_setup_git():207] setup git done
2021-07-13 11:20:40,874 DEBUG HandlerThread:350932 [meta.py:_save_pip():57] save pip
2021-07-13 11:20:40,875 DEBUG HandlerThread:350932 [meta.py:_save_pip():71] save pip done
2021-07-13 11:20:40,875 DEBUG HandlerThread:350932 [meta.py:probe():252] probe done
2021-07-13 11:20:40,878 DEBUG SenderThread:350932 [sender.py:send():179] send: files
2021-07-13 11:20:40,878 INFO SenderThread:350932 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-13 11:20:40,885 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:20:40,886 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:20:41,013 DEBUG SenderThread:350932 [sender.py:send():179] send: config
2021-07-13 11:20:41,014 DEBUG SenderThread:350932 [sender.py:send():179] send: config
2021-07-13 11:20:41,014 DEBUG SenderThread:350932 [sender.py:send():179] send: config
2021-07-13 11:20:41,368 INFO Thread-11 :350932 [upload_job.py:push():137] Uploaded file /tmp/tmp_o2vfehlwandb/wusrh9hx-wandb-metadata.json
2021-07-13 11:20:41,700 INFO Thread-8 :350932 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/output.log
2021-07-13 11:20:41,700 INFO Thread-8 :350932 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/wandb-metadata.json
2021-07-13 11:20:41,700 INFO Thread-8 :350932 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/requirements.txt
2021-07-13 11:20:55,705 INFO Thread-8 :350932 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/output.log
2021-07-13 11:20:56,015 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:20:56,016 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:20:57,706 INFO Thread-8 :350932 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/output.log
2021-07-13 11:21:08,929 DEBUG SenderThread:350932 [sender.py:send():179] send: stats
2021-07-13 11:21:10,712 INFO Thread-8 :350932 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/config.yaml
2021-07-13 11:21:11,150 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:21:11,150 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:21:26,282 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:21:26,282 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:21:39,007 DEBUG SenderThread:350932 [sender.py:send():179] send: stats
2021-07-13 11:21:41,413 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:21:41,414 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:21:56,566 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:21:56,566 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:22:09,085 DEBUG SenderThread:350932 [sender.py:send():179] send: stats
2021-07-13 11:22:11,700 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:22:11,700 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:22:26,842 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:22:26,842 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:22:27,737 INFO Thread-8 :350932 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/output.log
2021-07-13 11:22:39,163 DEBUG SenderThread:350932 [sender.py:send():179] send: stats
2021-07-13 11:22:41,997 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:22:41,997 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:22:57,140 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:22:57,140 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:23:09,248 DEBUG SenderThread:350932 [sender.py:send():179] send: stats
2021-07-13 11:23:12,271 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:23:12,272 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:23:27,405 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:23:27,405 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:23:39,327 DEBUG SenderThread:350932 [sender.py:send():179] send: stats
2021-07-13 11:23:42,540 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:23:42,540 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:23:47,766 INFO Thread-8 :350932 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/output.log
2021-07-13 11:23:49,172 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:49,172 DEBUG SenderThread:350932 [sender.py:send():179] send: telemetry
2021-07-13 11:23:49,172 DEBUG SenderThread:350932 [sender.py:send():179] send: exit
2021-07-13 11:23:49,172 INFO SenderThread:350932 [sender.py:send_exit():287] handling exit code: 1
2021-07-13 11:23:49,173 INFO SenderThread:350932 [sender.py:send_exit():295] send defer
2021-07-13 11:23:49,173 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:49,173 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:49,174 INFO HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-13 11:23:49,174 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:49,174 INFO SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-13 11:23:49,174 INFO SenderThread:350932 [sender.py:transition_state():308] send defer: 1
2021-07-13 11:23:49,175 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:49,175 INFO HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-13 11:23:49,253 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:49,254 INFO SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-13 11:23:49,254 INFO SenderThread:350932 [sender.py:transition_state():308] send defer: 2
2021-07-13 11:23:49,254 DEBUG SenderThread:350932 [sender.py:send():179] send: stats
2021-07-13 11:23:49,254 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:49,254 INFO HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-13 11:23:49,255 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:49,255 INFO SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-13 11:23:49,255 INFO SenderThread:350932 [sender.py:transition_state():308] send defer: 3
2021-07-13 11:23:49,255 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:49,255 INFO HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-13 11:23:49,255 DEBUG SenderThread:350932 [sender.py:send():179] send: summary
2021-07-13 11:23:49,256 INFO SenderThread:350932 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 11:23:49,256 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:49,256 INFO SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-13 11:23:49,256 INFO SenderThread:350932 [sender.py:transition_state():308] send defer: 4
2021-07-13 11:23:49,256 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:49,256 INFO HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-13 11:23:49,256 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:49,256 INFO SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-13 11:23:49,275 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:49,445 INFO SenderThread:350932 [sender.py:transition_state():308] send defer: 5
2021-07-13 11:23:49,445 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:49,445 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:49,445 INFO HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-13 11:23:49,446 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:49,446 INFO SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-13 11:23:49,446 INFO SenderThread:350932 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-13 11:23:49,547 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:49,766 INFO Thread-8 :350932 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/config.yaml
2021-07-13 11:23:49,767 INFO SenderThread:350932 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/output.log
2021-07-13 11:23:49,767 INFO SenderThread:350932 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/wandb-summary.json
2021-07-13 11:23:49,767 INFO SenderThread:350932 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files
2021-07-13 11:23:49,767 INFO SenderThread:350932 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/requirements.txt requirements.txt
2021-07-13 11:23:49,767 INFO SenderThread:350932 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/output.log output.log
2021-07-13 11:23:49,767 INFO SenderThread:350932 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/wandb-metadata.json wandb-metadata.json
2021-07-13 11:23:49,767 INFO SenderThread:350932 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/config.yaml config.yaml
2021-07-13 11:23:49,767 INFO SenderThread:350932 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/wandb-summary.json wandb-summary.json
2021-07-13 11:23:49,768 INFO SenderThread:350932 [sender.py:transition_state():308] send defer: 6
2021-07-13 11:23:49,768 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:49,771 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:49,771 INFO HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-13 11:23:49,771 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:49,772 INFO SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-13 11:23:49,772 INFO SenderThread:350932 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 11:23:49,873 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:49,873 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:49,975 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:49,976 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:50,078 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:50,078 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:50,180 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:50,180 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:50,215 INFO Thread-15 :350932 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/wandb-summary.json
2021-07-13 11:23:50,216 INFO Thread-14 :350932 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/config.yaml
2021-07-13 11:23:50,221 INFO Thread-12 :350932 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/requirements.txt
2021-07-13 11:23:50,225 INFO Thread-13 :350932 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/files/output.log
2021-07-13 11:23:50,282 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:50,282 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:50,384 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:50,384 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:50,425 INFO Thread-7 :350932 [sender.py:transition_state():308] send defer: 7
2021-07-13 11:23:50,426 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:50,426 INFO HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-13 11:23:50,426 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:50,426 INFO SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-13 11:23:50,486 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:50,554 INFO SenderThread:350932 [sender.py:transition_state():308] send defer: 8
2021-07-13 11:23:50,554 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:50,555 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:50,555 INFO HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-13 11:23:50,555 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:50,555 INFO SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-13 11:23:50,555 INFO SenderThread:350932 [sender.py:transition_state():308] send defer: 9
2021-07-13 11:23:50,556 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:23:50,556 INFO HandlerThread:350932 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-13 11:23:50,556 DEBUG SenderThread:350932 [sender.py:send():179] send: final
2021-07-13 11:23:50,556 DEBUG SenderThread:350932 [sender.py:send():179] send: footer
2021-07-13 11:23:50,556 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: defer
2021-07-13 11:23:50,556 INFO SenderThread:350932 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-13 11:23:50,656 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:23:50,656 DEBUG SenderThread:350932 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:23:50,657 INFO SenderThread:350932 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 11:23:50,658 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: get_summary
2021-07-13 11:23:50,659 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-13 11:23:50,659 DEBUG HandlerThread:350932 [handler.py:handle_request():124] handle_request: shutdown
2021-07-13 11:23:50,660 INFO HandlerThread:350932 [handler.py:finish():638] shutting down handler
2021-07-13 11:23:51,557 INFO WriterThread:350932 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_112038-qbdrt0c0/run-qbdrt0c0.wandb
2021-07-13 11:23:51,657 INFO SenderThread:350932 [sender.py:finish():945] shutting down sender
2021-07-13 11:23:51,657 INFO SenderThread:350932 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 11:23:51,657 INFO SenderThread:350932 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 11:23:51,659 INFO MainThread:350932 [internal.py:handle_exit():78] Internal process exited
|