File size: 20,079 Bytes
f291f93 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 |
2021-07-13 13:16:16,698 INFO MainThread:363863 [internal.py:wandb_internal():88] W&B internal server running at pid: 363863, started at: 2021-07-13 13:16:16.698045
2021-07-13 13:16:16,700 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: check_version
2021-07-13 13:16:16,700 INFO WriterThread:363863 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/run-w761mg4c.wandb
2021-07-13 13:16:16,701 DEBUG SenderThread:363863 [sender.py:send():179] send: header
2021-07-13 13:16:16,701 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: check_version
2021-07-13 13:16:16,739 DEBUG SenderThread:363863 [sender.py:send():179] send: run
2021-07-13 13:16:16,916 INFO SenderThread:363863 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files
2021-07-13 13:16:16,916 INFO SenderThread:363863 [sender.py:_start_run_threads():716] run started: w761mg4c with start time 1626182176
2021-07-13 13:16:16,916 DEBUG SenderThread:363863 [sender.py:send():179] send: summary
2021-07-13 13:16:16,917 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: run_start
2021-07-13 13:16:16,917 INFO SenderThread:363863 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 13:16:17,918 INFO Thread-8 :363863 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/wandb-summary.json
2021-07-13 13:16:18,041 DEBUG HandlerThread:363863 [meta.py:__init__():39] meta init
2021-07-13 13:16:18,041 DEBUG HandlerThread:363863 [meta.py:__init__():53] meta init done
2021-07-13 13:16:18,041 DEBUG HandlerThread:363863 [meta.py:probe():210] probe
2021-07-13 13:16:18,042 DEBUG HandlerThread:363863 [meta.py:_setup_git():200] setup git
2021-07-13 13:16:18,071 DEBUG HandlerThread:363863 [meta.py:_setup_git():207] setup git done
2021-07-13 13:16:18,071 DEBUG HandlerThread:363863 [meta.py:_save_pip():57] save pip
2021-07-13 13:16:18,072 DEBUG HandlerThread:363863 [meta.py:_save_pip():71] save pip done
2021-07-13 13:16:18,072 DEBUG HandlerThread:363863 [meta.py:probe():252] probe done
2021-07-13 13:16:18,075 DEBUG SenderThread:363863 [sender.py:send():179] send: files
2021-07-13 13:16:18,075 INFO SenderThread:363863 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-13 13:16:18,083 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:16:18,083 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:16:18,213 DEBUG SenderThread:363863 [sender.py:send():179] send: config
2021-07-13 13:16:18,214 DEBUG SenderThread:363863 [sender.py:send():179] send: config
2021-07-13 13:16:18,214 DEBUG SenderThread:363863 [sender.py:send():179] send: config
2021-07-13 13:16:18,530 INFO Thread-11 :363863 [upload_job.py:push():137] Uploaded file /tmp/tmpe18zfifmwandb/okixm8uo-wandb-metadata.json
2021-07-13 13:16:18,918 INFO Thread-8 :363863 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/output.log
2021-07-13 13:16:18,918 INFO Thread-8 :363863 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/wandb-metadata.json
2021-07-13 13:16:18,918 INFO Thread-8 :363863 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/requirements.txt
2021-07-13 13:16:33,215 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:16:33,215 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:16:34,924 INFO Thread-8 :363863 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/output.log
2021-07-13 13:16:46,124 DEBUG SenderThread:363863 [sender.py:send():179] send: stats
2021-07-13 13:16:47,928 INFO Thread-8 :363863 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/config.yaml
2021-07-13 13:16:48,349 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:16:48,350 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:17:03,483 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:17:03,484 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:17:16,202 DEBUG SenderThread:363863 [sender.py:send():179] send: stats
2021-07-13 13:17:18,616 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:17:18,617 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:17:33,748 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:17:33,749 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:17:46,277 DEBUG SenderThread:363863 [sender.py:send():179] send: stats
2021-07-13 13:17:48,883 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:17:48,883 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:18:04,023 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:18:04,023 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:18:04,956 INFO Thread-8 :363863 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/output.log
2021-07-13 13:18:16,347 DEBUG SenderThread:363863 [sender.py:send():179] send: stats
2021-07-13 13:18:19,179 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:18:19,179 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:18:34,324 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:18:34,325 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:18:46,418 DEBUG SenderThread:363863 [sender.py:send():179] send: stats
2021-07-13 13:18:49,456 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:18:49,456 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:19:04,586 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:19:04,586 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:19:16,489 DEBUG SenderThread:363863 [sender.py:send():179] send: stats
2021-07-13 13:19:19,718 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:19:19,718 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:19:34,851 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:19:34,852 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:19:46,568 DEBUG SenderThread:363863 [sender.py:send():179] send: stats
2021-07-13 13:19:49,985 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:19:49,985 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:20:00,715 DEBUG SenderThread:363863 [sender.py:send():179] send: telemetry
2021-07-13 13:20:00,715 DEBUG SenderThread:363863 [sender.py:send():179] send: exit
2021-07-13 13:20:00,715 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:00,715 INFO SenderThread:363863 [sender.py:send_exit():287] handling exit code: 1
2021-07-13 13:20:00,716 INFO SenderThread:363863 [sender.py:send_exit():295] send defer
2021-07-13 13:20:00,716 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:00,716 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:00,716 INFO HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-13 13:20:00,717 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:00,717 INFO SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-13 13:20:00,717 INFO SenderThread:363863 [sender.py:transition_state():308] send defer: 1
2021-07-13 13:20:00,717 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:00,717 INFO HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-13 13:20:00,805 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:00,805 INFO SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-13 13:20:00,805 INFO SenderThread:363863 [sender.py:transition_state():308] send defer: 2
2021-07-13 13:20:00,806 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:00,806 INFO HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-13 13:20:00,806 DEBUG SenderThread:363863 [sender.py:send():179] send: stats
2021-07-13 13:20:00,806 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:00,806 INFO SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-13 13:20:00,806 INFO SenderThread:363863 [sender.py:transition_state():308] send defer: 3
2021-07-13 13:20:00,806 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:00,807 INFO HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-13 13:20:00,807 DEBUG SenderThread:363863 [sender.py:send():179] send: summary
2021-07-13 13:20:00,807 INFO SenderThread:363863 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 13:20:00,807 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:00,807 INFO SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-13 13:20:00,807 INFO SenderThread:363863 [sender.py:transition_state():308] send defer: 4
2021-07-13 13:20:00,808 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:00,808 INFO HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-13 13:20:00,808 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:00,808 INFO SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-13 13:20:00,818 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:00,985 INFO SenderThread:363863 [sender.py:transition_state():308] send defer: 5
2021-07-13 13:20:00,986 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:00,986 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:00,986 INFO HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-13 13:20:00,987 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:00,987 INFO SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-13 13:20:00,987 INFO SenderThread:363863 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-13 13:20:00,998 INFO Thread-8 :363863 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/output.log
2021-07-13 13:20:00,999 INFO SenderThread:363863 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/config.yaml
2021-07-13 13:20:00,999 INFO SenderThread:363863 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/wandb-summary.json
2021-07-13 13:20:00,999 INFO SenderThread:363863 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files
2021-07-13 13:20:00,999 INFO SenderThread:363863 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/requirements.txt requirements.txt
2021-07-13 13:20:00,999 INFO SenderThread:363863 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/output.log output.log
2021-07-13 13:20:00,999 INFO SenderThread:363863 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/wandb-metadata.json wandb-metadata.json
2021-07-13 13:20:01,000 INFO SenderThread:363863 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/config.yaml config.yaml
2021-07-13 13:20:01,000 INFO SenderThread:363863 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/wandb-summary.json wandb-summary.json
2021-07-13 13:20:01,000 INFO SenderThread:363863 [sender.py:transition_state():308] send defer: 6
2021-07-13 13:20:01,012 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:01,013 INFO HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-13 13:20:01,013 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:01,013 INFO SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-13 13:20:01,013 INFO SenderThread:363863 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 13:20:01,088 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:01,088 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:01,190 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:01,191 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:01,292 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:01,293 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:01,395 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:01,395 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:01,443 INFO Thread-13 :363863 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/output.log
2021-07-13 13:20:01,467 INFO Thread-15 :363863 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/wandb-summary.json
2021-07-13 13:20:01,468 INFO Thread-14 :363863 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/config.yaml
2021-07-13 13:20:01,474 INFO Thread-12 :363863 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/requirements.txt
2021-07-13 13:20:01,497 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:01,497 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:01,599 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:01,599 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:01,674 INFO Thread-7 :363863 [sender.py:transition_state():308] send defer: 7
2021-07-13 13:20:01,675 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:01,675 INFO HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-13 13:20:01,675 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:01,675 INFO SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-13 13:20:01,701 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:01,941 INFO SenderThread:363863 [sender.py:transition_state():308] send defer: 8
2021-07-13 13:20:01,942 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:01,942 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:01,942 INFO HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-13 13:20:01,942 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:01,943 INFO SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-13 13:20:01,943 INFO SenderThread:363863 [sender.py:transition_state():308] send defer: 9
2021-07-13 13:20:01,943 DEBUG SenderThread:363863 [sender.py:send():179] send: final
2021-07-13 13:20:01,943 DEBUG SenderThread:363863 [sender.py:send():179] send: footer
2021-07-13 13:20:01,943 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:01,943 INFO HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-13 13:20:01,944 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:01,944 INFO SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-13 13:20:02,044 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:02,044 DEBUG SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:02,044 INFO SenderThread:363863 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 13:20:02,046 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: get_summary
2021-07-13 13:20:02,046 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-13 13:20:02,047 DEBUG HandlerThread:363863 [handler.py:handle_request():124] handle_request: shutdown
2021-07-13 13:20:02,047 INFO HandlerThread:363863 [handler.py:finish():638] shutting down handler
2021-07-13 13:20:02,944 INFO WriterThread:363863 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/run-w761mg4c.wandb
2021-07-13 13:20:03,044 INFO SenderThread:363863 [sender.py:finish():945] shutting down sender
2021-07-13 13:20:03,044 INFO SenderThread:363863 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 13:20:03,045 INFO SenderThread:363863 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 13:20:03,047 INFO MainThread:363863 [internal.py:handle_exit():78] Internal process exited
|