File size: 19,385 Bytes
f291f93 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 |
2021-07-13 12:47:49,474 INFO MainThread:355550 [internal.py:wandb_internal():88] W&B internal server running at pid: 355550, started at: 2021-07-13 12:47:49.474151
2021-07-13 12:47:49,476 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: check_version
2021-07-13 12:47:49,477 INFO WriterThread:355550 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/run-vubzdnhu.wandb
2021-07-13 12:47:49,477 DEBUG SenderThread:355550 [sender.py:send():179] send: header
2021-07-13 12:47:49,477 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: check_version
2021-07-13 12:47:49,517 DEBUG SenderThread:355550 [sender.py:send():179] send: run
2021-07-13 12:47:49,718 INFO SenderThread:355550 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files
2021-07-13 12:47:49,718 INFO SenderThread:355550 [sender.py:_start_run_threads():716] run started: vubzdnhu with start time 1626180468
2021-07-13 12:47:49,718 DEBUG SenderThread:355550 [sender.py:send():179] send: summary
2021-07-13 12:47:49,718 INFO SenderThread:355550 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 12:47:49,719 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: run_start
2021-07-13 12:47:50,719 INFO Thread-8 :355550 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/wandb-summary.json
2021-07-13 12:47:50,855 DEBUG HandlerThread:355550 [meta.py:__init__():39] meta init
2021-07-13 12:47:50,855 DEBUG HandlerThread:355550 [meta.py:__init__():53] meta init done
2021-07-13 12:47:50,855 DEBUG HandlerThread:355550 [meta.py:probe():210] probe
2021-07-13 12:47:50,856 DEBUG HandlerThread:355550 [meta.py:_setup_git():200] setup git
2021-07-13 12:47:50,887 DEBUG HandlerThread:355550 [meta.py:_setup_git():207] setup git done
2021-07-13 12:47:50,888 DEBUG HandlerThread:355550 [meta.py:_save_pip():57] save pip
2021-07-13 12:47:50,888 DEBUG HandlerThread:355550 [meta.py:_save_pip():71] save pip done
2021-07-13 12:47:50,888 DEBUG HandlerThread:355550 [meta.py:probe():252] probe done
2021-07-13 12:47:50,892 DEBUG SenderThread:355550 [sender.py:send():179] send: files
2021-07-13 12:47:50,892 INFO SenderThread:355550 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-13 12:47:50,899 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:47:50,899 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:47:51,026 DEBUG SenderThread:355550 [sender.py:send():179] send: config
2021-07-13 12:47:51,027 DEBUG SenderThread:355550 [sender.py:send():179] send: config
2021-07-13 12:47:51,027 DEBUG SenderThread:355550 [sender.py:send():179] send: config
2021-07-13 12:47:51,378 INFO Thread-11 :355550 [upload_job.py:push():137] Uploaded file /tmp/tmpy_8wrz9rwandb/2yxmm3l6-wandb-metadata.json
2021-07-13 12:47:51,719 INFO Thread-8 :355550 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/requirements.txt
2021-07-13 12:47:51,719 INFO Thread-8 :355550 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/output.log
2021-07-13 12:47:51,719 INFO Thread-8 :355550 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/wandb-metadata.json
2021-07-13 12:48:05,724 INFO Thread-8 :355550 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/output.log
2021-07-13 12:48:06,028 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:48:06,029 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:48:07,724 INFO Thread-8 :355550 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/output.log
2021-07-13 12:48:18,943 DEBUG SenderThread:355550 [sender.py:send():179] send: stats
2021-07-13 12:48:20,728 INFO Thread-8 :355550 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/config.yaml
2021-07-13 12:48:21,161 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:48:21,161 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:48:35,733 INFO Thread-8 :355550 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/output.log
2021-07-13 12:48:36,292 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:48:36,292 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:48:49,027 DEBUG SenderThread:355550 [sender.py:send():179] send: stats
2021-07-13 12:48:51,451 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:48:51,451 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:49:06,603 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:49:06,604 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:49:19,111 DEBUG SenderThread:355550 [sender.py:send():179] send: stats
2021-07-13 12:49:21,736 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:49:21,737 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:49:36,867 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:49:36,868 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:49:49,193 DEBUG SenderThread:355550 [sender.py:send():179] send: stats
2021-07-13 12:49:51,999 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:49:52,000 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:50:07,132 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:50:07,132 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:50:19,272 DEBUG SenderThread:355550 [sender.py:send():179] send: stats
2021-07-13 12:50:19,770 INFO Thread-8 :355550 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/output.log
2021-07-13 12:50:20,563 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:20,564 DEBUG SenderThread:355550 [sender.py:send():179] send: telemetry
2021-07-13 12:50:20,564 DEBUG SenderThread:355550 [sender.py:send():179] send: exit
2021-07-13 12:50:20,564 INFO SenderThread:355550 [sender.py:send_exit():287] handling exit code: 1
2021-07-13 12:50:20,565 INFO SenderThread:355550 [sender.py:send_exit():295] send defer
2021-07-13 12:50:20,565 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:20,566 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:20,566 INFO HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-13 12:50:20,566 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:20,567 INFO SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-13 12:50:20,567 INFO SenderThread:355550 [sender.py:transition_state():308] send defer: 1
2021-07-13 12:50:20,567 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:20,567 INFO HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-13 12:50:20,576 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:20,576 INFO SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-13 12:50:20,576 INFO SenderThread:355550 [sender.py:transition_state():308] send defer: 2
2021-07-13 12:50:20,576 DEBUG SenderThread:355550 [sender.py:send():179] send: stats
2021-07-13 12:50:20,577 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:20,577 INFO HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-13 12:50:20,577 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:20,577 INFO SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-13 12:50:20,577 INFO SenderThread:355550 [sender.py:transition_state():308] send defer: 3
2021-07-13 12:50:20,577 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:20,577 INFO HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-13 12:50:20,578 DEBUG SenderThread:355550 [sender.py:send():179] send: summary
2021-07-13 12:50:20,578 INFO SenderThread:355550 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 12:50:20,578 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:20,578 INFO SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-13 12:50:20,578 INFO SenderThread:355550 [sender.py:transition_state():308] send defer: 4
2021-07-13 12:50:20,579 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:20,579 INFO HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-13 12:50:20,579 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:20,579 INFO SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-13 12:50:20,668 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:20,746 INFO SenderThread:355550 [sender.py:transition_state():308] send defer: 5
2021-07-13 12:50:20,746 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:20,746 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:20,746 INFO HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-13 12:50:20,747 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:20,747 INFO SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-13 12:50:20,747 INFO SenderThread:355550 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-13 12:50:20,771 INFO Thread-8 :355550 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/config.yaml
2021-07-13 12:50:20,771 INFO SenderThread:355550 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/output.log
2021-07-13 12:50:20,771 INFO SenderThread:355550 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/wandb-summary.json
2021-07-13 12:50:20,772 INFO SenderThread:355550 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files
2021-07-13 12:50:20,772 INFO SenderThread:355550 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/requirements.txt requirements.txt
2021-07-13 12:50:20,772 INFO SenderThread:355550 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/output.log output.log
2021-07-13 12:50:20,772 INFO SenderThread:355550 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/wandb-metadata.json wandb-metadata.json
2021-07-13 12:50:20,772 INFO SenderThread:355550 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/config.yaml config.yaml
2021-07-13 12:50:20,772 INFO SenderThread:355550 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/wandb-summary.json wandb-summary.json
2021-07-13 12:50:20,775 INFO SenderThread:355550 [sender.py:transition_state():308] send defer: 6
2021-07-13 12:50:20,777 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:20,778 INFO HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-13 12:50:20,784 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:20,784 INFO SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-13 12:50:20,784 INFO SenderThread:355550 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 12:50:20,848 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:20,848 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:20,950 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:20,950 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:21,052 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:21,053 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:21,154 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:21,155 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:21,218 INFO Thread-15 :355550 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/wandb-summary.json
2021-07-13 12:50:21,219 INFO Thread-13 :355550 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/output.log
2021-07-13 12:50:21,224 INFO Thread-14 :355550 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/config.yaml
2021-07-13 12:50:21,256 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:21,257 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:21,271 INFO Thread-12 :355550 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/requirements.txt
2021-07-13 12:50:21,359 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:21,359 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:21,461 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:21,461 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:21,471 INFO Thread-7 :355550 [sender.py:transition_state():308] send defer: 7
2021-07-13 12:50:21,471 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:21,471 INFO HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-13 12:50:21,472 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:21,472 INFO SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-13 12:50:21,562 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:21,867 INFO SenderThread:355550 [sender.py:transition_state():308] send defer: 8
2021-07-13 12:50:21,867 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:21,868 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:21,868 INFO HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-13 12:50:21,868 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:21,868 INFO SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-13 12:50:21,868 INFO SenderThread:355550 [sender.py:transition_state():308] send defer: 9
2021-07-13 12:50:21,869 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:21,869 INFO HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-13 12:50:21,869 DEBUG SenderThread:355550 [sender.py:send():179] send: final
2021-07-13 12:50:21,869 DEBUG SenderThread:355550 [sender.py:send():179] send: footer
2021-07-13 12:50:21,869 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:21,869 INFO SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-13 12:50:21,969 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:21,970 DEBUG SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:21,970 INFO SenderThread:355550 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 12:50:21,971 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: get_summary
2021-07-13 12:50:21,972 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-13 12:50:21,973 DEBUG HandlerThread:355550 [handler.py:handle_request():124] handle_request: shutdown
2021-07-13 12:50:21,973 INFO HandlerThread:355550 [handler.py:finish():638] shutting down handler
2021-07-13 12:50:22,870 INFO WriterThread:355550 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/run-vubzdnhu.wandb
2021-07-13 12:50:22,970 INFO SenderThread:355550 [sender.py:finish():945] shutting down sender
2021-07-13 12:50:22,970 INFO SenderThread:355550 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 12:50:22,970 INFO SenderThread:355550 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 12:50:22,973 INFO MainThread:355550 [internal.py:handle_exit():78] Internal process exited
|