File size: 19,609 Bytes
9bd1fec |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 |
2021-07-13 00:49:11,524 INFO MainThread:325318 [internal.py:wandb_internal():88] W&B internal server running at pid: 325318, started at: 2021-07-13 00:49:11.523864
2021-07-13 00:49:11,526 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: check_version
2021-07-13 00:49:11,526 INFO WriterThread:325318 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/run-3mu9pog5.wandb
2021-07-13 00:49:11,527 DEBUG SenderThread:325318 [sender.py:send():179] send: header
2021-07-13 00:49:11,527 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: check_version
2021-07-13 00:49:11,564 DEBUG SenderThread:325318 [sender.py:send():179] send: run
2021-07-13 00:49:11,738 INFO SenderThread:325318 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files
2021-07-13 00:49:11,739 INFO SenderThread:325318 [sender.py:_start_run_threads():716] run started: 3mu9pog5 with start time 1626137350
2021-07-13 00:49:11,739 DEBUG SenderThread:325318 [sender.py:send():179] send: summary
2021-07-13 00:49:11,739 INFO SenderThread:325318 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 00:49:11,739 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: run_start
2021-07-13 00:49:12,741 INFO Thread-8 :325318 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/wandb-summary.json
2021-07-13 00:49:12,868 DEBUG HandlerThread:325318 [meta.py:__init__():39] meta init
2021-07-13 00:49:12,868 DEBUG HandlerThread:325318 [meta.py:__init__():53] meta init done
2021-07-13 00:49:12,868 DEBUG HandlerThread:325318 [meta.py:probe():210] probe
2021-07-13 00:49:12,870 DEBUG HandlerThread:325318 [meta.py:_setup_git():200] setup git
2021-07-13 00:49:12,899 DEBUG HandlerThread:325318 [meta.py:_setup_git():207] setup git done
2021-07-13 00:49:12,899 DEBUG HandlerThread:325318 [meta.py:_save_pip():57] save pip
2021-07-13 00:49:12,899 DEBUG HandlerThread:325318 [meta.py:_save_pip():71] save pip done
2021-07-13 00:49:12,899 DEBUG HandlerThread:325318 [meta.py:probe():252] probe done
2021-07-13 00:49:12,903 DEBUG SenderThread:325318 [sender.py:send():179] send: files
2021-07-13 00:49:12,903 INFO SenderThread:325318 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-13 00:49:12,910 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:49:12,911 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:49:13,042 DEBUG SenderThread:325318 [sender.py:send():179] send: config
2021-07-13 00:49:13,043 DEBUG SenderThread:325318 [sender.py:send():179] send: config
2021-07-13 00:49:13,043 DEBUG SenderThread:325318 [sender.py:send():179] send: config
2021-07-13 00:49:13,348 INFO Thread-11 :325318 [upload_job.py:push():137] Uploaded file /tmp/tmpkvnk9e30wandb/65yetzns-wandb-metadata.json
2021-07-13 00:49:13,741 INFO Thread-8 :325318 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/output.log
2021-07-13 00:49:13,741 INFO Thread-8 :325318 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/requirements.txt
2021-07-13 00:49:13,741 INFO Thread-8 :325318 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/wandb-metadata.json
2021-07-13 00:49:28,044 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:49:28,044 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:49:29,748 INFO Thread-8 :325318 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/output.log
2021-07-13 00:49:31,749 INFO Thread-8 :325318 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/output.log
2021-07-13 00:49:40,952 DEBUG SenderThread:325318 [sender.py:send():179] send: stats
2021-07-13 00:49:42,754 INFO Thread-8 :325318 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/config.yaml
2021-07-13 00:49:43,176 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:49:43,177 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:49:58,307 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:49:58,307 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:50:11,029 DEBUG SenderThread:325318 [sender.py:send():179] send: stats
2021-07-13 00:50:13,441 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:50:13,442 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:50:21,769 INFO Thread-8 :325318 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/output.log
2021-07-13 00:50:28,590 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:50:28,590 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:50:41,106 DEBUG SenderThread:325318 [sender.py:send():179] send: stats
2021-07-13 00:50:43,758 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:50:43,759 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:50:58,908 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:50:58,909 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:51:11,187 DEBUG SenderThread:325318 [sender.py:send():179] send: stats
2021-07-13 00:51:14,040 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:51:14,041 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:51:29,172 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:51:29,173 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:51:41,267 DEBUG SenderThread:325318 [sender.py:send():179] send: stats
2021-07-13 00:51:44,303 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:51:44,304 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:51:53,809 INFO Thread-8 :325318 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/output.log
2021-07-13 00:51:54,323 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:54,323 DEBUG SenderThread:325318 [sender.py:send():179] send: telemetry
2021-07-13 00:51:54,323 DEBUG SenderThread:325318 [sender.py:send():179] send: exit
2021-07-13 00:51:54,323 INFO SenderThread:325318 [sender.py:send_exit():287] handling exit code: 1
2021-07-13 00:51:54,323 INFO SenderThread:325318 [sender.py:send_exit():295] send defer
2021-07-13 00:51:54,323 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:54,324 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:54,324 INFO HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-13 00:51:54,324 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:54,324 INFO SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-13 00:51:54,324 INFO SenderThread:325318 [sender.py:transition_state():308] send defer: 1
2021-07-13 00:51:54,325 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:54,325 INFO HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-13 00:51:54,400 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:54,400 INFO SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-13 00:51:54,400 INFO SenderThread:325318 [sender.py:transition_state():308] send defer: 2
2021-07-13 00:51:54,401 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:54,401 INFO HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-13 00:51:54,401 DEBUG SenderThread:325318 [sender.py:send():179] send: stats
2021-07-13 00:51:54,401 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:54,401 INFO SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-13 00:51:54,401 INFO SenderThread:325318 [sender.py:transition_state():308] send defer: 3
2021-07-13 00:51:54,402 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:54,402 INFO HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-13 00:51:54,402 DEBUG SenderThread:325318 [sender.py:send():179] send: summary
2021-07-13 00:51:54,402 INFO SenderThread:325318 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 00:51:54,403 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:54,403 INFO SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-13 00:51:54,403 INFO SenderThread:325318 [sender.py:transition_state():308] send defer: 4
2021-07-13 00:51:54,403 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:54,403 INFO HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-13 00:51:54,403 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:54,403 INFO SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-13 00:51:54,426 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:54,590 INFO SenderThread:325318 [sender.py:transition_state():308] send defer: 5
2021-07-13 00:51:54,590 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:54,591 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:54,591 INFO HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-13 00:51:54,591 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:54,591 INFO SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-13 00:51:54,591 INFO SenderThread:325318 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-13 00:51:54,693 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:54,809 INFO Thread-8 :325318 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/output.log
2021-07-13 00:51:54,810 INFO SenderThread:325318 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/config.yaml
2021-07-13 00:51:54,810 INFO SenderThread:325318 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/wandb-summary.json
2021-07-13 00:51:54,810 INFO SenderThread:325318 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files
2021-07-13 00:51:54,810 INFO SenderThread:325318 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/requirements.txt requirements.txt
2021-07-13 00:51:54,810 INFO SenderThread:325318 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/output.log output.log
2021-07-13 00:51:54,811 INFO SenderThread:325318 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/wandb-metadata.json wandb-metadata.json
2021-07-13 00:51:54,811 INFO SenderThread:325318 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/config.yaml config.yaml
2021-07-13 00:51:54,811 INFO SenderThread:325318 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/wandb-summary.json wandb-summary.json
2021-07-13 00:51:54,811 INFO SenderThread:325318 [sender.py:transition_state():308] send defer: 6
2021-07-13 00:51:54,811 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:54,812 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:54,812 INFO HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-13 00:51:54,812 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:54,814 INFO SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-13 00:51:54,814 INFO SenderThread:325318 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 00:51:54,913 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:54,914 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:55,016 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:55,016 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:55,118 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:55,118 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:55,220 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:55,220 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:55,257 INFO Thread-14 :325318 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/config.yaml
2021-07-13 00:51:55,266 INFO Thread-12 :325318 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/requirements.txt
2021-07-13 00:51:55,277 INFO Thread-13 :325318 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/output.log
2021-07-13 00:51:55,288 INFO Thread-15 :325318 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/wandb-summary.json
2021-07-13 00:51:55,322 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:55,322 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:55,424 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:55,425 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:55,489 INFO Thread-7 :325318 [sender.py:transition_state():308] send defer: 7
2021-07-13 00:51:55,489 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:55,489 INFO HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-13 00:51:55,489 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:55,490 INFO SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-13 00:51:55,526 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:55,771 INFO SenderThread:325318 [sender.py:transition_state():308] send defer: 8
2021-07-13 00:51:55,772 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:55,772 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:55,772 INFO HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-13 00:51:55,772 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:55,772 INFO SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-13 00:51:55,773 INFO SenderThread:325318 [sender.py:transition_state():308] send defer: 9
2021-07-13 00:51:55,773 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:55,773 INFO HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-13 00:51:55,773 DEBUG SenderThread:325318 [sender.py:send():179] send: final
2021-07-13 00:51:55,773 DEBUG SenderThread:325318 [sender.py:send():179] send: footer
2021-07-13 00:51:55,773 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:55,773 INFO SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-13 00:51:55,874 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:55,874 DEBUG SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:55,874 INFO SenderThread:325318 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 00:51:55,876 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: get_summary
2021-07-13 00:51:55,877 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-13 00:51:55,877 DEBUG HandlerThread:325318 [handler.py:handle_request():124] handle_request: shutdown
2021-07-13 00:51:55,877 INFO HandlerThread:325318 [handler.py:finish():638] shutting down handler
2021-07-13 00:51:56,774 INFO WriterThread:325318 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/run-3mu9pog5.wandb
2021-07-13 00:51:56,875 INFO SenderThread:325318 [sender.py:finish():945] shutting down sender
2021-07-13 00:51:56,875 INFO SenderThread:325318 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 00:51:56,875 INFO SenderThread:325318 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 00:51:56,877 INFO MainThread:325318 [internal.py:handle_exit():78] Internal process exited
|