dat
Saving weights and logs at step 1252
f291f93
raw
history blame
20.6 kB
2021-07-13 11:11:12,514 INFO MainThread:348072 [internal.py:wandb_internal():88] W&B internal server running at pid: 348072, started at: 2021-07-13 11:11:12.514575
2021-07-13 11:11:12,516 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: check_version
2021-07-13 11:11:12,517 INFO WriterThread:348072 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/run-3cwslju0.wandb
2021-07-13 11:11:12,518 DEBUG SenderThread:348072 [sender.py:send():179] send: header
2021-07-13 11:11:12,518 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: check_version
2021-07-13 11:11:12,553 DEBUG SenderThread:348072 [sender.py:send():179] send: run
2021-07-13 11:11:12,741 INFO SenderThread:348072 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files
2021-07-13 11:11:12,741 INFO SenderThread:348072 [sender.py:_start_run_threads():716] run started: 3cwslju0 with start time 1626174671
2021-07-13 11:11:12,741 DEBUG SenderThread:348072 [sender.py:send():179] send: summary
2021-07-13 11:11:12,742 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: run_start
2021-07-13 11:11:12,743 INFO SenderThread:348072 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 11:11:13,745 INFO Thread-8 :348072 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/wandb-summary.json
2021-07-13 11:11:14,000 DEBUG HandlerThread:348072 [meta.py:__init__():39] meta init
2021-07-13 11:11:14,000 DEBUG HandlerThread:348072 [meta.py:__init__():53] meta init done
2021-07-13 11:11:14,000 DEBUG HandlerThread:348072 [meta.py:probe():210] probe
2021-07-13 11:11:14,001 DEBUG HandlerThread:348072 [meta.py:_setup_git():200] setup git
2021-07-13 11:11:14,031 DEBUG HandlerThread:348072 [meta.py:_setup_git():207] setup git done
2021-07-13 11:11:14,031 DEBUG HandlerThread:348072 [meta.py:_save_pip():57] save pip
2021-07-13 11:11:14,031 DEBUG HandlerThread:348072 [meta.py:_save_pip():71] save pip done
2021-07-13 11:11:14,031 DEBUG HandlerThread:348072 [meta.py:probe():252] probe done
2021-07-13 11:11:14,035 DEBUG SenderThread:348072 [sender.py:send():179] send: files
2021-07-13 11:11:14,035 INFO SenderThread:348072 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-13 11:11:14,043 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:11:14,043 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:11:14,170 DEBUG SenderThread:348072 [sender.py:send():179] send: config
2021-07-13 11:11:14,170 DEBUG SenderThread:348072 [sender.py:send():179] send: config
2021-07-13 11:11:14,170 DEBUG SenderThread:348072 [sender.py:send():179] send: config
2021-07-13 11:11:14,515 INFO Thread-11 :348072 [upload_job.py:push():137] Uploaded file /tmp/tmp64u985tqwandb/1u3tchat-wandb-metadata.json
2021-07-13 11:11:14,745 INFO Thread-8 :348072 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/wandb-metadata.json
2021-07-13 11:11:14,745 INFO Thread-8 :348072 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:11:14,745 INFO Thread-8 :348072 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/requirements.txt
2021-07-13 11:11:28,753 INFO Thread-8 :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:11:29,172 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:11:29,172 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:11:30,754 INFO Thread-8 :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:11:42,078 DEBUG SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:11:43,760 INFO Thread-8 :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/config.yaml
2021-07-13 11:11:44,303 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:11:44,304 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:11:59,435 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:11:59,436 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:12:12,149 DEBUG SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:12:14,568 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:12:14,569 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:12:29,703 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:12:29,704 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:12:42,213 DEBUG SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:12:44,838 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:12:44,839 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:12:59,973 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:12:59,974 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:13:06,802 INFO Thread-8 :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:13:12,281 DEBUG SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:13:15,129 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:13:15,129 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:13:30,277 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:13:30,277 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:13:42,360 DEBUG SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:13:45,408 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:13:45,409 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:14:00,541 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:14:00,555 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:14:12,440 DEBUG SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:14:15,687 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:14:15,687 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:14:26,833 INFO Thread-8 :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:14:27,502 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:27,503 DEBUG SenderThread:348072 [sender.py:send():179] send: telemetry
2021-07-13 11:14:27,503 DEBUG SenderThread:348072 [sender.py:send():179] send: exit
2021-07-13 11:14:27,503 INFO SenderThread:348072 [sender.py:send_exit():287] handling exit code: 1
2021-07-13 11:14:27,503 INFO SenderThread:348072 [sender.py:send_exit():295] send defer
2021-07-13 11:14:27,504 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:27,504 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:27,504 INFO HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-13 11:14:27,504 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:27,505 INFO SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-13 11:14:27,505 INFO SenderThread:348072 [sender.py:transition_state():308] send defer: 1
2021-07-13 11:14:27,505 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:27,505 INFO HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-13 11:14:27,574 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:27,574 INFO SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-13 11:14:27,574 INFO SenderThread:348072 [sender.py:transition_state():308] send defer: 2
2021-07-13 11:14:27,575 DEBUG SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:14:27,575 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:27,575 INFO HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-13 11:14:27,576 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:27,576 INFO SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-13 11:14:27,576 INFO SenderThread:348072 [sender.py:transition_state():308] send defer: 3
2021-07-13 11:14:27,576 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:27,576 INFO HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-13 11:14:27,576 DEBUG SenderThread:348072 [sender.py:send():179] send: summary
2021-07-13 11:14:27,606 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:27,704 INFO SenderThread:348072 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 11:14:27,705 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:27,705 INFO SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-13 11:14:27,705 INFO SenderThread:348072 [sender.py:transition_state():308] send defer: 4
2021-07-13 11:14:27,705 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:27,706 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:27,706 INFO HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-13 11:14:27,706 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:27,706 INFO SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-13 11:14:27,807 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:27,834 INFO Thread-8 :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:14:27,834 INFO Thread-8 :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/wandb-summary.json
2021-07-13 11:14:27,901 INFO SenderThread:348072 [sender.py:transition_state():308] send defer: 5
2021-07-13 11:14:27,901 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:27,902 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:27,902 INFO HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-13 11:14:27,902 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:27,902 INFO SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-13 11:14:27,902 INFO SenderThread:348072 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-13 11:14:28,004 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:28,834 INFO Thread-8 :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/config.yaml
2021-07-13 11:14:28,836 INFO SenderThread:348072 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files
2021-07-13 11:14:28,836 INFO SenderThread:348072 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/requirements.txt requirements.txt
2021-07-13 11:14:28,836 INFO SenderThread:348072 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log output.log
2021-07-13 11:14:28,836 INFO SenderThread:348072 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/wandb-metadata.json wandb-metadata.json
2021-07-13 11:14:28,836 INFO SenderThread:348072 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/config.yaml config.yaml
2021-07-13 11:14:28,836 INFO SenderThread:348072 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/wandb-summary.json wandb-summary.json
2021-07-13 11:14:28,837 INFO SenderThread:348072 [sender.py:transition_state():308] send defer: 6
2021-07-13 11:14:28,837 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:28,840 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:28,840 INFO HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-13 11:14:28,843 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:28,844 INFO SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-13 11:14:28,844 INFO SenderThread:348072 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 11:14:28,943 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:28,943 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,046 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,046 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,148 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,149 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,251 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,251 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,300 INFO Thread-14 :348072 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/config.yaml
2021-07-13 11:14:29,309 INFO Thread-12 :348072 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/requirements.txt
2021-07-13 11:14:29,332 INFO Thread-15 :348072 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/wandb-summary.json
2021-07-13 11:14:29,353 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,354 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,401 INFO Thread-13 :348072 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:14:29,456 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,479 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,581 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,582 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,679 INFO Thread-7 :348072 [sender.py:transition_state():308] send defer: 7
2021-07-13 11:14:29,680 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:29,680 INFO HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-13 11:14:29,680 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:29,680 INFO SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-13 11:14:29,684 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,949 INFO SenderThread:348072 [sender.py:transition_state():308] send defer: 8
2021-07-13 11:14:29,949 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,950 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:29,950 INFO HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-13 11:14:29,950 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:29,950 INFO SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-13 11:14:29,950 INFO SenderThread:348072 [sender.py:transition_state():308] send defer: 9
2021-07-13 11:14:29,951 DEBUG SenderThread:348072 [sender.py:send():179] send: final
2021-07-13 11:14:29,951 DEBUG SenderThread:348072 [sender.py:send():179] send: footer
2021-07-13 11:14:29,951 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:29,951 INFO HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-13 11:14:29,951 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:29,951 INFO SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-13 11:14:30,051 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:30,052 DEBUG SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:30,052 INFO SenderThread:348072 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 11:14:30,054 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: get_summary
2021-07-13 11:14:30,055 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-13 11:14:30,055 DEBUG HandlerThread:348072 [handler.py:handle_request():124] handle_request: shutdown
2021-07-13 11:14:30,056 INFO HandlerThread:348072 [handler.py:finish():638] shutting down handler
2021-07-13 11:14:30,951 INFO WriterThread:348072 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/run-3cwslju0.wandb
2021-07-13 11:14:31,052 INFO SenderThread:348072 [sender.py:finish():945] shutting down sender
2021-07-13 11:14:31,052 INFO SenderThread:348072 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 11:14:31,052 INFO SenderThread:348072 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 11:14:31,054 INFO MainThread:348072 [internal.py:handle_exit():78] Internal process exited