2021-07-15 17:41:48,594 INFO MainThread:723175 [internal.py:wandb_internal():88] W&B internal server running at pid: 723175, started at: 2021-07-15 17:41:48.594473 2021-07-15 17:41:48,596 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: check_version 2021-07-15 17:41:48,597 INFO WriterThread:723175 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/run-3nkn7hxg.wandb 2021-07-15 17:41:48,598 DEBUG SenderThread:723175 [sender.py:send():179] send: header 2021-07-15 17:41:48,598 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: check_version 2021-07-15 17:41:48,634 DEBUG SenderThread:723175 [sender.py:send():179] send: run 2021-07-15 17:41:48,804 INFO SenderThread:723175 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files 2021-07-15 17:41:48,805 INFO SenderThread:723175 [sender.py:_start_run_threads():716] run started: 3nkn7hxg with start time 1626370908 2021-07-15 17:41:48,805 DEBUG SenderThread:723175 [sender.py:send():179] send: summary 2021-07-15 17:41:48,805 INFO SenderThread:723175 [sender.py:_save_file():841] saving file wandb-summary.json with policy end 2021-07-15 17:41:48,805 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: run_start 2021-07-15 17:41:49,817 INFO Thread-8 :723175 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/wandb-summary.json 2021-07-15 17:41:49,929 DEBUG HandlerThread:723175 [meta.py:__init__():39] meta init 2021-07-15 17:41:49,930 DEBUG HandlerThread:723175 [meta.py:__init__():53] meta init done 2021-07-15 17:41:49,930 DEBUG HandlerThread:723175 [meta.py:probe():210] probe 2021-07-15 17:41:49,931 DEBUG HandlerThread:723175 [meta.py:_setup_git():200] setup git 2021-07-15 17:41:49,960 DEBUG HandlerThread:723175 [meta.py:_setup_git():207] setup git done 2021-07-15 17:41:49,961 DEBUG HandlerThread:723175 [meta.py:_save_pip():57] save pip 2021-07-15 17:41:49,961 DEBUG HandlerThread:723175 [meta.py:_save_pip():71] save pip done 2021-07-15 17:41:49,961 DEBUG HandlerThread:723175 [meta.py:probe():252] probe done 2021-07-15 17:41:49,964 DEBUG SenderThread:723175 [sender.py:send():179] send: files 2021-07-15 17:41:49,964 INFO SenderThread:723175 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now 2021-07-15 17:41:49,970 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:41:49,971 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:41:50,099 DEBUG SenderThread:723175 [sender.py:send():179] send: config 2021-07-15 17:41:50,100 DEBUG SenderThread:723175 [sender.py:send():179] send: config 2021-07-15 17:41:50,100 DEBUG SenderThread:723175 [sender.py:send():179] send: config 2021-07-15 17:41:50,404 INFO Thread-11 :723175 [upload_job.py:push():137] Uploaded file /tmp/tmplksa3t0ywandb/5lvlwhj5-wandb-metadata.json 2021-07-15 17:41:50,815 INFO Thread-8 :723175 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/requirements.txt 2021-07-15 17:41:50,816 INFO Thread-8 :723175 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/output.log 2021-07-15 17:41:50,816 INFO Thread-8 :723175 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/wandb-metadata.json 2021-07-15 17:42:04,821 INFO Thread-8 :723175 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/output.log 2021-07-15 17:42:05,202 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:42:05,203 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:42:06,821 INFO Thread-8 :723175 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/output.log 2021-07-15 17:42:08,822 INFO Thread-8 :723175 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/output.log 2021-07-15 17:42:18,016 DEBUG SenderThread:723175 [sender.py:send():179] send: stats 2021-07-15 17:42:19,827 INFO Thread-8 :723175 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/config.yaml 2021-07-15 17:42:20,347 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:42:20,347 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:42:35,479 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:42:35,480 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:42:48,096 DEBUG SenderThread:723175 [sender.py:send():179] send: stats 2021-07-15 17:42:50,612 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:42:50,612 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:43:05,746 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:43:05,747 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:43:18,173 DEBUG SenderThread:723175 [sender.py:send():179] send: stats 2021-07-15 17:43:20,877 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:43:20,878 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:43:36,007 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:43:36,008 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:43:48,239 DEBUG SenderThread:723175 [sender.py:send():179] send: stats 2021-07-15 17:43:51,139 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:43:51,139 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:44:06,269 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:44:06,269 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:44:18,311 DEBUG SenderThread:723175 [sender.py:send():179] send: stats 2021-07-15 17:44:21,400 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:44:21,400 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:44:36,530 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:44:36,530 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:44:48,385 DEBUG SenderThread:723175 [sender.py:send():179] send: stats 2021-07-15 17:44:51,661 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:44:51,662 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:45:06,796 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:45:06,796 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:45:18,460 DEBUG SenderThread:723175 [sender.py:send():179] send: stats 2021-07-15 17:45:21,928 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:45:21,928 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:45:37,059 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:45:37,059 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:45:48,530 DEBUG SenderThread:723175 [sender.py:send():179] send: stats 2021-07-15 17:45:52,197 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:45:52,198 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:46:07,329 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:46:07,329 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:46:18,600 DEBUG SenderThread:723175 [sender.py:send():179] send: stats 2021-07-15 17:46:22,464 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:46:22,464 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:46:37,595 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:46:37,595 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:46:48,677 DEBUG SenderThread:723175 [sender.py:send():179] send: stats 2021-07-15 17:46:52,729 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:46:52,729 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:47:07,861 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:47:07,862 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:47:18,753 DEBUG SenderThread:723175 [sender.py:send():179] send: stats 2021-07-15 17:47:22,995 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: stop_status 2021-07-15 17:47:22,995 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: stop_status 2021-07-15 17:47:31,673 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: poll_exit 2021-07-15 17:47:31,673 DEBUG SenderThread:723175 [sender.py:send():179] send: telemetry 2021-07-15 17:47:31,673 DEBUG SenderThread:723175 [sender.py:send():179] send: exit 2021-07-15 17:47:31,673 INFO SenderThread:723175 [sender.py:send_exit():287] handling exit code: 1 2021-07-15 17:47:31,674 INFO SenderThread:723175 [sender.py:send_exit():295] send defer 2021-07-15 17:47:31,674 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: poll_exit 2021-07-15 17:47:31,674 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: defer 2021-07-15 17:47:31,674 INFO HandlerThread:723175 [handler.py:handle_request_defer():141] handle defer: 0 2021-07-15 17:47:31,674 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: defer 2021-07-15 17:47:31,674 INFO SenderThread:723175 [sender.py:send_request_defer():304] handle sender defer: 0 2021-07-15 17:47:31,675 INFO SenderThread:723175 [sender.py:transition_state():308] send defer: 1 2021-07-15 17:47:31,675 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: defer 2021-07-15 17:47:31,675 INFO HandlerThread:723175 [handler.py:handle_request_defer():141] handle defer: 1 2021-07-15 17:47:31,686 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: defer 2021-07-15 17:47:31,686 INFO SenderThread:723175 [sender.py:send_request_defer():304] handle sender defer: 1 2021-07-15 17:47:31,686 INFO SenderThread:723175 [sender.py:transition_state():308] send defer: 2 2021-07-15 17:47:31,686 DEBUG SenderThread:723175 [sender.py:send():179] send: stats 2021-07-15 17:47:31,687 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: defer 2021-07-15 17:47:31,687 INFO HandlerThread:723175 [handler.py:handle_request_defer():141] handle defer: 2 2021-07-15 17:47:31,687 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: defer 2021-07-15 17:47:31,687 INFO SenderThread:723175 [sender.py:send_request_defer():304] handle sender defer: 2 2021-07-15 17:47:31,687 INFO SenderThread:723175 [sender.py:transition_state():308] send defer: 3 2021-07-15 17:47:31,687 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: defer 2021-07-15 17:47:31,687 INFO HandlerThread:723175 [handler.py:handle_request_defer():141] handle defer: 3 2021-07-15 17:47:31,687 DEBUG SenderThread:723175 [sender.py:send():179] send: summary 2021-07-15 17:47:31,688 INFO SenderThread:723175 [sender.py:_save_file():841] saving file wandb-summary.json with policy end 2021-07-15 17:47:31,688 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: defer 2021-07-15 17:47:31,688 INFO SenderThread:723175 [sender.py:send_request_defer():304] handle sender defer: 3 2021-07-15 17:47:31,688 INFO SenderThread:723175 [sender.py:transition_state():308] send defer: 4 2021-07-15 17:47:31,688 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: defer 2021-07-15 17:47:31,688 INFO HandlerThread:723175 [handler.py:handle_request_defer():141] handle defer: 4 2021-07-15 17:47:31,688 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: defer 2021-07-15 17:47:31,688 INFO SenderThread:723175 [sender.py:send_request_defer():304] handle sender defer: 4 2021-07-15 17:47:31,776 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: poll_exit 2021-07-15 17:47:31,881 INFO SenderThread:723175 [sender.py:transition_state():308] send defer: 5 2021-07-15 17:47:31,881 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: poll_exit 2021-07-15 17:47:31,881 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: defer 2021-07-15 17:47:31,881 INFO HandlerThread:723175 [handler.py:handle_request_defer():141] handle defer: 5 2021-07-15 17:47:31,882 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: defer 2021-07-15 17:47:31,882 INFO SenderThread:723175 [sender.py:send_request_defer():304] handle sender defer: 5 2021-07-15 17:47:31,882 INFO SenderThread:723175 [dir_watcher.py:finish():282] shutting down directory watcher 2021-07-15 17:47:31,891 INFO SenderThread:723175 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/config.yaml 2021-07-15 17:47:31,891 INFO SenderThread:723175 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/wandb-summary.json 2021-07-15 17:47:31,891 INFO SenderThread:723175 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/output.log 2021-07-15 17:47:31,891 INFO SenderThread:723175 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files 2021-07-15 17:47:31,891 INFO SenderThread:723175 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/requirements.txt requirements.txt 2021-07-15 17:47:31,892 INFO SenderThread:723175 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/output.log output.log 2021-07-15 17:47:31,892 INFO SenderThread:723175 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/wandb-metadata.json wandb-metadata.json 2021-07-15 17:47:31,892 INFO SenderThread:723175 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/config.yaml config.yaml 2021-07-15 17:47:31,892 INFO SenderThread:723175 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/wandb-summary.json wandb-summary.json 2021-07-15 17:47:31,895 INFO SenderThread:723175 [sender.py:transition_state():308] send defer: 6 2021-07-15 17:47:31,896 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: defer 2021-07-15 17:47:31,896 INFO HandlerThread:723175 [handler.py:handle_request_defer():141] handle defer: 6 2021-07-15 17:47:31,896 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: defer 2021-07-15 17:47:31,896 INFO SenderThread:723175 [sender.py:send_request_defer():304] handle sender defer: 6 2021-07-15 17:47:31,896 INFO SenderThread:723175 [file_pusher.py:finish():177] shutting down file pusher 2021-07-15 17:47:31,983 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: poll_exit 2021-07-15 17:47:31,983 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: poll_exit 2021-07-15 17:47:32,085 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: poll_exit 2021-07-15 17:47:32,085 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: poll_exit 2021-07-15 17:47:32,187 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: poll_exit 2021-07-15 17:47:32,187 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: poll_exit 2021-07-15 17:47:32,289 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: poll_exit 2021-07-15 17:47:32,289 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: poll_exit 2021-07-15 17:47:32,333 INFO Thread-12 :723175 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/requirements.txt 2021-07-15 17:47:32,347 INFO Thread-13 :723175 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/output.log 2021-07-15 17:47:32,391 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: poll_exit 2021-07-15 17:47:32,391 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: poll_exit 2021-07-15 17:47:32,408 INFO Thread-14 :723175 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/config.yaml 2021-07-15 17:47:32,444 INFO Thread-15 :723175 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/files/wandb-summary.json 2021-07-15 17:47:32,493 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: poll_exit 2021-07-15 17:47:32,493 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: poll_exit 2021-07-15 17:47:32,595 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: poll_exit 2021-07-15 17:47:32,595 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: poll_exit 2021-07-15 17:47:32,645 INFO Thread-7 :723175 [sender.py:transition_state():308] send defer: 7 2021-07-15 17:47:32,645 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: defer 2021-07-15 17:47:32,645 INFO HandlerThread:723175 [handler.py:handle_request_defer():141] handle defer: 7 2021-07-15 17:47:32,646 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: defer 2021-07-15 17:47:32,646 INFO SenderThread:723175 [sender.py:send_request_defer():304] handle sender defer: 7 2021-07-15 17:47:32,697 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: poll_exit 2021-07-15 17:47:32,951 INFO SenderThread:723175 [sender.py:transition_state():308] send defer: 8 2021-07-15 17:47:32,951 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: poll_exit 2021-07-15 17:47:32,952 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: defer 2021-07-15 17:47:32,952 INFO HandlerThread:723175 [handler.py:handle_request_defer():141] handle defer: 8 2021-07-15 17:47:32,952 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: defer 2021-07-15 17:47:32,952 INFO SenderThread:723175 [sender.py:send_request_defer():304] handle sender defer: 8 2021-07-15 17:47:32,952 INFO SenderThread:723175 [sender.py:transition_state():308] send defer: 9 2021-07-15 17:47:32,953 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: defer 2021-07-15 17:47:32,953 INFO HandlerThread:723175 [handler.py:handle_request_defer():141] handle defer: 9 2021-07-15 17:47:32,953 DEBUG SenderThread:723175 [sender.py:send():179] send: final 2021-07-15 17:47:32,953 DEBUG SenderThread:723175 [sender.py:send():179] send: footer 2021-07-15 17:47:32,953 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: defer 2021-07-15 17:47:32,953 INFO SenderThread:723175 [sender.py:send_request_defer():304] handle sender defer: 9 2021-07-15 17:47:33,053 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: poll_exit 2021-07-15 17:47:33,053 DEBUG SenderThread:723175 [sender.py:send_request():193] send_request: poll_exit 2021-07-15 17:47:33,053 INFO SenderThread:723175 [file_pusher.py:join():182] waiting for file pusher 2021-07-15 17:47:33,055 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: get_summary 2021-07-15 17:47:33,055 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: sampled_history 2021-07-15 17:47:33,056 DEBUG HandlerThread:723175 [handler.py:handle_request():124] handle_request: shutdown 2021-07-15 17:47:33,056 INFO HandlerThread:723175 [handler.py:finish():638] shutting down handler 2021-07-15 17:47:33,953 INFO WriterThread:723175 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210715_174147-3nkn7hxg/run-3nkn7hxg.wandb 2021-07-15 17:47:34,054 INFO SenderThread:723175 [sender.py:finish():945] shutting down sender 2021-07-15 17:47:34,054 INFO SenderThread:723175 [file_pusher.py:finish():177] shutting down file pusher 2021-07-15 17:47:34,054 INFO SenderThread:723175 [file_pusher.py:join():182] waiting for file pusher 2021-07-15 17:47:34,056 INFO MainThread:723175 [internal.py:handle_exit():78] Internal process exited