| 2021-07-14 18:33:05,441 INFO MainThread:551470 [internal.py:wandb_internal():88] W&B internal server running at pid: 551470, started at: 2021-07-14 18:33:05.441281 | |
| 2021-07-14 18:33:05,443 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: check_version | |
| 2021-07-14 18:33:05,443 INFO WriterThread:551470 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/run-3d4rorca.wandb | |
| 2021-07-14 18:33:05,444 DEBUG SenderThread:551470 [sender.py:send():179] send: header | |
| 2021-07-14 18:33:05,444 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: check_version | |
| 2021-07-14 18:33:05,481 DEBUG SenderThread:551470 [sender.py:send():179] send: run | |
| 2021-07-14 18:33:05,658 INFO SenderThread:551470 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files | |
| 2021-07-14 18:33:05,658 INFO SenderThread:551470 [sender.py:_start_run_threads():716] run started: 3d4rorca with start time 1626287584 | |
| 2021-07-14 18:33:05,658 DEBUG SenderThread:551470 [sender.py:send():179] send: summary | |
| 2021-07-14 18:33:05,659 INFO SenderThread:551470 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 18:33:05,659 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: run_start | |
| 2021-07-14 18:33:06,662 INFO Thread-8 :551470 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/wandb-summary.json | |
| 2021-07-14 18:33:06,772 DEBUG HandlerThread:551470 [meta.py:__init__():39] meta init | |
| 2021-07-14 18:33:06,772 DEBUG HandlerThread:551470 [meta.py:__init__():53] meta init done | |
| 2021-07-14 18:33:06,772 DEBUG HandlerThread:551470 [meta.py:probe():210] probe | |
| 2021-07-14 18:33:06,773 DEBUG HandlerThread:551470 [meta.py:_setup_git():200] setup git | |
| 2021-07-14 18:33:06,799 DEBUG HandlerThread:551470 [meta.py:_setup_git():207] setup git done | |
| 2021-07-14 18:33:06,799 DEBUG HandlerThread:551470 [meta.py:_save_pip():57] save pip | |
| 2021-07-14 18:33:06,799 DEBUG HandlerThread:551470 [meta.py:_save_pip():71] save pip done | |
| 2021-07-14 18:33:06,799 DEBUG HandlerThread:551470 [meta.py:probe():252] probe done | |
| 2021-07-14 18:33:06,802 DEBUG SenderThread:551470 [sender.py:send():179] send: files | |
| 2021-07-14 18:33:06,802 INFO SenderThread:551470 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now | |
| 2021-07-14 18:33:06,808 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:33:06,808 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:33:06,939 DEBUG SenderThread:551470 [sender.py:send():179] send: config | |
| 2021-07-14 18:33:06,940 DEBUG SenderThread:551470 [sender.py:send():179] send: config | |
| 2021-07-14 18:33:06,940 DEBUG SenderThread:551470 [sender.py:send():179] send: config | |
| 2021-07-14 18:33:07,267 INFO Thread-11 :551470 [upload_job.py:push():137] Uploaded file /tmp/tmpwkna8gakwandb/1910yhst-wandb-metadata.json | |
| 2021-07-14 18:33:07,661 INFO Thread-8 :551470 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/output.log | |
| 2021-07-14 18:33:07,661 INFO Thread-8 :551470 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/requirements.txt | |
| 2021-07-14 18:33:07,661 INFO Thread-8 :551470 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/wandb-metadata.json | |
| 2021-07-14 18:33:21,941 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:33:21,941 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:33:25,667 INFO Thread-8 :551470 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/output.log | |
| 2021-07-14 18:33:34,849 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:33:36,671 INFO Thread-8 :551470 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/config.yaml | |
| 2021-07-14 18:33:37,073 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:33:37,073 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:33:52,207 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:33:52,207 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:34:04,916 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:34:07,343 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:34:07,343 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:34:22,477 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:34:22,477 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:34:34,976 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:34:37,613 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:34:37,613 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:34:52,747 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:34:52,747 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:35:05,044 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:35:07,878 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:35:07,879 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:35:23,013 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:35:23,013 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:35:35,110 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:35:38,147 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:35:38,148 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:35:53,282 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:35:53,282 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:36:05,181 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:36:08,416 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:36:08,417 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:36:23,549 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:36:23,549 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:36:35,251 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:36:38,681 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:36:38,681 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:36:53,817 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:36:53,817 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:37:05,320 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:37:08,952 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:37:08,953 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:37:24,087 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:37:24,087 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:37:35,392 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:37:39,221 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:37:39,222 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:37:54,352 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:37:54,353 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:38:05,464 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:38:09,485 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:38:09,485 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:38:24,626 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:38:24,626 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:38:35,535 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:38:39,761 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:38:39,761 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:38:54,893 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:38:54,893 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:39:05,597 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:39:10,025 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:39:10,026 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:39:25,164 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:39:25,164 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:39:35,672 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:39:40,296 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:39:40,297 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:39:55,430 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:39:55,430 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:40:05,746 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:40:10,563 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:40:10,564 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:40:25,697 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:40:25,698 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:40:35,821 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:40:40,834 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:40:40,834 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:40:55,966 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:40:55,967 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:41:05,888 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:41:11,114 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:41:11,114 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:41:11,814 INFO Thread-8 :551470 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/output.log | |
| 2021-07-14 18:41:26,276 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:41:26,277 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:41:35,966 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:41:41,417 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:41:41,418 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:41:56,548 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:41:56,548 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:42:06,047 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:42:11,683 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:42:11,684 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:42:26,818 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:42:26,818 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:42:36,121 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:42:41,950 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:42:41,951 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:42:51,640 DEBUG SenderThread:551470 [sender.py:send():179] send: telemetry | |
| 2021-07-14 18:42:51,641 DEBUG SenderThread:551470 [sender.py:send():179] send: exit | |
| 2021-07-14 18:42:51,641 INFO SenderThread:551470 [sender.py:send_exit():287] handling exit code: 1 | |
| 2021-07-14 18:42:51,641 INFO SenderThread:551470 [sender.py:send_exit():295] send defer | |
| 2021-07-14 18:42:51,641 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 18:42:51,642 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 18:42:51,642 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 0 | |
| 2021-07-14 18:42:51,642 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 18:42:51,642 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 18:42:51,642 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 0 | |
| 2021-07-14 18:42:51,642 INFO SenderThread:551470 [sender.py:transition_state():308] send defer: 1 | |
| 2021-07-14 18:42:51,643 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 18:42:51,643 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 1 | |
| 2021-07-14 18:42:51,659 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 18:42:51,660 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 1 | |
| 2021-07-14 18:42:51,660 INFO SenderThread:551470 [sender.py:transition_state():308] send defer: 2 | |
| 2021-07-14 18:42:51,660 DEBUG SenderThread:551470 [sender.py:send():179] send: stats | |
| 2021-07-14 18:42:51,660 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 18:42:51,660 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 2 | |
| 2021-07-14 18:42:51,660 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 18:42:51,660 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 2 | |
| 2021-07-14 18:42:51,660 INFO SenderThread:551470 [sender.py:transition_state():308] send defer: 3 | |
| 2021-07-14 18:42:51,661 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 18:42:51,661 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 3 | |
| 2021-07-14 18:42:51,661 DEBUG SenderThread:551470 [sender.py:send():179] send: summary | |
| 2021-07-14 18:42:51,661 INFO SenderThread:551470 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 18:42:51,661 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 18:42:51,661 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 3 | |
| 2021-07-14 18:42:51,661 INFO SenderThread:551470 [sender.py:transition_state():308] send defer: 4 | |
| 2021-07-14 18:42:51,662 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 18:42:51,662 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 4 | |
| 2021-07-14 18:42:51,662 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 18:42:51,662 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 4 | |
| 2021-07-14 18:42:51,744 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 18:42:51,838 INFO SenderThread:551470 [sender.py:transition_state():308] send defer: 5 | |
| 2021-07-14 18:42:51,838 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 18:42:51,838 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 18:42:51,838 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 5 | |
| 2021-07-14 18:42:51,839 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 18:42:51,839 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 5 | |
| 2021-07-14 18:42:51,839 INFO SenderThread:551470 [dir_watcher.py:finish():282] shutting down directory watcher | |
| 2021-07-14 18:42:51,854 INFO Thread-8 :551470 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/config.yaml | |
| 2021-07-14 18:42:51,855 INFO SenderThread:551470 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/output.log | |
| 2021-07-14 18:42:51,855 INFO SenderThread:551470 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/wandb-summary.json | |
| 2021-07-14 18:42:51,855 INFO SenderThread:551470 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files | |
| 2021-07-14 18:42:51,855 INFO SenderThread:551470 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/requirements.txt requirements.txt | |
| 2021-07-14 18:42:51,855 INFO SenderThread:551470 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/output.log output.log | |
| 2021-07-14 18:42:51,856 INFO SenderThread:551470 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/wandb-metadata.json wandb-metadata.json | |
| 2021-07-14 18:42:51,856 INFO SenderThread:551470 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/config.yaml config.yaml | |
| 2021-07-14 18:42:51,856 INFO SenderThread:551470 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/wandb-summary.json wandb-summary.json | |
| 2021-07-14 18:42:51,856 INFO SenderThread:551470 [sender.py:transition_state():308] send defer: 6 | |
| 2021-07-14 18:42:51,862 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 18:42:51,862 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 6 | |
| 2021-07-14 18:42:51,865 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 18:42:51,865 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 6 | |
| 2021-07-14 18:42:51,865 INFO SenderThread:551470 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-14 18:42:51,939 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 18:42:51,940 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 18:42:52,041 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 18:42:52,041 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 18:42:52,143 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 18:42:52,143 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 18:42:52,245 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 18:42:52,245 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 18:42:52,308 INFO Thread-12 :551470 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/requirements.txt | |
| 2021-07-14 18:42:52,325 INFO Thread-13 :551470 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/output.log | |
| 2021-07-14 18:42:52,341 INFO Thread-15 :551470 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/wandb-summary.json | |
| 2021-07-14 18:42:52,346 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 18:42:52,346 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 18:42:52,361 INFO Thread-14 :551470 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/files/config.yaml | |
| 2021-07-14 18:42:52,448 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 18:42:52,448 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 18:42:52,549 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 18:42:52,550 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 18:42:52,562 INFO Thread-7 :551470 [sender.py:transition_state():308] send defer: 7 | |
| 2021-07-14 18:42:52,562 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 18:42:52,562 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 7 | |
| 2021-07-14 18:42:52,562 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 18:42:52,562 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 7 | |
| 2021-07-14 18:42:52,651 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 18:42:52,837 INFO SenderThread:551470 [sender.py:transition_state():308] send defer: 8 | |
| 2021-07-14 18:42:52,837 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 18:42:52,837 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 18:42:52,837 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 8 | |
| 2021-07-14 18:42:52,838 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 18:42:52,838 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 8 | |
| 2021-07-14 18:42:52,838 INFO SenderThread:551470 [sender.py:transition_state():308] send defer: 9 | |
| 2021-07-14 18:42:52,838 DEBUG SenderThread:551470 [sender.py:send():179] send: final | |
| 2021-07-14 18:42:52,838 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 18:42:52,838 INFO HandlerThread:551470 [handler.py:handle_request_defer():141] handle defer: 9 | |
| 2021-07-14 18:42:52,839 DEBUG SenderThread:551470 [sender.py:send():179] send: footer | |
| 2021-07-14 18:42:52,839 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 18:42:52,839 INFO SenderThread:551470 [sender.py:send_request_defer():304] handle sender defer: 9 | |
| 2021-07-14 18:42:52,938 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 18:42:52,939 DEBUG SenderThread:551470 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 18:42:52,939 INFO SenderThread:551470 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-14 18:42:52,940 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: get_summary | |
| 2021-07-14 18:42:52,941 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: sampled_history | |
| 2021-07-14 18:42:52,941 DEBUG HandlerThread:551470 [handler.py:handle_request():124] handle_request: shutdown | |
| 2021-07-14 18:42:52,941 INFO HandlerThread:551470 [handler.py:finish():638] shutting down handler | |
| 2021-07-14 18:42:53,839 INFO WriterThread:551470 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_183304-3d4rorca/run-3d4rorca.wandb | |
| 2021-07-14 18:42:53,939 INFO SenderThread:551470 [sender.py:finish():945] shutting down sender | |
| 2021-07-14 18:42:53,939 INFO SenderThread:551470 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-14 18:42:53,939 INFO SenderThread:551470 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-14 18:42:53,941 INFO MainThread:551470 [internal.py:handle_exit():78] Internal process exited | |