| 2021-07-15 00:37:39,369 INFO MainThread:622170 [internal.py:wandb_internal():88] W&B internal server running at pid: 622170, started at: 2021-07-15 00:37:39.369673 | |
| 2021-07-15 00:37:39,371 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: check_version | |
| 2021-07-15 00:37:39,372 INFO WriterThread:622170 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/run-kj539nne.wandb | |
| 2021-07-15 00:37:39,373 DEBUG SenderThread:622170 [sender.py:send():179] send: header | |
| 2021-07-15 00:37:39,373 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: check_version | |
| 2021-07-15 00:37:39,410 DEBUG SenderThread:622170 [sender.py:send():179] send: run | |
| 2021-07-15 00:37:39,578 INFO SenderThread:622170 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files | |
| 2021-07-15 00:37:39,578 INFO SenderThread:622170 [sender.py:_start_run_threads():716] run started: kj539nne with start time 1626309458 | |
| 2021-07-15 00:37:39,578 DEBUG SenderThread:622170 [sender.py:send():179] send: summary | |
| 2021-07-15 00:37:39,578 INFO SenderThread:622170 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-15 00:37:39,579 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: run_start | |
| 2021-07-15 00:37:40,586 INFO Thread-8 :622170 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/wandb-summary.json | |
| 2021-07-15 00:37:40,758 DEBUG HandlerThread:622170 [meta.py:__init__():39] meta init | |
| 2021-07-15 00:37:40,758 DEBUG HandlerThread:622170 [meta.py:__init__():53] meta init done | |
| 2021-07-15 00:37:40,759 DEBUG HandlerThread:622170 [meta.py:probe():210] probe | |
| 2021-07-15 00:37:40,760 DEBUG HandlerThread:622170 [meta.py:_setup_git():200] setup git | |
| 2021-07-15 00:37:40,791 DEBUG HandlerThread:622170 [meta.py:_setup_git():207] setup git done | |
| 2021-07-15 00:37:40,791 DEBUG HandlerThread:622170 [meta.py:_save_pip():57] save pip | |
| 2021-07-15 00:37:40,792 DEBUG HandlerThread:622170 [meta.py:_save_pip():71] save pip done | |
| 2021-07-15 00:37:40,792 DEBUG HandlerThread:622170 [meta.py:probe():252] probe done | |
| 2021-07-15 00:37:40,796 DEBUG SenderThread:622170 [sender.py:send():179] send: files | |
| 2021-07-15 00:37:40,796 INFO SenderThread:622170 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now | |
| 2021-07-15 00:37:40,802 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:37:40,802 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:37:40,930 DEBUG SenderThread:622170 [sender.py:send():179] send: config | |
| 2021-07-15 00:37:40,931 DEBUG SenderThread:622170 [sender.py:send():179] send: config | |
| 2021-07-15 00:37:40,931 DEBUG SenderThread:622170 [sender.py:send():179] send: config | |
| 2021-07-15 00:37:41,235 INFO Thread-11 :622170 [upload_job.py:push():137] Uploaded file /tmp/tmpzvfgl2cpwandb/2r82kf3f-wandb-metadata.json | |
| 2021-07-15 00:37:41,584 INFO Thread-8 :622170 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/output.log | |
| 2021-07-15 00:37:41,585 INFO Thread-8 :622170 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/requirements.txt | |
| 2021-07-15 00:37:41,585 INFO Thread-8 :622170 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/wandb-metadata.json | |
| 2021-07-15 00:37:55,591 INFO Thread-8 :622170 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/output.log | |
| 2021-07-15 00:37:55,933 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:37:55,933 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:37:57,592 INFO Thread-8 :622170 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/output.log | |
| 2021-07-15 00:38:08,843 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:38:10,597 INFO Thread-8 :622170 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/config.yaml | |
| 2021-07-15 00:38:11,072 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:38:11,073 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:38:26,208 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:38:26,208 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:38:38,927 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:38:41,341 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:38:41,341 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:38:56,476 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:38:56,476 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:39:09,001 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:39:11,609 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:39:11,609 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:39:26,745 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:39:26,746 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:39:39,080 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:39:41,877 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:39:41,877 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:39:57,009 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:39:57,010 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:40:09,159 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:40:12,144 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:40:12,145 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:40:27,280 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:40:27,280 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:40:39,237 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:40:42,413 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:40:42,414 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:40:57,550 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:40:57,551 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:41:09,313 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:41:12,683 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:41:12,684 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:41:27,817 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:41:27,818 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:41:39,376 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:41:42,956 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:41:42,956 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:41:58,089 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:41:58,090 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:42:09,437 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:42:13,226 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:42:13,226 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:42:28,355 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:42:28,356 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:42:39,515 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:42:43,487 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:42:43,488 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:42:58,620 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:42:58,621 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:43:09,593 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:43:13,756 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:43:13,756 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:43:28,889 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:43:28,889 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:43:39,663 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:43:44,020 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:43:44,020 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:43:59,151 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:43:59,151 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:44:09,739 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:44:14,283 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:44:14,284 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:44:29,414 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:44:29,414 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:44:39,817 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:44:44,546 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:44:44,547 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:44:59,680 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:44:59,681 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:45:09,895 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:45:14,816 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:45:14,816 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:45:29,950 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:45:29,950 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:45:39,970 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:45:45,082 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:45:45,082 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:46:00,214 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:46:00,215 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:46:10,046 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:46:15,347 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:46:15,347 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:46:30,482 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:46:30,483 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:46:40,123 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:46:45,616 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:46:45,616 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:47:00,745 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:47:00,745 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:47:10,194 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:47:15,881 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:47:15,881 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:47:31,011 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:47:31,012 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:47:40,259 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:47:46,146 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:47:46,147 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:48:01,282 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:48:01,283 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:48:10,323 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:48:16,413 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:48:16,414 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:48:31,562 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:48:31,563 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:48:40,388 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:48:46,694 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:48:46,695 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:49:02,047 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:49:02,047 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:49:10,453 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:49:17,178 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:49:17,179 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:49:32,311 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:49:32,311 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:49:40,521 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:49:47,444 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:49:47,444 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:50:02,576 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:50:02,576 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:50:10,594 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:50:17,705 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:50:17,706 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:50:32,835 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:50:32,836 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:50:40,670 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:50:47,970 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:50:47,970 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:51:03,105 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:51:03,105 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:51:10,739 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:51:18,234 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 00:51:18,235 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 00:51:21,907 INFO Thread-8 :622170 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/output.log | |
| 2021-07-15 00:51:23,131 DEBUG SenderThread:622170 [sender.py:send():179] send: telemetry | |
| 2021-07-15 00:51:23,132 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 00:51:23,132 DEBUG SenderThread:622170 [sender.py:send():179] send: exit | |
| 2021-07-15 00:51:23,132 INFO SenderThread:622170 [sender.py:send_exit():287] handling exit code: 1 | |
| 2021-07-15 00:51:23,133 INFO SenderThread:622170 [sender.py:send_exit():295] send defer | |
| 2021-07-15 00:51:23,133 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 00:51:23,134 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 00:51:23,134 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 0 | |
| 2021-07-15 00:51:23,134 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 00:51:23,134 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 0 | |
| 2021-07-15 00:51:23,135 INFO SenderThread:622170 [sender.py:transition_state():308] send defer: 1 | |
| 2021-07-15 00:51:23,135 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 00:51:23,135 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 1 | |
| 2021-07-15 00:51:23,170 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 00:51:23,170 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 1 | |
| 2021-07-15 00:51:23,170 INFO SenderThread:622170 [sender.py:transition_state():308] send defer: 2 | |
| 2021-07-15 00:51:23,170 DEBUG SenderThread:622170 [sender.py:send():179] send: stats | |
| 2021-07-15 00:51:23,170 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 00:51:23,171 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 2 | |
| 2021-07-15 00:51:23,171 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 00:51:23,171 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 2 | |
| 2021-07-15 00:51:23,171 INFO SenderThread:622170 [sender.py:transition_state():308] send defer: 3 | |
| 2021-07-15 00:51:23,171 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 00:51:23,171 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 3 | |
| 2021-07-15 00:51:23,171 DEBUG SenderThread:622170 [sender.py:send():179] send: summary | |
| 2021-07-15 00:51:23,172 INFO SenderThread:622170 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-15 00:51:23,172 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 00:51:23,172 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 3 | |
| 2021-07-15 00:51:23,172 INFO SenderThread:622170 [sender.py:transition_state():308] send defer: 4 | |
| 2021-07-15 00:51:23,172 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 00:51:23,172 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 4 | |
| 2021-07-15 00:51:23,173 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 00:51:23,173 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 4 | |
| 2021-07-15 00:51:23,236 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 00:51:23,352 INFO SenderThread:622170 [sender.py:transition_state():308] send defer: 5 | |
| 2021-07-15 00:51:23,352 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 00:51:23,353 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 00:51:23,353 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 5 | |
| 2021-07-15 00:51:23,353 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 00:51:23,353 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 5 | |
| 2021-07-15 00:51:23,353 INFO SenderThread:622170 [dir_watcher.py:finish():282] shutting down directory watcher | |
| 2021-07-15 00:51:23,454 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 00:51:23,908 INFO Thread-8 :622170 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/output.log | |
| 2021-07-15 00:51:23,909 INFO SenderThread:622170 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/wandb-summary.json | |
| 2021-07-15 00:51:23,909 INFO SenderThread:622170 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/config.yaml | |
| 2021-07-15 00:51:23,909 INFO SenderThread:622170 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files | |
| 2021-07-15 00:51:23,909 INFO SenderThread:622170 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/requirements.txt requirements.txt | |
| 2021-07-15 00:51:23,909 INFO SenderThread:622170 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/output.log output.log | |
| 2021-07-15 00:51:23,909 INFO SenderThread:622170 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/wandb-metadata.json wandb-metadata.json | |
| 2021-07-15 00:51:23,910 INFO SenderThread:622170 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/config.yaml config.yaml | |
| 2021-07-15 00:51:23,913 INFO SenderThread:622170 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/wandb-summary.json wandb-summary.json | |
| 2021-07-15 00:51:23,913 INFO SenderThread:622170 [sender.py:transition_state():308] send defer: 6 | |
| 2021-07-15 00:51:23,913 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 00:51:23,917 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 00:51:23,917 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 6 | |
| 2021-07-15 00:51:23,917 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 00:51:23,917 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 6 | |
| 2021-07-15 00:51:23,917 INFO SenderThread:622170 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-15 00:51:24,016 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 00:51:24,016 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 00:51:24,118 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 00:51:24,118 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 00:51:24,220 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 00:51:24,220 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 00:51:24,322 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 00:51:24,322 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 00:51:24,352 INFO Thread-15 :622170 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/wandb-summary.json | |
| 2021-07-15 00:51:24,355 INFO Thread-12 :622170 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/requirements.txt | |
| 2021-07-15 00:51:24,384 INFO Thread-14 :622170 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/config.yaml | |
| 2021-07-15 00:51:24,394 INFO Thread-13 :622170 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/output.log | |
| 2021-07-15 00:51:24,424 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 00:51:24,424 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 00:51:24,526 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 00:51:24,526 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 00:51:24,595 INFO Thread-7 :622170 [sender.py:transition_state():308] send defer: 7 | |
| 2021-07-15 00:51:24,595 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 00:51:24,595 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 7 | |
| 2021-07-15 00:51:24,595 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 00:51:24,595 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 7 | |
| 2021-07-15 00:51:24,627 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 00:51:24,878 INFO SenderThread:622170 [sender.py:transition_state():308] send defer: 8 | |
| 2021-07-15 00:51:24,879 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 00:51:24,879 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 00:51:24,879 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 8 | |
| 2021-07-15 00:51:24,879 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 00:51:24,879 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 8 | |
| 2021-07-15 00:51:24,879 INFO SenderThread:622170 [sender.py:transition_state():308] send defer: 9 | |
| 2021-07-15 00:51:24,880 DEBUG SenderThread:622170 [sender.py:send():179] send: final | |
| 2021-07-15 00:51:24,880 DEBUG SenderThread:622170 [sender.py:send():179] send: footer | |
| 2021-07-15 00:51:24,880 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 00:51:24,880 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 9 | |
| 2021-07-15 00:51:24,880 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 00:51:24,880 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 9 | |
| 2021-07-15 00:51:24,980 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 00:51:24,981 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 00:51:24,981 INFO SenderThread:622170 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-15 00:51:24,982 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: get_summary | |
| 2021-07-15 00:51:24,983 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: sampled_history | |
| 2021-07-15 00:51:24,984 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: shutdown | |
| 2021-07-15 00:51:24,984 INFO HandlerThread:622170 [handler.py:finish():638] shutting down handler | |
| 2021-07-15 00:51:25,880 INFO WriterThread:622170 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/run-kj539nne.wandb | |
| 2021-07-15 00:51:25,981 INFO SenderThread:622170 [sender.py:finish():945] shutting down sender | |
| 2021-07-15 00:51:25,981 INFO SenderThread:622170 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-15 00:51:25,981 INFO SenderThread:622170 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-15 00:51:25,984 INFO MainThread:622170 [internal.py:handle_exit():78] Internal process exited | |