| 2021-07-14 23:27:04,669 INFO MainThread:606786 [internal.py:wandb_internal():88] W&B internal server running at pid: 606786, started at: 2021-07-14 23:27:04.669257 | |
| 2021-07-14 23:27:04,671 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: check_version | |
| 2021-07-14 23:27:04,671 INFO WriterThread:606786 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/run-1jijl27o.wandb | |
| 2021-07-14 23:27:04,672 DEBUG SenderThread:606786 [sender.py:send():179] send: header | |
| 2021-07-14 23:27:04,672 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: check_version | |
| 2021-07-14 23:27:04,709 DEBUG SenderThread:606786 [sender.py:send():179] send: run | |
| 2021-07-14 23:27:04,879 INFO SenderThread:606786 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files | |
| 2021-07-14 23:27:04,880 INFO SenderThread:606786 [sender.py:_start_run_threads():716] run started: 1jijl27o with start time 1626305224 | |
| 2021-07-14 23:27:04,880 DEBUG SenderThread:606786 [sender.py:send():179] send: summary | |
| 2021-07-14 23:27:04,880 INFO SenderThread:606786 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 23:27:04,880 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: run_start | |
| 2021-07-14 23:27:05,883 INFO Thread-8 :606786 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/wandb-summary.json | |
| 2021-07-14 23:27:05,989 DEBUG HandlerThread:606786 [meta.py:__init__():39] meta init | |
| 2021-07-14 23:27:05,989 DEBUG HandlerThread:606786 [meta.py:__init__():53] meta init done | |
| 2021-07-14 23:27:05,989 DEBUG HandlerThread:606786 [meta.py:probe():210] probe | |
| 2021-07-14 23:27:05,990 DEBUG HandlerThread:606786 [meta.py:_setup_git():200] setup git | |
| 2021-07-14 23:27:06,020 DEBUG HandlerThread:606786 [meta.py:_setup_git():207] setup git done | |
| 2021-07-14 23:27:06,020 DEBUG HandlerThread:606786 [meta.py:_save_pip():57] save pip | |
| 2021-07-14 23:27:06,021 DEBUG HandlerThread:606786 [meta.py:_save_pip():71] save pip done | |
| 2021-07-14 23:27:06,021 DEBUG HandlerThread:606786 [meta.py:probe():252] probe done | |
| 2021-07-14 23:27:06,024 DEBUG SenderThread:606786 [sender.py:send():179] send: files | |
| 2021-07-14 23:27:06,024 INFO SenderThread:606786 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now | |
| 2021-07-14 23:27:06,032 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:27:06,033 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:27:06,166 DEBUG SenderThread:606786 [sender.py:send():179] send: config | |
| 2021-07-14 23:27:06,166 DEBUG SenderThread:606786 [sender.py:send():179] send: config | |
| 2021-07-14 23:27:06,166 DEBUG SenderThread:606786 [sender.py:send():179] send: config | |
| 2021-07-14 23:27:06,504 INFO Thread-11 :606786 [upload_job.py:push():137] Uploaded file /tmp/tmpf02i07o_wandb/gludn8x0-wandb-metadata.json | |
| 2021-07-14 23:27:06,881 INFO Thread-8 :606786 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/requirements.txt | |
| 2021-07-14 23:27:06,881 INFO Thread-8 :606786 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/wandb-metadata.json | |
| 2021-07-14 23:27:06,881 INFO Thread-8 :606786 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/output.log | |
| 2021-07-14 23:27:20,887 INFO Thread-8 :606786 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/output.log | |
| 2021-07-14 23:27:21,168 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:27:21,168 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:27:22,888 INFO Thread-8 :606786 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/output.log | |
| 2021-07-14 23:27:34,069 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:27:35,894 INFO Thread-8 :606786 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/config.yaml | |
| 2021-07-14 23:27:36,301 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:27:36,302 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:27:51,435 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:27:51,435 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:28:04,145 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:28:06,571 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:28:06,571 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:28:21,704 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:28:21,704 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:28:34,210 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:28:36,838 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:28:36,838 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:28:51,969 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:28:51,970 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:29:04,276 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:29:11,606 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:29:11,606 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:29:26,738 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:29:26,739 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:29:34,341 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:29:41,875 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:29:41,875 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:29:57,008 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:29:57,009 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:30:04,407 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:30:12,141 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:30:12,142 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:30:27,275 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:30:27,276 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:30:34,475 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:30:42,407 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:30:42,407 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:30:57,540 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:30:57,541 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:31:04,540 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:31:12,674 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:31:12,674 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:31:27,806 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:31:27,807 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:31:34,612 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:31:42,940 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:31:42,940 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:31:58,074 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:31:58,075 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:32:04,688 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:32:13,208 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:32:13,208 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:32:28,339 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:32:28,340 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:32:34,767 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:32:43,471 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:32:43,472 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:32:58,604 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:32:58,604 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:33:04,843 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:33:13,739 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:33:13,739 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:33:28,873 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:33:28,874 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:33:34,917 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:33:44,007 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:33:44,007 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:33:59,140 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:33:59,141 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:34:04,989 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:34:14,274 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:34:14,275 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:34:29,406 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:34:29,406 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:34:35,063 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:34:44,538 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:34:44,538 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:34:59,670 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:34:59,671 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:35:05,136 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:35:14,806 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:35:14,806 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:35:29,937 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:35:29,938 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:35:35,212 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:35:45,083 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:35:45,084 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:36:00,215 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:36:00,215 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:36:05,289 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:36:15,359 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:36:15,359 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:36:30,491 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:36:30,492 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:36:35,354 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:36:45,626 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:36:45,626 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:37:00,758 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:37:00,758 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:37:05,418 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:37:15,896 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:37:15,896 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:37:31,030 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:37:31,031 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:37:35,484 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:37:46,162 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:37:46,162 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:38:01,293 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:38:01,294 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:38:05,552 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:38:16,425 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:38:16,425 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:38:31,557 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:38:31,557 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:38:35,624 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:38:46,691 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:38:46,691 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:39:01,823 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:39:01,824 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:39:05,695 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:39:16,955 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:39:16,955 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:39:32,087 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:39:32,088 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:39:35,769 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:39:47,220 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:39:47,221 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:40:02,350 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:40:02,351 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:40:05,843 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:40:17,481 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:40:17,482 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:40:32,615 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:40:32,616 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:40:35,918 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:40:47,205 INFO Thread-8 :606786 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/output.log | |
| 2021-07-14 23:40:48,363 DEBUG SenderThread:606786 [sender.py:send():179] send: telemetry | |
| 2021-07-14 23:40:48,364 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:40:48,364 DEBUG SenderThread:606786 [sender.py:send():179] send: exit | |
| 2021-07-14 23:40:48,364 INFO SenderThread:606786 [sender.py:send_exit():287] handling exit code: 1 | |
| 2021-07-14 23:40:48,366 INFO SenderThread:606786 [sender.py:send_exit():295] send defer | |
| 2021-07-14 23:40:48,366 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:40:48,367 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:40:48,367 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 0 | |
| 2021-07-14 23:40:48,367 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:40:48,367 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 0 | |
| 2021-07-14 23:40:48,367 INFO SenderThread:606786 [sender.py:transition_state():308] send defer: 1 | |
| 2021-07-14 23:40:48,368 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:40:48,368 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 1 | |
| 2021-07-14 23:40:48,446 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:40:48,446 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 1 | |
| 2021-07-14 23:40:48,446 INFO SenderThread:606786 [sender.py:transition_state():308] send defer: 2 | |
| 2021-07-14 23:40:48,446 DEBUG SenderThread:606786 [sender.py:send():179] send: stats | |
| 2021-07-14 23:40:48,447 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:40:48,447 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 2 | |
| 2021-07-14 23:40:48,447 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:40:48,447 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 2 | |
| 2021-07-14 23:40:48,447 INFO SenderThread:606786 [sender.py:transition_state():308] send defer: 3 | |
| 2021-07-14 23:40:48,447 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:40:48,448 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 3 | |
| 2021-07-14 23:40:48,448 DEBUG SenderThread:606786 [sender.py:send():179] send: summary | |
| 2021-07-14 23:40:48,448 INFO SenderThread:606786 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 23:40:48,449 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:40:48,449 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 3 | |
| 2021-07-14 23:40:48,449 INFO SenderThread:606786 [sender.py:transition_state():308] send defer: 4 | |
| 2021-07-14 23:40:48,449 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:40:48,449 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 4 | |
| 2021-07-14 23:40:48,449 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:40:48,449 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 4 | |
| 2021-07-14 23:40:48,469 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:40:48,629 INFO SenderThread:606786 [sender.py:transition_state():308] send defer: 5 | |
| 2021-07-14 23:40:48,629 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:40:48,630 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:40:48,630 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 5 | |
| 2021-07-14 23:40:48,630 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:40:48,630 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 5 | |
| 2021-07-14 23:40:48,630 INFO SenderThread:606786 [dir_watcher.py:finish():282] shutting down directory watcher | |
| 2021-07-14 23:40:48,732 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:40:49,206 INFO Thread-8 :606786 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/config.yaml | |
| 2021-07-14 23:40:49,207 INFO SenderThread:606786 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/output.log | |
| 2021-07-14 23:40:49,207 INFO SenderThread:606786 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/wandb-summary.json | |
| 2021-07-14 23:40:49,207 INFO SenderThread:606786 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files | |
| 2021-07-14 23:40:49,207 INFO SenderThread:606786 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/requirements.txt requirements.txt | |
| 2021-07-14 23:40:49,208 INFO SenderThread:606786 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/output.log output.log | |
| 2021-07-14 23:40:49,208 INFO SenderThread:606786 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/wandb-metadata.json wandb-metadata.json | |
| 2021-07-14 23:40:49,208 INFO SenderThread:606786 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/config.yaml config.yaml | |
| 2021-07-14 23:40:49,208 INFO SenderThread:606786 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/wandb-summary.json wandb-summary.json | |
| 2021-07-14 23:40:49,209 INFO SenderThread:606786 [sender.py:transition_state():308] send defer: 6 | |
| 2021-07-14 23:40:49,209 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:40:49,210 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:40:49,210 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 6 | |
| 2021-07-14 23:40:49,215 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:40:49,216 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 6 | |
| 2021-07-14 23:40:49,216 INFO SenderThread:606786 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-14 23:40:49,311 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:40:49,311 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:40:49,413 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:40:49,413 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:40:49,515 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:40:49,515 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:40:49,617 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:40:49,618 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:40:49,652 INFO Thread-12 :606786 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/requirements.txt | |
| 2021-07-14 23:40:49,653 INFO Thread-14 :606786 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/config.yaml | |
| 2021-07-14 23:40:49,698 INFO Thread-15 :606786 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/wandb-summary.json | |
| 2021-07-14 23:40:49,719 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:40:49,720 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:40:49,739 INFO Thread-13 :606786 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/output.log | |
| 2021-07-14 23:40:49,821 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:40:49,822 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:40:49,923 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:40:49,923 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:40:49,940 INFO Thread-7 :606786 [sender.py:transition_state():308] send defer: 7 | |
| 2021-07-14 23:40:49,940 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:40:49,940 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 7 | |
| 2021-07-14 23:40:49,940 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:40:49,940 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 7 | |
| 2021-07-14 23:40:50,025 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:40:50,227 INFO SenderThread:606786 [sender.py:transition_state():308] send defer: 8 | |
| 2021-07-14 23:40:50,228 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:40:50,228 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:40:50,228 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 8 | |
| 2021-07-14 23:40:50,229 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:40:50,229 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 8 | |
| 2021-07-14 23:40:50,229 INFO SenderThread:606786 [sender.py:transition_state():308] send defer: 9 | |
| 2021-07-14 23:40:50,229 DEBUG SenderThread:606786 [sender.py:send():179] send: final | |
| 2021-07-14 23:40:50,229 DEBUG SenderThread:606786 [sender.py:send():179] send: footer | |
| 2021-07-14 23:40:50,230 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:40:50,230 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 9 | |
| 2021-07-14 23:40:50,230 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:40:50,230 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 9 | |
| 2021-07-14 23:40:50,330 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:40:50,330 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:40:50,330 INFO SenderThread:606786 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-14 23:40:50,332 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: get_summary | |
| 2021-07-14 23:40:50,332 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: sampled_history | |
| 2021-07-14 23:40:50,333 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: shutdown | |
| 2021-07-14 23:40:50,333 INFO HandlerThread:606786 [handler.py:finish():638] shutting down handler | |
| 2021-07-14 23:40:51,230 INFO WriterThread:606786 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/run-1jijl27o.wandb | |
| 2021-07-14 23:40:51,330 INFO SenderThread:606786 [sender.py:finish():945] shutting down sender | |
| 2021-07-14 23:40:51,331 INFO SenderThread:606786 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-14 23:40:51,331 INFO SenderThread:606786 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-14 23:40:51,334 INFO MainThread:606786 [internal.py:handle_exit():78] Internal process exited | |