| 2021-07-14 18:55:37,982 INFO MainThread:555087 [internal.py:wandb_internal():88] W&B internal server running at pid: 555087, started at: 2021-07-14 18:55:37.982262 | |
| 2021-07-14 18:55:37,984 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: check_version | |
| 2021-07-14 18:55:37,985 INFO WriterThread:555087 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/run-379h21h2.wandb | |
| 2021-07-14 18:55:37,986 DEBUG SenderThread:555087 [sender.py:send():179] send: header | |
| 2021-07-14 18:55:37,986 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: check_version | |
| 2021-07-14 18:55:38,025 DEBUG SenderThread:555087 [sender.py:send():179] send: run | |
| 2021-07-14 18:55:38,206 INFO SenderThread:555087 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files | |
| 2021-07-14 18:55:38,206 INFO SenderThread:555087 [sender.py:_start_run_threads():716] run started: 379h21h2 with start time 1626288937 | |
| 2021-07-14 18:55:38,206 DEBUG SenderThread:555087 [sender.py:send():179] send: summary | |
| 2021-07-14 18:55:38,206 INFO SenderThread:555087 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 18:55:38,207 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: run_start | |
| 2021-07-14 18:55:39,207 INFO Thread-8 :555087 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/wandb-summary.json | |
| 2021-07-14 18:55:39,338 DEBUG HandlerThread:555087 [meta.py:__init__():39] meta init | |
| 2021-07-14 18:55:39,338 DEBUG HandlerThread:555087 [meta.py:__init__():53] meta init done | |
| 2021-07-14 18:55:39,338 DEBUG HandlerThread:555087 [meta.py:probe():210] probe | |
| 2021-07-14 18:55:39,340 DEBUG HandlerThread:555087 [meta.py:_setup_git():200] setup git | |
| 2021-07-14 18:55:39,371 DEBUG HandlerThread:555087 [meta.py:_setup_git():207] setup git done | |
| 2021-07-14 18:55:39,371 DEBUG HandlerThread:555087 [meta.py:_save_pip():57] save pip | |
| 2021-07-14 18:55:39,372 DEBUG HandlerThread:555087 [meta.py:_save_pip():71] save pip done | |
| 2021-07-14 18:55:39,372 DEBUG HandlerThread:555087 [meta.py:probe():252] probe done | |
| 2021-07-14 18:55:39,397 DEBUG SenderThread:555087 [sender.py:send():179] send: files | |
| 2021-07-14 18:55:39,397 INFO SenderThread:555087 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now | |
| 2021-07-14 18:55:39,404 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:55:39,405 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:55:39,533 DEBUG SenderThread:555087 [sender.py:send():179] send: config | |
| 2021-07-14 18:55:39,534 DEBUG SenderThread:555087 [sender.py:send():179] send: config | |
| 2021-07-14 18:55:39,534 DEBUG SenderThread:555087 [sender.py:send():179] send: config | |
| 2021-07-14 18:55:39,858 INFO Thread-11 :555087 [upload_job.py:push():137] Uploaded file /tmp/tmpywmfy6u4wandb/302tickz-wandb-metadata.json | |
| 2021-07-14 18:55:40,207 INFO Thread-8 :555087 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/output.log | |
| 2021-07-14 18:55:40,207 INFO Thread-8 :555087 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/wandb-metadata.json | |
| 2021-07-14 18:55:40,207 INFO Thread-8 :555087 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/requirements.txt | |
| 2021-07-14 18:55:54,543 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:55:54,543 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:55:58,216 INFO Thread-8 :555087 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/output.log | |
| 2021-07-14 18:56:07,426 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 18:56:09,220 INFO Thread-8 :555087 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/config.yaml | |
| 2021-07-14 18:56:09,675 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:56:09,675 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:56:24,806 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:56:24,806 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:56:37,502 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 18:56:39,942 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:56:39,942 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:56:55,073 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:56:55,073 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:57:07,579 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 18:57:10,210 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:57:10,210 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:57:25,342 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:57:25,343 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:57:37,658 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 18:57:40,474 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:57:40,475 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:57:55,605 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:57:55,606 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:58:07,732 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 18:58:10,737 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:58:10,737 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:58:25,870 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:58:25,870 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:58:37,810 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 18:58:41,005 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:58:41,005 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:58:56,137 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:58:56,138 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:59:07,886 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 18:59:11,274 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:59:11,274 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:59:26,406 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:59:26,406 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:59:37,963 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 18:59:41,543 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:59:41,543 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 18:59:56,673 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 18:59:56,674 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:00:08,040 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:00:11,805 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:00:11,806 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:00:26,938 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:00:26,938 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:00:38,115 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:00:42,071 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:00:42,073 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:00:57,206 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:00:57,206 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:01:08,189 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:01:12,338 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:01:12,338 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:01:27,470 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:01:27,470 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:01:38,253 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:01:42,602 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:01:42,602 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:01:57,736 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:01:57,736 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:02:08,323 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:02:12,869 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:02:12,869 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:02:28,002 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:02:28,002 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:02:38,399 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:02:43,167 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:02:43,167 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:02:58,298 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:02:58,299 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:03:08,473 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:03:13,435 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:03:13,435 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:03:28,570 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:03:28,570 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:03:38,550 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:03:43,702 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:03:43,702 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:03:58,838 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:03:58,838 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:04:08,624 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:04:13,976 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:04:13,976 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:04:29,120 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:04:29,121 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:04:38,701 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:04:44,257 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:04:44,258 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:04:59,397 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:04:59,397 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:05:08,777 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:05:14,531 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:05:14,532 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:05:29,687 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:05:29,688 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:05:38,851 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:05:44,819 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:05:44,820 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:05:59,950 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:05:59,951 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:06:08,925 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:06:15,090 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:06:15,091 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:06:30,226 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:06:30,226 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:06:39,001 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:06:45,360 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:06:45,360 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:07:00,496 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:07:00,496 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:07:09,079 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:07:15,632 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:07:15,633 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:07:30,767 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:07:30,767 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:07:39,157 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:07:45,903 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:07:45,903 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:08:01,041 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:08:01,041 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:08:09,235 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:08:16,178 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:08:16,179 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:08:31,336 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:08:31,336 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:08:39,312 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:08:46,469 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:08:46,470 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:09:01,600 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:09:01,601 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:09:09,391 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:09:16,736 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:09:16,736 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:09:31,869 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:09:31,870 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:09:39,454 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:09:47,008 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:09:47,008 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:10:02,139 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:10:02,140 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:10:09,532 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:10:17,277 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:10:17,277 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:10:32,410 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:10:32,410 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:10:39,609 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:10:47,545 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:10:47,545 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:11:01,425 DEBUG SenderThread:555087 [sender.py:send():179] send: telemetry | |
| 2021-07-14 19:11:01,425 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 19:11:01,425 DEBUG SenderThread:555087 [sender.py:send():179] send: exit | |
| 2021-07-14 19:11:01,426 INFO SenderThread:555087 [sender.py:send_exit():287] handling exit code: 1 | |
| 2021-07-14 19:11:01,426 INFO SenderThread:555087 [sender.py:send_exit():295] send defer | |
| 2021-07-14 19:11:01,427 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 19:11:01,427 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 19:11:01,427 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 0 | |
| 2021-07-14 19:11:01,428 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 19:11:01,428 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 0 | |
| 2021-07-14 19:11:01,428 INFO SenderThread:555087 [sender.py:transition_state():308] send defer: 1 | |
| 2021-07-14 19:11:01,428 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 19:11:01,428 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 1 | |
| 2021-07-14 19:11:01,460 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 19:11:01,460 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 1 | |
| 2021-07-14 19:11:01,460 INFO SenderThread:555087 [sender.py:transition_state():308] send defer: 2 | |
| 2021-07-14 19:11:01,460 DEBUG SenderThread:555087 [sender.py:send():179] send: stats | |
| 2021-07-14 19:11:01,461 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 19:11:01,461 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 2 | |
| 2021-07-14 19:11:01,461 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 19:11:01,461 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 2 | |
| 2021-07-14 19:11:01,461 INFO SenderThread:555087 [sender.py:transition_state():308] send defer: 3 | |
| 2021-07-14 19:11:01,461 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 19:11:01,462 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 3 | |
| 2021-07-14 19:11:01,462 DEBUG SenderThread:555087 [sender.py:send():179] send: summary | |
| 2021-07-14 19:11:01,462 INFO SenderThread:555087 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 19:11:01,462 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 19:11:01,462 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 3 | |
| 2021-07-14 19:11:01,462 INFO SenderThread:555087 [sender.py:transition_state():308] send defer: 4 | |
| 2021-07-14 19:11:01,463 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 19:11:01,463 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 4 | |
| 2021-07-14 19:11:01,463 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 19:11:01,463 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 4 | |
| 2021-07-14 19:11:01,529 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 19:11:01,566 INFO Thread-8 :555087 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/output.log | |
| 2021-07-14 19:11:01,566 INFO Thread-8 :555087 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/wandb-summary.json | |
| 2021-07-14 19:11:01,667 INFO SenderThread:555087 [sender.py:transition_state():308] send defer: 5 | |
| 2021-07-14 19:11:01,667 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 19:11:01,668 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 19:11:01,668 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 5 | |
| 2021-07-14 19:11:01,668 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 19:11:01,668 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 5 | |
| 2021-07-14 19:11:01,668 INFO SenderThread:555087 [dir_watcher.py:finish():282] shutting down directory watcher | |
| 2021-07-14 19:11:01,769 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 19:11:02,566 INFO Thread-8 :555087 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/config.yaml | |
| 2021-07-14 19:11:02,567 INFO SenderThread:555087 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files | |
| 2021-07-14 19:11:02,567 INFO SenderThread:555087 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/requirements.txt requirements.txt | |
| 2021-07-14 19:11:02,567 INFO SenderThread:555087 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/output.log output.log | |
| 2021-07-14 19:11:02,567 INFO SenderThread:555087 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/wandb-metadata.json wandb-metadata.json | |
| 2021-07-14 19:11:02,567 INFO SenderThread:555087 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/config.yaml config.yaml | |
| 2021-07-14 19:11:02,568 INFO SenderThread:555087 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/wandb-summary.json wandb-summary.json | |
| 2021-07-14 19:11:02,568 INFO SenderThread:555087 [sender.py:transition_state():308] send defer: 6 | |
| 2021-07-14 19:11:02,569 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 19:11:02,572 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 19:11:02,572 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 6 | |
| 2021-07-14 19:11:02,575 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 19:11:02,575 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 6 | |
| 2021-07-14 19:11:02,575 INFO SenderThread:555087 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-14 19:11:02,671 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 19:11:02,671 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 19:11:02,773 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 19:11:02,774 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 19:11:02,876 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 19:11:02,876 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 19:11:02,977 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 19:11:02,978 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 19:11:03,031 INFO Thread-14 :555087 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/config.yaml | |
| 2021-07-14 19:11:03,066 INFO Thread-13 :555087 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/output.log | |
| 2021-07-14 19:11:03,074 INFO Thread-12 :555087 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/requirements.txt | |
| 2021-07-14 19:11:03,079 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 19:11:03,079 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 19:11:03,088 INFO Thread-15 :555087 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/wandb-summary.json | |
| 2021-07-14 19:11:03,181 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 19:11:03,181 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 19:11:03,283 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 19:11:03,283 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 19:11:03,288 INFO Thread-7 :555087 [sender.py:transition_state():308] send defer: 7 | |
| 2021-07-14 19:11:03,288 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 19:11:03,289 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 7 | |
| 2021-07-14 19:11:03,289 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 19:11:03,289 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 7 | |
| 2021-07-14 19:11:03,385 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 19:11:03,599 INFO SenderThread:555087 [sender.py:transition_state():308] send defer: 8 | |
| 2021-07-14 19:11:03,600 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 19:11:03,600 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 19:11:03,600 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 8 | |
| 2021-07-14 19:11:03,600 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 19:11:03,600 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 8 | |
| 2021-07-14 19:11:03,601 INFO SenderThread:555087 [sender.py:transition_state():308] send defer: 9 | |
| 2021-07-14 19:11:03,601 DEBUG SenderThread:555087 [sender.py:send():179] send: final | |
| 2021-07-14 19:11:03,601 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 19:11:03,601 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 9 | |
| 2021-07-14 19:11:03,601 DEBUG SenderThread:555087 [sender.py:send():179] send: footer | |
| 2021-07-14 19:11:03,602 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 19:11:03,602 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 9 | |
| 2021-07-14 19:11:03,702 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 19:11:03,702 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 19:11:03,702 INFO SenderThread:555087 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-14 19:11:03,704 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: get_summary | |
| 2021-07-14 19:11:03,704 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: sampled_history | |
| 2021-07-14 19:11:03,705 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: shutdown | |
| 2021-07-14 19:11:03,705 INFO HandlerThread:555087 [handler.py:finish():638] shutting down handler | |
| 2021-07-14 19:11:04,602 INFO WriterThread:555087 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/run-379h21h2.wandb | |
| 2021-07-14 19:11:04,703 INFO SenderThread:555087 [sender.py:finish():945] shutting down sender | |
| 2021-07-14 19:11:04,703 INFO SenderThread:555087 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-14 19:11:04,703 INFO SenderThread:555087 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-14 19:11:04,706 INFO MainThread:555087 [internal.py:handle_exit():78] Internal process exited | |