| 2021-07-14 21:03:52,249 INFO MainThread:576692 [internal.py:wandb_internal():88] W&B internal server running at pid: 576692, started at: 2021-07-14 21:03:52.249691 | |
| 2021-07-14 21:03:52,251 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: check_version | |
| 2021-07-14 21:03:52,252 INFO WriterThread:576692 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/run-1msvb4w4.wandb | |
| 2021-07-14 21:03:52,253 DEBUG SenderThread:576692 [sender.py:send():179] send: header | |
| 2021-07-14 21:03:52,253 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: check_version | |
| 2021-07-14 21:03:52,290 DEBUG SenderThread:576692 [sender.py:send():179] send: run | |
| 2021-07-14 21:03:52,467 INFO SenderThread:576692 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files | |
| 2021-07-14 21:03:52,467 INFO SenderThread:576692 [sender.py:_start_run_threads():716] run started: 1msvb4w4 with start time 1626296631 | |
| 2021-07-14 21:03:52,467 DEBUG SenderThread:576692 [sender.py:send():179] send: summary | |
| 2021-07-14 21:03:52,467 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: run_start | |
| 2021-07-14 21:03:52,468 INFO SenderThread:576692 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 21:03:53,469 INFO Thread-8 :576692 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json | |
| 2021-07-14 21:03:53,556 DEBUG HandlerThread:576692 [meta.py:__init__():39] meta init | |
| 2021-07-14 21:03:53,556 DEBUG HandlerThread:576692 [meta.py:__init__():53] meta init done | |
| 2021-07-14 21:03:53,556 DEBUG HandlerThread:576692 [meta.py:probe():210] probe | |
| 2021-07-14 21:03:53,557 DEBUG HandlerThread:576692 [meta.py:_setup_git():200] setup git | |
| 2021-07-14 21:03:53,583 DEBUG HandlerThread:576692 [meta.py:_setup_git():207] setup git done | |
| 2021-07-14 21:03:53,583 DEBUG HandlerThread:576692 [meta.py:_save_pip():57] save pip | |
| 2021-07-14 21:03:53,584 DEBUG HandlerThread:576692 [meta.py:_save_pip():71] save pip done | |
| 2021-07-14 21:03:53,584 DEBUG HandlerThread:576692 [meta.py:probe():252] probe done | |
| 2021-07-14 21:03:53,587 DEBUG SenderThread:576692 [sender.py:send():179] send: files | |
| 2021-07-14 21:03:53,587 INFO SenderThread:576692 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now | |
| 2021-07-14 21:03:53,595 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:03:53,595 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:03:53,722 DEBUG SenderThread:576692 [sender.py:send():179] send: config | |
| 2021-07-14 21:03:53,723 DEBUG SenderThread:576692 [sender.py:send():179] send: config | |
| 2021-07-14 21:03:53,723 DEBUG SenderThread:576692 [sender.py:send():179] send: config | |
| 2021-07-14 21:03:54,058 INFO Thread-11 :576692 [upload_job.py:push():137] Uploaded file /tmp/tmpuukxu7k9wandb/jrhx1loi-wandb-metadata.json | |
| 2021-07-14 21:03:54,469 INFO Thread-8 :576692 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/requirements.txt | |
| 2021-07-14 21:03:54,469 INFO Thread-8 :576692 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:03:54,469 INFO Thread-8 :576692 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-metadata.json | |
| 2021-07-14 21:04:08,475 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:04:08,724 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:04:08,725 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:04:14,478 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:04:21,634 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:04:23,482 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/config.yaml | |
| 2021-07-14 21:04:23,872 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:04:23,872 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:04:39,023 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:04:39,024 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:04:51,718 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:04:54,154 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:04:54,155 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:05:09,287 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:05:09,288 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:05:21,799 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:05:24,422 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:05:24,422 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:05:36,513 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:05:38,514 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:05:39,555 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:05:39,555 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:05:40,515 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:05:42,516 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:05:44,517 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:05:46,518 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:05:48,520 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:05:50,521 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:05:51,877 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:05:52,522 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:05:54,523 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:05:54,691 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:05:54,692 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:05:56,524 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:05:58,525 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:00,526 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:02,527 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:04,528 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:06,529 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:08,530 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:09,827 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:06:09,828 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:06:10,531 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:12,533 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:14,534 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:16,535 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:18,536 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:20,537 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:21,952 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:06:22,538 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:24,539 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:24,965 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:06:24,966 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:06:26,540 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:28,541 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:30,542 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:32,543 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:34,544 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:36,545 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:38,546 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:40,104 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:06:40,104 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:06:40,547 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:42,548 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:44,549 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:46,550 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:48,551 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:50,552 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:52,028 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:06:52,553 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:54,554 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:55,241 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:06:55,242 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:06:56,555 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:06:58,556 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:00,557 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:02,558 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:04,559 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:06,559 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:08,560 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:10,382 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:07:10,382 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:07:10,561 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:12,562 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:14,563 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:16,564 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:18,565 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:20,566 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:22,105 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:07:22,567 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:24,568 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:25,516 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:07:25,516 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:07:26,569 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:28,570 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:30,571 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:32,572 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:34,572 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:36,573 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:38,574 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:40,575 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:40,655 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:07:40,656 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:07:42,576 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:44,577 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:46,578 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:48,579 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:50,580 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:52,180 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:07:52,581 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:54,582 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:55,795 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:07:55,795 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:07:56,582 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:07:58,583 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:00,584 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:02,585 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:04,586 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:06,587 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:08,588 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:10,934 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:08:10,935 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:08:11,590 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:13,591 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:22,255 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:08:26,077 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:08:26,077 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:08:35,048 DEBUG SenderThread:576692 [sender.py:send():179] send: history | |
| 2021-07-14 21:08:35,049 DEBUG SenderThread:576692 [sender.py:send():179] send: summary | |
| 2021-07-14 21:08:35,051 INFO SenderThread:576692 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 21:08:35,601 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json | |
| 2021-07-14 21:08:36,602 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:37,602 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:39,603 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:41,212 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:08:41,212 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:08:41,604 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:43,605 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:45,606 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:47,607 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:49,609 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:51,609 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:52,330 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:08:53,611 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:55,611 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:56,345 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:08:56,345 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:08:57,612 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:08:59,613 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:01,614 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:03,615 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:05,616 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:07,618 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:09,619 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:11,477 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:09:11,477 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:09:11,621 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:13,622 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:15,623 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:17,624 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:19,625 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:21,626 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:22,404 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:09:23,627 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:25,628 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:26,623 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:09:26,624 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:09:27,629 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:29,631 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:31,632 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:33,633 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:35,634 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:37,635 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:39,636 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:41,637 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:41,755 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:09:41,756 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:09:43,638 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:45,640 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:47,641 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:49,642 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:51,643 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:52,477 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:09:53,644 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:55,645 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:56,888 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:09:56,888 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:09:57,648 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:09:59,649 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:01,650 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:03,651 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:05,652 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:07,653 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:09,655 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:11,656 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:12,020 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:10:12,021 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:10:13,657 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:15,658 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:17,659 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:19,660 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:21,661 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:22,552 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:10:23,662 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:25,663 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:27,151 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:10:27,152 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:10:27,664 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:29,665 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:31,666 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:33,667 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:35,668 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:37,669 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:39,671 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:41,672 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:42,283 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:10:42,284 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:10:43,673 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:45,674 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:47,675 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:49,676 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:51,677 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:52,629 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:10:53,678 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:55,679 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:57,423 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:10:57,423 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:10:57,680 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:10:59,681 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:11:01,682 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:11:03,683 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:11:05,684 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:11:07,685 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:11:09,686 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:11:11,687 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:11:12,554 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:11:12,555 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:11:13,688 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:11:22,703 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:11:27,687 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:11:27,688 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:11:35,660 DEBUG SenderThread:576692 [sender.py:send():179] send: history | |
| 2021-07-14 21:11:35,661 DEBUG SenderThread:576692 [sender.py:send():179] send: summary | |
| 2021-07-14 21:11:35,661 INFO SenderThread:576692 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 21:11:35,697 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json | |
| 2021-07-14 21:11:36,698 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:11:37,698 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:11:42,819 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:11:42,819 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:11:52,784 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:11:55,707 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:11:57,708 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:11:57,959 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:11:57,959 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:12:01,273 DEBUG SenderThread:576692 [sender.py:send():179] send: history | |
| 2021-07-14 21:12:01,273 DEBUG SenderThread:576692 [sender.py:send():179] send: summary | |
| 2021-07-14 21:12:01,274 INFO SenderThread:576692 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 21:12:01,710 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:01,710 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json | |
| 2021-07-14 21:12:03,710 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:05,711 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:07,712 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:09,713 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:11,714 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:13,090 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:12:13,091 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:12:13,715 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:15,716 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:17,717 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:20,718 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:22,719 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:22,858 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:12:24,720 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:26,722 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:28,222 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:12:28,222 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:12:28,722 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:30,723 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:32,724 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:34,726 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:36,727 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:38,728 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:40,729 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:42,729 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:43,353 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:12:43,353 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:12:44,730 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:46,732 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:48,733 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:50,734 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:52,735 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:52,932 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:12:54,736 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:56,737 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:12:58,487 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:12:58,488 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:12:58,738 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:00,739 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:02,740 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:04,741 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:06,742 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:08,743 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:10,744 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:12,745 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:13,623 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:13:13,624 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:13:14,746 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:16,747 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:18,748 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:20,749 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:22,750 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:23,009 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:13:24,751 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:26,752 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:28,753 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:28,757 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:13:28,758 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:13:30,754 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:32,755 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:34,756 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:36,757 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:38,758 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:40,759 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:42,761 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:43,898 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:13:43,898 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:13:44,761 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:46,762 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:48,763 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:50,765 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:52,765 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:53,082 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:13:54,766 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:56,767 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:58,768 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:13:59,033 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:13:59,033 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:14:00,769 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:02,770 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:04,771 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:06,772 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:08,773 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:10,774 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:12,775 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:14,168 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:14:14,169 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:14:14,776 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:16,777 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:18,778 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:20,779 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:22,780 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:23,155 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:14:24,781 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:26,783 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:28,784 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:29,304 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:14:29,304 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:14:30,785 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:32,785 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:34,786 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:36,788 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:38,789 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:14:44,435 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:14:44,436 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:14:53,228 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:14:59,567 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:14:59,567 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:15:01,849 DEBUG SenderThread:576692 [sender.py:send():179] send: history | |
| 2021-07-14 21:15:01,850 DEBUG SenderThread:576692 [sender.py:send():179] send: summary | |
| 2021-07-14 21:15:01,850 INFO SenderThread:576692 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 21:15:02,799 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:02,799 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json | |
| 2021-07-14 21:15:03,799 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:04,800 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:06,801 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:08,801 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:10,802 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:12,803 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:14,697 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:15:14,697 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:15:14,804 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:16,805 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:18,806 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:20,807 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:22,808 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:23,303 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:15:24,809 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:26,810 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:28,811 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:29,829 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:15:29,829 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:15:30,812 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:32,813 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:34,814 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:36,815 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:38,816 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:40,817 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:42,818 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:44,819 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:44,966 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:15:44,967 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:15:46,821 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:48,822 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:50,823 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:52,824 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:53,375 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:15:54,825 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:56,826 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:15:58,826 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:00,097 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:16:00,097 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:16:00,827 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:02,828 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:04,829 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:06,830 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:08,831 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:10,832 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:12,833 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:14,834 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:15,228 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:16:15,229 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:16:16,835 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:18,836 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:20,837 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:22,838 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:23,451 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:16:24,839 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:26,840 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:28,841 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:30,361 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:16:30,362 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:16:30,842 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:32,843 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:34,844 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:36,845 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:38,846 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:40,847 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:42,848 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:44,848 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:45,493 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:16:45,493 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:16:46,849 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:48,850 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:51,852 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:53,525 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:16:53,852 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:55,853 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:57,854 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:16:59,855 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:00,625 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:17:00,625 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:17:01,856 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:03,857 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:05,858 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:07,859 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:09,860 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:11,861 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:13,862 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:15,765 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:17:15,765 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:17:15,863 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:17,864 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:19,865 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:21,866 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:23,601 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:17:23,867 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:25,868 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:27,869 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:29,870 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:30,898 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:17:30,898 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:17:31,871 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:33,872 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:35,873 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:37,874 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:39,875 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:17:46,031 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:17:46,032 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:17:53,675 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:18:01,164 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:18:01,164 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:18:02,458 DEBUG SenderThread:576692 [sender.py:send():179] send: history | |
| 2021-07-14 21:18:02,458 DEBUG SenderThread:576692 [sender.py:send():179] send: summary | |
| 2021-07-14 21:18:02,459 INFO SenderThread:576692 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 21:18:02,885 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json | |
| 2021-07-14 21:18:03,886 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:05,887 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:07,888 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:09,729 DEBUG SenderThread:576692 [sender.py:send():179] send: history | |
| 2021-07-14 21:18:09,729 DEBUG SenderThread:576692 [sender.py:send():179] send: summary | |
| 2021-07-14 21:18:09,730 INFO SenderThread:576692 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 21:18:09,889 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json | |
| 2021-07-14 21:18:11,890 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:13,891 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:15,892 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:16,293 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:18:16,294 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:18:17,893 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:19,894 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:21,895 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:23,751 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:18:23,896 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:25,897 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:27,898 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:29,899 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:31,425 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:18:31,426 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:18:31,900 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:33,901 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:35,902 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:37,903 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:39,904 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:41,905 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:43,906 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:45,907 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:46,561 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:18:46,561 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:18:47,908 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:49,909 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:51,910 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:53,825 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:18:53,911 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:55,912 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:57,913 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:18:59,914 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:01,703 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:19:01,703 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:19:01,915 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:03,916 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:05,917 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:07,918 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:09,919 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:11,920 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:13,921 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:15,922 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:16,843 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:19:16,844 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:19:17,923 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:19,924 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:21,925 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:23,899 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:19:23,926 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:25,926 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:27,927 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:29,928 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:31,929 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:31,979 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:19:31,980 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:19:33,930 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:35,931 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:37,932 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:39,933 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:41,934 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:43,935 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:45,936 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:47,118 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:19:47,118 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:19:47,937 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:49,938 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:51,939 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:53,940 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:53,975 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:19:55,941 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:57,942 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:19:59,943 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:01,944 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:02,431 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:20:02,431 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:20:03,945 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:05,946 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:07,948 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:09,949 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:11,950 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:13,950 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:15,951 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:17,562 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:20:17,562 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:20:17,952 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:19,953 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:21,954 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:23,955 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:24,053 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:20:25,956 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:27,957 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:29,958 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:31,959 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:32,928 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:20:32,929 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:20:33,960 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:35,961 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:37,962 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:39,963 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:41,964 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:43,965 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:45,966 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:47,967 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:20:48,061 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:20:48,061 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:20:54,126 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:21:03,199 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:21:03,200 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:21:11,977 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:21:12,763 DEBUG SenderThread:576692 [sender.py:send():179] send: history | |
| 2021-07-14 21:21:12,763 DEBUG SenderThread:576692 [sender.py:send():179] send: summary | |
| 2021-07-14 21:21:12,764 INFO SenderThread:576692 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 21:21:12,977 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json | |
| 2021-07-14 21:21:13,978 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:21:15,979 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:21:18,334 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:21:18,334 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:21:24,199 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:21:33,465 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:21:33,466 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:21:48,598 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:21:48,598 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:21:54,276 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:22:03,999 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:22:04,937 DEBUG SenderThread:576692 [sender.py:send():179] send: telemetry | |
| 2021-07-14 21:22:04,938 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:22:04,938 DEBUG SenderThread:576692 [sender.py:send():179] send: exit | |
| 2021-07-14 21:22:04,938 INFO SenderThread:576692 [sender.py:send_exit():287] handling exit code: 1 | |
| 2021-07-14 21:22:04,940 INFO SenderThread:576692 [sender.py:send_exit():295] send defer | |
| 2021-07-14 21:22:04,940 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:22:04,941 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:22:04,941 INFO HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 0 | |
| 2021-07-14 21:22:04,941 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:22:04,941 INFO SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 0 | |
| 2021-07-14 21:22:04,941 INFO SenderThread:576692 [sender.py:transition_state():308] send defer: 1 | |
| 2021-07-14 21:22:04,942 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:22:04,942 INFO HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 1 | |
| 2021-07-14 21:22:04,999 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:22:05,004 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:22:05,004 INFO SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 1 | |
| 2021-07-14 21:22:05,004 INFO SenderThread:576692 [sender.py:transition_state():308] send defer: 2 | |
| 2021-07-14 21:22:05,004 DEBUG SenderThread:576692 [sender.py:send():179] send: stats | |
| 2021-07-14 21:22:05,005 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:22:05,005 INFO HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 2 | |
| 2021-07-14 21:22:05,005 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:22:05,005 INFO SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 2 | |
| 2021-07-14 21:22:05,005 INFO SenderThread:576692 [sender.py:transition_state():308] send defer: 3 | |
| 2021-07-14 21:22:05,006 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:22:05,006 INFO HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 3 | |
| 2021-07-14 21:22:05,006 DEBUG SenderThread:576692 [sender.py:send():179] send: summary | |
| 2021-07-14 21:22:05,007 INFO SenderThread:576692 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 21:22:05,007 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:22:05,007 INFO SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 3 | |
| 2021-07-14 21:22:05,007 INFO SenderThread:576692 [sender.py:transition_state():308] send defer: 4 | |
| 2021-07-14 21:22:05,007 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:22:05,007 INFO HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 4 | |
| 2021-07-14 21:22:05,007 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:22:05,008 INFO SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 4 | |
| 2021-07-14 21:22:05,043 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:22:05,215 INFO SenderThread:576692 [sender.py:transition_state():308] send defer: 5 | |
| 2021-07-14 21:22:05,215 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:22:05,216 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:22:05,216 INFO HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 5 | |
| 2021-07-14 21:22:05,216 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:22:05,216 INFO SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 5 | |
| 2021-07-14 21:22:05,216 INFO SenderThread:576692 [dir_watcher.py:finish():282] shutting down directory watcher | |
| 2021-07-14 21:22:05,317 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:22:06,000 INFO Thread-8 :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/config.yaml | |
| 2021-07-14 21:22:06,000 INFO SenderThread:576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json | |
| 2021-07-14 21:22:06,001 INFO SenderThread:576692 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files | |
| 2021-07-14 21:22:06,001 INFO SenderThread:576692 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/requirements.txt requirements.txt | |
| 2021-07-14 21:22:06,001 INFO SenderThread:576692 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log output.log | |
| 2021-07-14 21:22:06,001 INFO SenderThread:576692 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-metadata.json wandb-metadata.json | |
| 2021-07-14 21:22:06,001 INFO SenderThread:576692 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/config.yaml config.yaml | |
| 2021-07-14 21:22:06,001 INFO SenderThread:576692 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json wandb-summary.json | |
| 2021-07-14 21:22:06,002 INFO SenderThread:576692 [sender.py:transition_state():308] send defer: 6 | |
| 2021-07-14 21:22:06,003 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:22:06,003 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:22:06,003 INFO HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 6 | |
| 2021-07-14 21:22:06,006 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:22:06,006 INFO SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 6 | |
| 2021-07-14 21:22:06,007 INFO SenderThread:576692 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-14 21:22:06,105 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:22:06,105 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:22:06,207 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:22:06,207 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:22:06,309 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:22:06,309 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:22:06,411 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:22:06,411 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:22:06,455 INFO Thread-12 :576692 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/requirements.txt | |
| 2021-07-14 21:22:06,463 INFO Thread-15 :576692 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json | |
| 2021-07-14 21:22:06,495 INFO Thread-13 :576692 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log | |
| 2021-07-14 21:22:06,512 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:22:06,513 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:22:06,514 INFO Thread-14 :576692 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/config.yaml | |
| 2021-07-14 21:22:06,615 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:22:06,615 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:22:06,715 INFO Thread-7 :576692 [sender.py:transition_state():308] send defer: 7 | |
| 2021-07-14 21:22:06,715 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:22:06,716 INFO HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 7 | |
| 2021-07-14 21:22:06,716 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:22:06,716 INFO SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 7 | |
| 2021-07-14 21:22:06,718 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:22:06,999 INFO SenderThread:576692 [sender.py:transition_state():308] send defer: 8 | |
| 2021-07-14 21:22:06,999 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:22:07,000 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:22:07,000 INFO HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 8 | |
| 2021-07-14 21:22:07,000 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:22:07,000 INFO SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 8 | |
| 2021-07-14 21:22:07,000 INFO SenderThread:576692 [sender.py:transition_state():308] send defer: 9 | |
| 2021-07-14 21:22:07,001 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:22:07,001 DEBUG SenderThread:576692 [sender.py:send():179] send: final | |
| 2021-07-14 21:22:07,001 INFO HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 9 | |
| 2021-07-14 21:22:07,001 DEBUG SenderThread:576692 [sender.py:send():179] send: footer | |
| 2021-07-14 21:22:07,002 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:22:07,002 INFO SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 9 | |
| 2021-07-14 21:22:07,101 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:22:07,101 DEBUG SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:22:07,101 INFO SenderThread:576692 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-14 21:22:07,103 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: get_summary | |
| 2021-07-14 21:22:07,104 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: sampled_history | |
| 2021-07-14 21:22:07,105 DEBUG HandlerThread:576692 [handler.py:handle_request():124] handle_request: shutdown | |
| 2021-07-14 21:22:07,105 INFO HandlerThread:576692 [handler.py:finish():638] shutting down handler | |
| 2021-07-14 21:22:08,002 INFO WriterThread:576692 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/run-1msvb4w4.wandb | |
| 2021-07-14 21:22:08,102 INFO SenderThread:576692 [sender.py:finish():945] shutting down sender | |
| 2021-07-14 21:22:08,102 INFO SenderThread:576692 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-14 21:22:08,102 INFO SenderThread:576692 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-14 21:22:08,105 INFO MainThread:576692 [internal.py:handle_exit():78] Internal process exited | |