wandb/run-20210714_210351-1msvb4w4/logs/debug-internal.log

2021-07-14 21:03:52,249 INFO    MainThread:576692 [internal.py:wandb_internal():88] W&B internal server running at pid: 576692, started at: 2021-07-14 21:03:52.249691
2021-07-14 21:03:52,251 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 21:03:52,252 INFO    WriterThread:576692 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/run-1msvb4w4.wandb
2021-07-14 21:03:52,253 DEBUG   SenderThread:576692 [sender.py:send():179] send: header
2021-07-14 21:03:52,253 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: check_version
2021-07-14 21:03:52,290 DEBUG   SenderThread:576692 [sender.py:send():179] send: run
2021-07-14 21:03:52,467 INFO    SenderThread:576692 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files
2021-07-14 21:03:52,467 INFO    SenderThread:576692 [sender.py:_start_run_threads():716] run started: 1msvb4w4 with start time 1626296631
2021-07-14 21:03:52,467 DEBUG   SenderThread:576692 [sender.py:send():179] send: summary
2021-07-14 21:03:52,467 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 21:03:52,468 INFO    SenderThread:576692 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:03:53,469 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json
2021-07-14 21:03:53,556 DEBUG   HandlerThread:576692 [meta.py:__init__():39] meta init
2021-07-14 21:03:53,556 DEBUG   HandlerThread:576692 [meta.py:__init__():53] meta init done
2021-07-14 21:03:53,556 DEBUG   HandlerThread:576692 [meta.py:probe():210] probe
2021-07-14 21:03:53,557 DEBUG   HandlerThread:576692 [meta.py:_setup_git():200] setup git
2021-07-14 21:03:53,583 DEBUG   HandlerThread:576692 [meta.py:_setup_git():207] setup git done
2021-07-14 21:03:53,583 DEBUG   HandlerThread:576692 [meta.py:_save_pip():57] save pip
2021-07-14 21:03:53,584 DEBUG   HandlerThread:576692 [meta.py:_save_pip():71] save pip done
2021-07-14 21:03:53,584 DEBUG   HandlerThread:576692 [meta.py:probe():252] probe done
2021-07-14 21:03:53,587 DEBUG   SenderThread:576692 [sender.py:send():179] send: files
2021-07-14 21:03:53,587 INFO    SenderThread:576692 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 21:03:53,595 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:03:53,595 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:03:53,722 DEBUG   SenderThread:576692 [sender.py:send():179] send: config
2021-07-14 21:03:53,723 DEBUG   SenderThread:576692 [sender.py:send():179] send: config
2021-07-14 21:03:53,723 DEBUG   SenderThread:576692 [sender.py:send():179] send: config
2021-07-14 21:03:54,058 INFO    Thread-11 :576692 [upload_job.py:push():137] Uploaded file /tmp/tmpuukxu7k9wandb/jrhx1loi-wandb-metadata.json
2021-07-14 21:03:54,469 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/requirements.txt
2021-07-14 21:03:54,469 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:03:54,469 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-metadata.json
2021-07-14 21:04:08,475 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:04:08,724 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:04:08,725 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:04:14,478 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:04:21,634 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:04:23,482 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/config.yaml
2021-07-14 21:04:23,872 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:04:23,872 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:04:39,023 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:04:39,024 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:04:51,718 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:04:54,154 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:04:54,155 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:05:09,287 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:05:09,288 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:05:21,799 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:05:24,422 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:05:24,422 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:05:36,513 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:05:38,514 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:05:39,555 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:05:39,555 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:05:40,515 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:05:42,516 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:05:44,517 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:05:46,518 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:05:48,520 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:05:50,521 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:05:51,877 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:05:52,522 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:05:54,523 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:05:54,691 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:05:54,692 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:05:56,524 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:05:58,525 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:00,526 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:02,527 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:04,528 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:06,529 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:08,530 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:09,827 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:06:09,828 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:06:10,531 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:12,533 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:14,534 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:16,535 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:18,536 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:20,537 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:21,952 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:06:22,538 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:24,539 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:24,965 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:06:24,966 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:06:26,540 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:28,541 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:30,542 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:32,543 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:34,544 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:36,545 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:38,546 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:40,104 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:06:40,104 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:06:40,547 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:42,548 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:44,549 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:46,550 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:48,551 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:50,552 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:52,028 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:06:52,553 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:54,554 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:55,241 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:06:55,242 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:06:56,555 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:06:58,556 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:00,557 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:02,558 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:04,559 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:06,559 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:08,560 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:10,382 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:07:10,382 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:07:10,561 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:12,562 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:14,563 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:16,564 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:18,565 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:20,566 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:22,105 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:07:22,567 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:24,568 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:25,516 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:07:25,516 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:07:26,569 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:28,570 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:30,571 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:32,572 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:34,572 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:36,573 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:38,574 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:40,575 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:40,655 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:07:40,656 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:07:42,576 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:44,577 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:46,578 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:48,579 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:50,580 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:52,180 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:07:52,581 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:54,582 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:55,795 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:07:55,795 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:07:56,582 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:07:58,583 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:00,584 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:02,585 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:04,586 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:06,587 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:08,588 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:10,934 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:08:10,935 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:08:11,590 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:13,591 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:22,255 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:08:26,077 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:08:26,077 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:08:35,048 DEBUG   SenderThread:576692 [sender.py:send():179] send: history
2021-07-14 21:08:35,049 DEBUG   SenderThread:576692 [sender.py:send():179] send: summary
2021-07-14 21:08:35,051 INFO    SenderThread:576692 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:08:35,601 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json
2021-07-14 21:08:36,602 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:37,602 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:39,603 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:41,212 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:08:41,212 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:08:41,604 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:43,605 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:45,606 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:47,607 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:49,609 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:51,609 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:52,330 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:08:53,611 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:55,611 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:56,345 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:08:56,345 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:08:57,612 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:08:59,613 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:01,614 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:03,615 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:05,616 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:07,618 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:09,619 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:11,477 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:09:11,477 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:09:11,621 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:13,622 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:15,623 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:17,624 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:19,625 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:21,626 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:22,404 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:09:23,627 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:25,628 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:26,623 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:09:26,624 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:09:27,629 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:29,631 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:31,632 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:33,633 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:35,634 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:37,635 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:39,636 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:41,637 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:41,755 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:09:41,756 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:09:43,638 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:45,640 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:47,641 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:49,642 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:51,643 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:52,477 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:09:53,644 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:55,645 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:56,888 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:09:56,888 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:09:57,648 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:09:59,649 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:01,650 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:03,651 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:05,652 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:07,653 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:09,655 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:11,656 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:12,020 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:10:12,021 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:10:13,657 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:15,658 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:17,659 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:19,660 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:21,661 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:22,552 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:10:23,662 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:25,663 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:27,151 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:10:27,152 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:10:27,664 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:29,665 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:31,666 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:33,667 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:35,668 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:37,669 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:39,671 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:41,672 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:42,283 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:10:42,284 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:10:43,673 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:45,674 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:47,675 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:49,676 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:51,677 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:52,629 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:10:53,678 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:55,679 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:57,423 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:10:57,423 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:10:57,680 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:10:59,681 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:11:01,682 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:11:03,683 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:11:05,684 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:11:07,685 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:11:09,686 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:11:11,687 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:11:12,554 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:11:12,555 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:11:13,688 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:11:22,703 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:11:27,687 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:11:27,688 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:11:35,660 DEBUG   SenderThread:576692 [sender.py:send():179] send: history
2021-07-14 21:11:35,661 DEBUG   SenderThread:576692 [sender.py:send():179] send: summary
2021-07-14 21:11:35,661 INFO    SenderThread:576692 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:11:35,697 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json
2021-07-14 21:11:36,698 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:11:37,698 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:11:42,819 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:11:42,819 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:11:52,784 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:11:55,707 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:11:57,708 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:11:57,959 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:11:57,959 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:12:01,273 DEBUG   SenderThread:576692 [sender.py:send():179] send: history
2021-07-14 21:12:01,273 DEBUG   SenderThread:576692 [sender.py:send():179] send: summary
2021-07-14 21:12:01,274 INFO    SenderThread:576692 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:12:01,710 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:01,710 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json
2021-07-14 21:12:03,710 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:05,711 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:07,712 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:09,713 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:11,714 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:13,090 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:12:13,091 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:12:13,715 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:15,716 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:17,717 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:20,718 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:22,719 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:22,858 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:12:24,720 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:26,722 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:28,222 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:12:28,222 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:12:28,722 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:30,723 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:32,724 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:34,726 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:36,727 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:38,728 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:40,729 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:42,729 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:43,353 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:12:43,353 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:12:44,730 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:46,732 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:48,733 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:50,734 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:52,735 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:52,932 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:12:54,736 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:56,737 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:12:58,487 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:12:58,488 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:12:58,738 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:00,739 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:02,740 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:04,741 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:06,742 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:08,743 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:10,744 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:12,745 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:13,623 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:13:13,624 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:13:14,746 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:16,747 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:18,748 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:20,749 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:22,750 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:23,009 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:13:24,751 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:26,752 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:28,753 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:28,757 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:13:28,758 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:13:30,754 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:32,755 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:34,756 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:36,757 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:38,758 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:40,759 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:42,761 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:43,898 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:13:43,898 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:13:44,761 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:46,762 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:48,763 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:50,765 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:52,765 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:53,082 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:13:54,766 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:56,767 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:58,768 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:13:59,033 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:13:59,033 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:14:00,769 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:02,770 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:04,771 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:06,772 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:08,773 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:10,774 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:12,775 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:14,168 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:14:14,169 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:14:14,776 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:16,777 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:18,778 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:20,779 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:22,780 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:23,155 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:14:24,781 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:26,783 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:28,784 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:29,304 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:14:29,304 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:14:30,785 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:32,785 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:34,786 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:36,788 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:38,789 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:14:44,435 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:14:44,436 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:14:53,228 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:14:59,567 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:14:59,567 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:15:01,849 DEBUG   SenderThread:576692 [sender.py:send():179] send: history
2021-07-14 21:15:01,850 DEBUG   SenderThread:576692 [sender.py:send():179] send: summary
2021-07-14 21:15:01,850 INFO    SenderThread:576692 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:15:02,799 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:02,799 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json
2021-07-14 21:15:03,799 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:04,800 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:06,801 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:08,801 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:10,802 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:12,803 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:14,697 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:15:14,697 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:15:14,804 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:16,805 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:18,806 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:20,807 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:22,808 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:23,303 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:15:24,809 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:26,810 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:28,811 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:29,829 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:15:29,829 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:15:30,812 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:32,813 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:34,814 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:36,815 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:38,816 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:40,817 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:42,818 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:44,819 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:44,966 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:15:44,967 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:15:46,821 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:48,822 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:50,823 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:52,824 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:53,375 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:15:54,825 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:56,826 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:15:58,826 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:00,097 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:16:00,097 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:16:00,827 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:02,828 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:04,829 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:06,830 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:08,831 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:10,832 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:12,833 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:14,834 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:15,228 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:16:15,229 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:16:16,835 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:18,836 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:20,837 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:22,838 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:23,451 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:16:24,839 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:26,840 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:28,841 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:30,361 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:16:30,362 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:16:30,842 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:32,843 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:34,844 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:36,845 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:38,846 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:40,847 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:42,848 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:44,848 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:45,493 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:16:45,493 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:16:46,849 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:48,850 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:51,852 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:53,525 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:16:53,852 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:55,853 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:57,854 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:16:59,855 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:00,625 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:17:00,625 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:17:01,856 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:03,857 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:05,858 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:07,859 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:09,860 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:11,861 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:13,862 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:15,765 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:17:15,765 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:17:15,863 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:17,864 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:19,865 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:21,866 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:23,601 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:17:23,867 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:25,868 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:27,869 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:29,870 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:30,898 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:17:30,898 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:17:31,871 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:33,872 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:35,873 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:37,874 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:39,875 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:17:46,031 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:17:46,032 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:17:53,675 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:18:01,164 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:18:01,164 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:18:02,458 DEBUG   SenderThread:576692 [sender.py:send():179] send: history
2021-07-14 21:18:02,458 DEBUG   SenderThread:576692 [sender.py:send():179] send: summary
2021-07-14 21:18:02,459 INFO    SenderThread:576692 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:18:02,885 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json
2021-07-14 21:18:03,886 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:05,887 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:07,888 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:09,729 DEBUG   SenderThread:576692 [sender.py:send():179] send: history
2021-07-14 21:18:09,729 DEBUG   SenderThread:576692 [sender.py:send():179] send: summary
2021-07-14 21:18:09,730 INFO    SenderThread:576692 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:18:09,889 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json
2021-07-14 21:18:11,890 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:13,891 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:15,892 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:16,293 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:18:16,294 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:18:17,893 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:19,894 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:21,895 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:23,751 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:18:23,896 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:25,897 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:27,898 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:29,899 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:31,425 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:18:31,426 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:18:31,900 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:33,901 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:35,902 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:37,903 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:39,904 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:41,905 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:43,906 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:45,907 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:46,561 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:18:46,561 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:18:47,908 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:49,909 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:51,910 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:53,825 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:18:53,911 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:55,912 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:57,913 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:18:59,914 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:01,703 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:19:01,703 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:19:01,915 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:03,916 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:05,917 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:07,918 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:09,919 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:11,920 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:13,921 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:15,922 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:16,843 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:19:16,844 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:19:17,923 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:19,924 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:21,925 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:23,899 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:19:23,926 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:25,926 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:27,927 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:29,928 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:31,929 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:31,979 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:19:31,980 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:19:33,930 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:35,931 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:37,932 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:39,933 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:41,934 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:43,935 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:45,936 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:47,118 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:19:47,118 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:19:47,937 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:49,938 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:51,939 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:53,940 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:53,975 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:19:55,941 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:57,942 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:19:59,943 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:01,944 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:02,431 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:20:02,431 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:20:03,945 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:05,946 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:07,948 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:09,949 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:11,950 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:13,950 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:15,951 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:17,562 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:20:17,562 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:20:17,952 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:19,953 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:21,954 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:23,955 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:24,053 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:20:25,956 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:27,957 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:29,958 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:31,959 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:32,928 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:20:32,929 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:20:33,960 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:35,961 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:37,962 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:39,963 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:41,964 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:43,965 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:45,966 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:47,967 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:20:48,061 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:20:48,061 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:20:54,126 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:21:03,199 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:21:03,200 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:21:11,977 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:21:12,763 DEBUG   SenderThread:576692 [sender.py:send():179] send: history
2021-07-14 21:21:12,763 DEBUG   SenderThread:576692 [sender.py:send():179] send: summary
2021-07-14 21:21:12,764 INFO    SenderThread:576692 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:21:12,977 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json
2021-07-14 21:21:13,978 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:21:15,979 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:21:18,334 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:21:18,334 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:21:24,199 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:21:33,465 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:21:33,466 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:21:48,598 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:21:48,598 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:21:54,276 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:22:03,999 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:22:04,937 DEBUG   SenderThread:576692 [sender.py:send():179] send: telemetry
2021-07-14 21:22:04,938 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:22:04,938 DEBUG   SenderThread:576692 [sender.py:send():179] send: exit
2021-07-14 21:22:04,938 INFO    SenderThread:576692 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 21:22:04,940 INFO    SenderThread:576692 [sender.py:send_exit():295] send defer
2021-07-14 21:22:04,940 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:22:04,941 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:22:04,941 INFO    HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 21:22:04,941 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: defer
2021-07-14 21:22:04,941 INFO    SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 21:22:04,941 INFO    SenderThread:576692 [sender.py:transition_state():308] send defer: 1
2021-07-14 21:22:04,942 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:22:04,942 INFO    HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 21:22:04,999 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:22:05,004 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: defer
2021-07-14 21:22:05,004 INFO    SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 21:22:05,004 INFO    SenderThread:576692 [sender.py:transition_state():308] send defer: 2
2021-07-14 21:22:05,004 DEBUG   SenderThread:576692 [sender.py:send():179] send: stats
2021-07-14 21:22:05,005 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:22:05,005 INFO    HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 21:22:05,005 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: defer
2021-07-14 21:22:05,005 INFO    SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 21:22:05,005 INFO    SenderThread:576692 [sender.py:transition_state():308] send defer: 3
2021-07-14 21:22:05,006 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:22:05,006 INFO    HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 21:22:05,006 DEBUG   SenderThread:576692 [sender.py:send():179] send: summary
2021-07-14 21:22:05,007 INFO    SenderThread:576692 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:22:05,007 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: defer
2021-07-14 21:22:05,007 INFO    SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 21:22:05,007 INFO    SenderThread:576692 [sender.py:transition_state():308] send defer: 4
2021-07-14 21:22:05,007 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:22:05,007 INFO    HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 21:22:05,007 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: defer
2021-07-14 21:22:05,008 INFO    SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 21:22:05,043 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:22:05,215 INFO    SenderThread:576692 [sender.py:transition_state():308] send defer: 5
2021-07-14 21:22:05,215 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:22:05,216 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:22:05,216 INFO    HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 21:22:05,216 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: defer
2021-07-14 21:22:05,216 INFO    SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 21:22:05,216 INFO    SenderThread:576692 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 21:22:05,317 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:22:06,000 INFO    Thread-8  :576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/config.yaml
2021-07-14 21:22:06,000 INFO    SenderThread:576692 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json
2021-07-14 21:22:06,001 INFO    SenderThread:576692 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files
2021-07-14 21:22:06,001 INFO    SenderThread:576692 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/requirements.txt requirements.txt
2021-07-14 21:22:06,001 INFO    SenderThread:576692 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log output.log
2021-07-14 21:22:06,001 INFO    SenderThread:576692 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-metadata.json wandb-metadata.json
2021-07-14 21:22:06,001 INFO    SenderThread:576692 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/config.yaml config.yaml
2021-07-14 21:22:06,001 INFO    SenderThread:576692 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json wandb-summary.json
2021-07-14 21:22:06,002 INFO    SenderThread:576692 [sender.py:transition_state():308] send defer: 6
2021-07-14 21:22:06,003 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:22:06,003 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:22:06,003 INFO    HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 21:22:06,006 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: defer
2021-07-14 21:22:06,006 INFO    SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 21:22:06,007 INFO    SenderThread:576692 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 21:22:06,105 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:22:06,105 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:22:06,207 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:22:06,207 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:22:06,309 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:22:06,309 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:22:06,411 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:22:06,411 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:22:06,455 INFO    Thread-12 :576692 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/requirements.txt
2021-07-14 21:22:06,463 INFO    Thread-15 :576692 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/wandb-summary.json
2021-07-14 21:22:06,495 INFO    Thread-13 :576692 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/output.log
2021-07-14 21:22:06,512 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:22:06,513 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:22:06,514 INFO    Thread-14 :576692 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/files/config.yaml
2021-07-14 21:22:06,615 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:22:06,615 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:22:06,715 INFO    Thread-7  :576692 [sender.py:transition_state():308] send defer: 7
2021-07-14 21:22:06,715 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:22:06,716 INFO    HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 21:22:06,716 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: defer
2021-07-14 21:22:06,716 INFO    SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 21:22:06,718 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:22:06,999 INFO    SenderThread:576692 [sender.py:transition_state():308] send defer: 8
2021-07-14 21:22:06,999 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:22:07,000 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:22:07,000 INFO    HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 21:22:07,000 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: defer
2021-07-14 21:22:07,000 INFO    SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 21:22:07,000 INFO    SenderThread:576692 [sender.py:transition_state():308] send defer: 9
2021-07-14 21:22:07,001 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:22:07,001 DEBUG   SenderThread:576692 [sender.py:send():179] send: final
2021-07-14 21:22:07,001 INFO    HandlerThread:576692 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 21:22:07,001 DEBUG   SenderThread:576692 [sender.py:send():179] send: footer
2021-07-14 21:22:07,002 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: defer
2021-07-14 21:22:07,002 INFO    SenderThread:576692 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 21:22:07,101 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:22:07,101 DEBUG   SenderThread:576692 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:22:07,101 INFO    SenderThread:576692 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 21:22:07,103 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 21:22:07,104 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 21:22:07,105 DEBUG   HandlerThread:576692 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 21:22:07,105 INFO    HandlerThread:576692 [handler.py:finish():638] shutting down handler
2021-07-14 21:22:08,002 INFO    WriterThread:576692 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_210351-1msvb4w4/run-1msvb4w4.wandb
2021-07-14 21:22:08,102 INFO    SenderThread:576692 [sender.py:finish():945] shutting down sender
2021-07-14 21:22:08,102 INFO    SenderThread:576692 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 21:22:08,102 INFO    SenderThread:576692 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 21:22:08,105 INFO    MainThread:576692 [internal.py:handle_exit():78] Internal process exited