wandb/run-20210714_200731-1nr232tb/logs/debug-internal.log

2021-07-14 20:07:31,715 INFO    MainThread:566515 [internal.py:wandb_internal():88] W&B internal server running at pid: 566515, started at: 2021-07-14 20:07:31.714999
2021-07-14 20:07:31,717 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 20:07:31,717 INFO    WriterThread:566515 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/run-1nr232tb.wandb
2021-07-14 20:07:31,718 DEBUG   SenderThread:566515 [sender.py:send():179] send: header
2021-07-14 20:07:31,718 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: check_version
2021-07-14 20:07:31,754 DEBUG   SenderThread:566515 [sender.py:send():179] send: run
2021-07-14 20:07:31,935 INFO    SenderThread:566515 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files
2021-07-14 20:07:31,935 INFO    SenderThread:566515 [sender.py:_start_run_threads():716] run started: 1nr232tb with start time 1626293251
2021-07-14 20:07:31,935 DEBUG   SenderThread:566515 [sender.py:send():179] send: summary
2021-07-14 20:07:31,936 INFO    SenderThread:566515 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:07:31,936 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 20:07:32,938 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/wandb-summary.json
2021-07-14 20:07:33,069 DEBUG   HandlerThread:566515 [meta.py:__init__():39] meta init
2021-07-14 20:07:33,069 DEBUG   HandlerThread:566515 [meta.py:__init__():53] meta init done
2021-07-14 20:07:33,069 DEBUG   HandlerThread:566515 [meta.py:probe():210] probe
2021-07-14 20:07:33,071 DEBUG   HandlerThread:566515 [meta.py:_setup_git():200] setup git
2021-07-14 20:07:33,100 DEBUG   HandlerThread:566515 [meta.py:_setup_git():207] setup git done
2021-07-14 20:07:33,100 DEBUG   HandlerThread:566515 [meta.py:_save_pip():57] save pip
2021-07-14 20:07:33,101 DEBUG   HandlerThread:566515 [meta.py:_save_pip():71] save pip done
2021-07-14 20:07:33,101 DEBUG   HandlerThread:566515 [meta.py:probe():252] probe done
2021-07-14 20:07:33,104 DEBUG   SenderThread:566515 [sender.py:send():179] send: files
2021-07-14 20:07:33,104 INFO    SenderThread:566515 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 20:07:33,110 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:07:33,110 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:07:33,237 DEBUG   SenderThread:566515 [sender.py:send():179] send: config
2021-07-14 20:07:33,237 DEBUG   SenderThread:566515 [sender.py:send():179] send: config
2021-07-14 20:07:33,237 DEBUG   SenderThread:566515 [sender.py:send():179] send: config
2021-07-14 20:07:33,539 INFO    Thread-11 :566515 [upload_job.py:push():137] Uploaded file /tmp/tmpx7_gpcyxwandb/2fjtpy4h-wandb-metadata.json
2021-07-14 20:07:33,936 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:07:33,936 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/requirements.txt
2021-07-14 20:07:33,937 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/wandb-metadata.json
2021-07-14 20:07:47,942 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:07:48,239 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:07:48,239 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:07:53,944 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:08:01,150 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:08:02,948 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/config.yaml
2021-07-14 20:08:03,386 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:08:03,386 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:08:18,538 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:08:18,538 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:08:31,226 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:08:33,669 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:08:33,670 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:08:48,802 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:08:48,802 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:09:01,300 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:09:03,932 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:09:03,932 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:09:17,979 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:19,060 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:09:19,061 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:09:19,980 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:21,980 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:23,981 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:25,982 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:27,983 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:29,984 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:31,376 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:09:31,984 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:33,985 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:34,206 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:09:34,206 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:09:35,986 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:37,987 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:39,988 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:41,988 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:43,989 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:45,990 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:47,991 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:49,337 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:09:49,338 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:09:49,992 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:51,993 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:53,993 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:55,994 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:57,995 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:09:59,996 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:01,448 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:10:01,997 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:03,998 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:04,466 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:10:04,466 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:10:05,999 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:07,999 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:10,000 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:12,001 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:14,002 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:16,003 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:18,004 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:19,597 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:10:19,597 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:10:20,004 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:22,005 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:24,006 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:26,007 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:28,007 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:30,008 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:31,520 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:10:32,009 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:34,010 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:34,732 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:10:34,732 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:10:36,011 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:38,012 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:40,012 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:42,013 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:44,014 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:46,015 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:48,016 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:49,870 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:10:49,871 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:10:50,017 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:52,018 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:54,018 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:56,019 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:10:58,020 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:00,021 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:01,597 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:11:02,022 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:04,023 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:05,128 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:11:05,129 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:11:06,024 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:08,025 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:10,025 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:12,026 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:14,027 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:16,028 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:18,029 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:20,029 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:20,264 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:11:20,264 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:11:22,030 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:24,031 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:26,032 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:28,033 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:30,034 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:31,674 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:11:32,035 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:34,036 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:35,397 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:11:35,397 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:11:36,036 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:38,037 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:40,038 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:42,039 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:44,040 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:46,041 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:48,041 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:50,042 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:50,529 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:11:50,530 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:11:52,043 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:54,044 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:11:56,045 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:01,751 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:12:05,662 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:12:05,662 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:12:18,086 DEBUG   SenderThread:566515 [sender.py:send():179] send: history
2021-07-14 20:12:18,087 DEBUG   SenderThread:566515 [sender.py:send():179] send: summary
2021-07-14 20:12:18,089 INFO    SenderThread:566515 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:12:19,054 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/wandb-summary.json
2021-07-14 20:12:20,055 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:20,796 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:12:20,797 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:12:22,055 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:24,056 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:26,057 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:28,058 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:30,059 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:31,829 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:12:32,060 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:34,061 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:35,940 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:12:35,941 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:12:36,061 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:38,062 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:40,063 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:42,064 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:44,065 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:46,065 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:48,066 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:50,067 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:51,073 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:12:51,073 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:12:52,068 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:54,069 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:56,070 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:12:58,070 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:00,071 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:01,907 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:13:02,072 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:04,073 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:06,073 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:06,206 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:13:06,206 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:13:08,074 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:10,075 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:12,076 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:14,077 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:16,078 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:18,079 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:20,079 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:21,336 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:13:21,336 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:13:22,080 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:24,081 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:26,082 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:28,083 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:30,083 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:31,983 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:13:32,084 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:34,085 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:36,086 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:36,469 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:13:36,469 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:13:38,086 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:40,087 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:43,089 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:45,089 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:47,090 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:49,091 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:51,092 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:51,600 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:13:51,601 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:13:53,093 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:55,094 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:57,095 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:13:59,096 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:01,096 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:02,063 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:14:03,097 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:05,098 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:06,737 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:14:06,738 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:14:07,099 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:09,099 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:11,100 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:13,101 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:15,102 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:17,103 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:19,104 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:21,105 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:21,868 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:14:21,868 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:14:23,106 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:25,106 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:27,107 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:29,108 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:31,109 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:32,140 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:14:33,110 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:35,111 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:36,998 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:14:36,998 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:14:37,112 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:39,113 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:41,114 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:43,115 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:45,116 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:47,116 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:49,117 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:51,118 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:52,130 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:14:52,130 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:14:53,119 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:55,120 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:14:57,120 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:02,218 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:15:07,259 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:15:07,259 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:15:18,883 DEBUG   SenderThread:566515 [sender.py:send():179] send: history
2021-07-14 20:15:18,884 DEBUG   SenderThread:566515 [sender.py:send():179] send: summary
2021-07-14 20:15:18,884 INFO    SenderThread:566515 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:15:19,129 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/wandb-summary.json
2021-07-14 20:15:20,129 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:21,130 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:22,387 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:15:22,388 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:15:23,130 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:25,131 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:27,132 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:29,133 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:31,134 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:32,295 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:15:33,135 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:35,136 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:37,137 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:37,548 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:15:37,549 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:15:39,137 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:41,138 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:43,139 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:45,140 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:47,141 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:49,141 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:51,142 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:52,677 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:15:52,677 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:15:53,144 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:55,144 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:57,145 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:15:59,146 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:01,147 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:02,377 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:16:03,148 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:05,149 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:07,150 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:07,812 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:16:07,812 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:16:09,151 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:11,152 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:13,152 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:15,153 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:17,154 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:19,155 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:21,156 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:22,944 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:16:22,944 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:16:23,157 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:25,158 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:27,158 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:29,159 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:31,160 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:32,451 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:16:33,161 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:35,162 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:37,163 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:38,076 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:16:38,076 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:16:39,164 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:41,165 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:43,165 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:45,166 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:47,167 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:49,168 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:51,169 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:53,169 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:53,207 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:16:53,207 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:16:55,170 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:57,171 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:16:59,172 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:01,173 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:02,529 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:17:03,173 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:05,174 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:07,175 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:08,337 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:17:08,337 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:17:09,176 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:11,177 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:13,178 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:15,179 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:17,180 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:19,180 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:21,181 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:23,182 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:23,470 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:17:23,470 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:17:25,183 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:27,184 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:29,184 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:31,185 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:32,603 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:17:33,186 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:35,187 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:37,189 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:38,604 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:17:38,604 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:17:39,190 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:41,190 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:43,191 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:45,192 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:47,193 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:49,194 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:51,195 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:53,196 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:53,735 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:17:53,735 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:17:55,198 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:17:57,198 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:02,679 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:18:08,867 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:18:08,867 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:18:19,684 DEBUG   SenderThread:566515 [sender.py:send():179] send: history
2021-07-14 20:18:19,684 DEBUG   SenderThread:566515 [sender.py:send():179] send: summary
2021-07-14 20:18:19,685 INFO    SenderThread:566515 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:18:20,207 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/wandb-summary.json
2021-07-14 20:18:21,207 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:23,208 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:24,001 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:18:24,002 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:18:25,209 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:27,210 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:29,211 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:31,212 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:32,757 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:18:33,212 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:35,213 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:37,214 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:39,132 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:18:39,133 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:18:39,215 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:41,216 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:43,217 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:45,218 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:47,219 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:49,219 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:51,220 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:53,221 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:54,264 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:18:54,265 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:18:55,222 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:57,223 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:18:59,224 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:01,225 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:02,833 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:19:03,226 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:05,227 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:07,228 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:09,228 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:09,395 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:19:09,395 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:19:11,229 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:13,230 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:15,231 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:17,232 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:19,233 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:21,234 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:23,235 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:24,524 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:19:24,524 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:19:25,235 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:27,236 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:29,237 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:31,238 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:32,913 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:19:33,239 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:35,240 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:37,241 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:39,242 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:39,653 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:19:39,653 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:19:41,242 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:43,243 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:45,244 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:47,245 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:49,246 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:51,247 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:53,247 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:54,783 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:19:54,783 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:19:55,248 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:57,249 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:19:59,250 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:01,251 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:02,990 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:20:03,252 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:05,252 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:07,253 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:09,254 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:09,917 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:20:09,918 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:20:11,255 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:13,256 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:15,256 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:17,257 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:19,258 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:21,259 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:23,260 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:25,051 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:20:25,052 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:20:25,261 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:27,261 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:29,262 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:31,263 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:33,069 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:20:33,264 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:35,264 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:37,265 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:39,266 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:40,196 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:20:40,197 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:20:41,267 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:43,267 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:45,268 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:47,269 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:50,270 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:52,271 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:54,272 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:55,327 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:20:55,328 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:20:56,273 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:20:58,274 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:03,146 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:21:10,456 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:21:10,456 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:21:20,479 DEBUG   SenderThread:566515 [sender.py:send():179] send: history
2021-07-14 20:21:20,479 DEBUG   SenderThread:566515 [sender.py:send():179] send: summary
2021-07-14 20:21:20,479 INFO    SenderThread:566515 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:21:21,283 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:21,283 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/wandb-summary.json
2021-07-14 20:21:22,284 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:24,284 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:25,588 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:21:25,589 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:21:26,285 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:28,286 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:30,287 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:32,288 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:33,224 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:21:34,289 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:36,290 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:38,290 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:40,291 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:40,733 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:21:40,733 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:21:42,292 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:44,293 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:46,294 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:48,294 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:50,295 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:52,296 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:54,297 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:55,896 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:21:55,896 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:21:56,297 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:21:58,298 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:00,299 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:02,300 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:03,301 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:22:04,301 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:06,302 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:08,303 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:10,304 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:11,047 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:22:11,047 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:22:12,305 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:14,305 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:16,306 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:18,307 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:20,308 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:22,309 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:24,310 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:26,203 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:22:26,204 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:22:26,310 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:28,311 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:30,312 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:32,313 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:33,378 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:22:34,314 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:36,315 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:38,316 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:40,316 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:41,363 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:22:41,363 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:22:42,317 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:44,318 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:46,319 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:48,319 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:50,320 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:52,321 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:54,322 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:56,323 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:22:56,674 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:22:56,674 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:22:58,324 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:00,324 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:02,325 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:03,454 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:23:04,326 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:06,327 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:08,328 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:10,329 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:11,826 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:23:11,826 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:23:12,330 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:14,330 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:16,331 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:18,332 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:20,333 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:22,333 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:24,334 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:26,335 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:26,984 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:23:26,985 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:23:28,336 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:30,337 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:32,338 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:33,531 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:23:34,339 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:36,339 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:38,340 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:40,341 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:42,135 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:23:42,135 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:23:42,342 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:44,343 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:46,344 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:48,344 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:50,345 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:52,346 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:54,347 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:56,348 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:23:57,279 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:23:57,280 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:23:58,348 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:24:03,607 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:24:12,410 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:24:12,411 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:24:22,445 DEBUG   SenderThread:566515 [sender.py:send():179] send: history
2021-07-14 20:24:22,445 DEBUG   SenderThread:566515 [sender.py:send():179] send: summary
2021-07-14 20:24:22,446 INFO    SenderThread:566515 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:24:23,359 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/wandb-summary.json
2021-07-14 20:24:24,359 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:24:26,360 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:24:27,539 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:24:27,540 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:24:28,361 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:24:30,362 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:24:32,363 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:24:33,686 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:24:34,364 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:24:36,365 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:24:38,365 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:24:40,366 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:24:42,367 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:24:42,685 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:24:42,685 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:24:44,368 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:24:46,369 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:24:48,370 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:24:50,370 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:24:52,371 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:24:54,372 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:24:56,373 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:24:57,841 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:24:57,841 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:24:58,374 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:00,375 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:02,376 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:03,764 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:25:04,377 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:06,378 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:08,378 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:10,379 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:12,380 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:12,999 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:25:12,999 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:25:14,381 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:16,382 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:18,382 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:20,383 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:22,384 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:24,385 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:26,386 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:28,149 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:25:28,149 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:25:28,387 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:30,388 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:32,388 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:33,841 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:25:34,389 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:36,390 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:38,391 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:40,392 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:42,393 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:43,294 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:25:43,295 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:25:44,394 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:46,395 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:48,395 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:50,396 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:52,397 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:54,398 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:56,399 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:58,399 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:25:58,450 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:25:58,451 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:26:00,400 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:02,401 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:03,919 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:26:04,402 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:06,403 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:08,404 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:10,405 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:12,406 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:13,618 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:26:13,618 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:26:14,406 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:16,407 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:18,408 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:20,409 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:22,409 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:24,410 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:26,411 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:28,412 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:28,768 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:26:28,769 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:26:30,413 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:32,414 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:33,994 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:26:34,415 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:36,415 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:38,416 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:40,417 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:42,418 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:43,915 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:26:43,915 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:26:44,419 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:46,420 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:48,420 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:50,421 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:52,422 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:54,423 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:56,423 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:58,424 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:26:59,082 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:26:59,083 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:27:00,425 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:27:04,074 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:27:14,232 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:27:14,233 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:27:23,233 DEBUG   SenderThread:566515 [sender.py:send():179] send: history
2021-07-14 20:27:23,233 DEBUG   SenderThread:566515 [sender.py:send():179] send: summary
2021-07-14 20:27:23,234 INFO    SenderThread:566515 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:27:23,434 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/wandb-summary.json
2021-07-14 20:27:24,434 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:27:27,436 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:27:29,366 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:27:29,366 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:27:29,436 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:27:31,437 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:27:33,438 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:27:34,152 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:27:35,439 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:27:37,440 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:27:39,441 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:27:41,442 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:27:43,443 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:27:44,501 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:27:44,501 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:27:45,444 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:27:47,445 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:27:49,445 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:27:51,446 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:27:53,447 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:27:55,448 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:27:57,448 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:27:59,449 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:27:59,635 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:27:59,635 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:28:01,450 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:03,451 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:04,228 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:28:05,451 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:07,452 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:09,453 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:11,454 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:13,455 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:14,769 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:28:14,769 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:28:15,456 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:17,457 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:19,458 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:21,458 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:23,459 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:25,460 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:27,461 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:29,462 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:29,907 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:28:29,907 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:28:31,462 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:33,463 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:34,306 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:28:35,464 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:37,465 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:39,466 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:41,466 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:43,467 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:45,053 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:28:45,053 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:28:45,468 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:47,469 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:49,470 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:51,471 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:53,472 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:55,472 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:57,473 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:28:59,474 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:00,186 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:29:00,186 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:29:01,475 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:03,476 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:04,385 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:29:05,477 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:07,477 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:09,478 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:11,479 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:13,480 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:15,423 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:29:15,424 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:29:15,481 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:17,481 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:19,482 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:21,483 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:23,484 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:25,485 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:27,485 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:29,486 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:30,554 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:29:30,554 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:29:31,487 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:33,488 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:34,462 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:29:35,489 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:37,490 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:39,490 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:41,491 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:43,492 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:45,493 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:45,688 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:29:45,688 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:29:47,494 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:49,495 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:51,495 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:53,496 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:55,497 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:57,498 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:29:59,499 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:30:00,818 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:30:00,818 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:30:01,500 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:30:04,537 DEBUG   SenderThread:566515 [sender.py:send():179] send: stats
2021-07-14 20:30:15,952 DEBUG   HandlerThread:566515 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:30:15,952 DEBUG   SenderThread:566515 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:30:24,030 DEBUG   SenderThread:566515 [sender.py:send():179] send: history
2021-07-14 20:30:24,031 DEBUG   SenderThread:566515 [sender.py:send():179] send: summary
2021-07-14 20:30:24,031 INFO    SenderThread:566515 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:30:24,509 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/wandb-summary.json
2021-07-14 20:30:25,509 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:30:26,440 WARNING MainThread:566515 [internal.py:wandb_internal():147] Internal process interrupt: 1
2021-07-14 20:30:27,510 INFO    Thread-8  :566515 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:30:27,654 WARNING MainThread:566515 [internal.py:wandb_internal():147] Internal process interrupt: 2
2021-07-14 20:30:27,654 ERROR   MainThread:566515 [internal.py:wandb_internal():150] Internal process interrupted.
2021-07-14 20:30:27,903 INFO    SenderThread:566515 [sender.py:finish():945] shutting down sender
2021-07-14 20:30:27,903 INFO    WriterThread:566515 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/run-1nr232tb.wandb
2021-07-14 20:30:27,903 INFO    SenderThread:566515 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 20:30:27,904 INFO    HandlerThread:566515 [handler.py:finish():638] shutting down handler
2021-07-14 20:30:28,511 INFO    SenderThread:566515 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files
2021-07-14 20:30:28,511 INFO    SenderThread:566515 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/requirements.txt requirements.txt
2021-07-14 20:30:28,511 INFO    SenderThread:566515 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log output.log
2021-07-14 20:30:28,511 INFO    SenderThread:566515 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/wandb-metadata.json wandb-metadata.json
2021-07-14 20:30:28,511 INFO    SenderThread:566515 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/config.yaml config.yaml
2021-07-14 20:30:28,512 INFO    SenderThread:566515 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/wandb-summary.json wandb-summary.json
2021-07-14 20:30:28,515 INFO    SenderThread:566515 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 20:30:28,515 INFO    SenderThread:566515 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 20:30:29,047 INFO    Thread-15 :566515 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/wandb-summary.json
2021-07-14 20:30:29,220 INFO    Thread-12 :566515 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/requirements.txt
2021-07-14 20:30:29,221 INFO    Thread-14 :566515 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/config.yaml
2021-07-14 20:30:29,222 INFO    Thread-13 :566515 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_200731-1nr232tb/files/output.log
2021-07-14 20:30:29,224 INFO    MainThread:566515 [internal.py:handle_exit():78] Internal process exited