| 2021-07-13 10:47:45,828 INFO MainThread:342403 [internal.py:wandb_internal():88] W&B internal server running at pid: 342403, started at: 2021-07-13 10:47:45.828158 | |
| 2021-07-13 10:47:45,830 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: check_version | |
| 2021-07-13 10:47:45,830 INFO WriterThread:342403 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/run-1rl2j7or.wandb | |
| 2021-07-13 10:47:45,831 DEBUG SenderThread:342403 [sender.py:send():179] send: header | |
| 2021-07-13 10:47:45,831 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: check_version | |
| 2021-07-13 10:47:45,871 DEBUG SenderThread:342403 [sender.py:send():179] send: run | |
| 2021-07-13 10:47:46,041 INFO SenderThread:342403 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files | |
| 2021-07-13 10:47:46,041 INFO SenderThread:342403 [sender.py:_start_run_threads():716] run started: 1rl2j7or with start time 1626173265 | |
| 2021-07-13 10:47:46,041 DEBUG SenderThread:342403 [sender.py:send():179] send: summary | |
| 2021-07-13 10:47:46,041 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: run_start | |
| 2021-07-13 10:47:46,042 INFO SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-13 10:47:47,043 INFO Thread-8 :342403 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json | |
| 2021-07-13 10:47:47,215 DEBUG HandlerThread:342403 [meta.py:__init__():39] meta init | |
| 2021-07-13 10:47:47,215 DEBUG HandlerThread:342403 [meta.py:__init__():53] meta init done | |
| 2021-07-13 10:47:47,215 DEBUG HandlerThread:342403 [meta.py:probe():210] probe | |
| 2021-07-13 10:47:47,217 DEBUG HandlerThread:342403 [meta.py:_setup_git():200] setup git | |
| 2021-07-13 10:47:47,250 DEBUG HandlerThread:342403 [meta.py:_setup_git():207] setup git done | |
| 2021-07-13 10:47:47,250 DEBUG HandlerThread:342403 [meta.py:_save_pip():57] save pip | |
| 2021-07-13 10:47:47,251 DEBUG HandlerThread:342403 [meta.py:_save_pip():71] save pip done | |
| 2021-07-13 10:47:47,251 DEBUG HandlerThread:342403 [meta.py:probe():252] probe done | |
| 2021-07-13 10:47:47,255 DEBUG SenderThread:342403 [sender.py:send():179] send: files | |
| 2021-07-13 10:47:47,255 INFO SenderThread:342403 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now | |
| 2021-07-13 10:47:47,262 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:47:47,262 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:47:47,394 DEBUG SenderThread:342403 [sender.py:send():179] send: config | |
| 2021-07-13 10:47:47,394 DEBUG SenderThread:342403 [sender.py:send():179] send: config | |
| 2021-07-13 10:47:47,394 DEBUG SenderThread:342403 [sender.py:send():179] send: config | |
| 2021-07-13 10:47:47,719 INFO Thread-11 :342403 [upload_job.py:push():137] Uploaded file /tmp/tmpta17r5ywwandb/1f1555en-wandb-metadata.json | |
| 2021-07-13 10:47:48,042 INFO Thread-8 :342403 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-metadata.json | |
| 2021-07-13 10:47:48,042 INFO Thread-8 :342403 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/requirements.txt | |
| 2021-07-13 10:47:48,042 INFO Thread-8 :342403 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:48:02,047 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:48:02,398 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:48:02,398 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:48:04,048 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:48:15,296 DEBUG SenderThread:342403 [sender.py:send():179] send: stats | |
| 2021-07-13 10:48:17,054 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/config.yaml | |
| 2021-07-13 10:48:17,555 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:48:17,556 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:48:32,709 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:48:32,710 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:48:45,371 DEBUG SenderThread:342403 [sender.py:send():179] send: stats | |
| 2021-07-13 10:48:47,840 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:48:47,840 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:49:02,980 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:49:02,980 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:49:15,445 DEBUG SenderThread:342403 [sender.py:send():179] send: stats | |
| 2021-07-13 10:49:18,113 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:49:18,113 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:49:24,080 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:49:26,080 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:49:28,081 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:49:30,082 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:49:32,083 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:49:33,242 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:49:33,243 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:49:34,084 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:49:36,084 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:49:45,514 DEBUG SenderThread:342403 [sender.py:send():179] send: stats | |
| 2021-07-13 10:49:48,375 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:49:48,375 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:49:58,179 DEBUG SenderThread:342403 [sender.py:send():179] send: history | |
| 2021-07-13 10:49:58,180 DEBUG SenderThread:342403 [sender.py:send():179] send: summary | |
| 2021-07-13 10:49:58,180 INFO SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-13 10:49:59,093 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json | |
| 2021-07-13 10:50:00,093 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:50:02,094 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:50:03,510 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:50:03,510 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:50:04,095 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:50:15,583 DEBUG SenderThread:342403 [sender.py:send():179] send: stats | |
| 2021-07-13 10:50:18,643 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:50:18,643 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:50:24,102 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:50:28,758 DEBUG SenderThread:342403 [sender.py:send():179] send: history | |
| 2021-07-13 10:50:28,759 DEBUG SenderThread:342403 [sender.py:send():179] send: summary | |
| 2021-07-13 10:50:28,763 INFO SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-13 10:50:29,104 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json | |
| 2021-07-13 10:50:30,105 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:50:32,106 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:50:33,775 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:50:33,776 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:50:34,107 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:50:36,107 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:50:38,108 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:50:40,109 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:50:42,110 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:50:45,653 DEBUG SenderThread:342403 [sender.py:send():179] send: stats | |
| 2021-07-13 10:50:48,905 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:50:48,906 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:51:04,035 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:51:04,035 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:51:04,964 DEBUG SenderThread:342403 [sender.py:send():179] send: history | |
| 2021-07-13 10:51:04,964 DEBUG SenderThread:342403 [sender.py:send():179] send: summary | |
| 2021-07-13 10:51:04,964 INFO SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-13 10:51:05,119 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json | |
| 2021-07-13 10:51:06,119 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:51:08,120 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:51:15,726 DEBUG SenderThread:342403 [sender.py:send():179] send: stats | |
| 2021-07-13 10:51:19,168 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:51:19,168 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:51:24,126 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:51:26,127 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:51:34,303 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:51:34,303 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:51:35,557 DEBUG SenderThread:342403 [sender.py:send():179] send: history | |
| 2021-07-13 10:51:35,558 DEBUG SenderThread:342403 [sender.py:send():179] send: summary | |
| 2021-07-13 10:51:35,558 INFO SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-13 10:51:36,131 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json | |
| 2021-07-13 10:51:36,132 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:51:38,132 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:51:40,133 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:51:42,134 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:51:44,135 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:51:45,797 DEBUG SenderThread:342403 [sender.py:send():179] send: stats | |
| 2021-07-13 10:51:46,136 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:51:48,137 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:51:49,438 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:51:49,438 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:51:50,137 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:52:04,579 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:52:04,580 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:52:11,761 DEBUG SenderThread:342403 [sender.py:send():179] send: history | |
| 2021-07-13 10:52:11,762 DEBUG SenderThread:342403 [sender.py:send():179] send: summary | |
| 2021-07-13 10:52:11,763 INFO SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-13 10:52:12,146 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json | |
| 2021-07-13 10:52:14,147 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:52:15,867 DEBUG SenderThread:342403 [sender.py:send():179] send: stats | |
| 2021-07-13 10:52:19,709 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:52:19,710 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:52:24,150 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:52:26,151 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:52:34,838 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:52:34,839 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:52:42,378 DEBUG SenderThread:342403 [sender.py:send():179] send: history | |
| 2021-07-13 10:52:42,378 DEBUG SenderThread:342403 [sender.py:send():179] send: summary | |
| 2021-07-13 10:52:42,379 INFO SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-13 10:52:43,158 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json | |
| 2021-07-13 10:52:45,159 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:52:45,939 DEBUG SenderThread:342403 [sender.py:send():179] send: stats | |
| 2021-07-13 10:52:47,160 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:52:49,161 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:52:49,969 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:52:49,970 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:52:51,161 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:52:53,162 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:52:55,163 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:52:57,164 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:53:05,101 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:53:05,101 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:53:16,014 DEBUG SenderThread:342403 [sender.py:send():179] send: stats | |
| 2021-07-13 10:53:18,580 DEBUG SenderThread:342403 [sender.py:send():179] send: history | |
| 2021-07-13 10:53:18,580 DEBUG SenderThread:342403 [sender.py:send():179] send: summary | |
| 2021-07-13 10:53:18,580 INFO SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-13 10:53:19,173 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json | |
| 2021-07-13 10:53:20,233 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 10:53:20,234 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 10:53:21,173 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:53:25,175 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:53:27,176 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:53:29,177 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:53:34,237 WARNING MainThread:342403 [internal.py:wandb_internal():147] Internal process interrupt: 1 | |
| 2021-07-13 10:53:34,484 WARNING MainThread:342403 [internal.py:wandb_internal():147] Internal process interrupt: 2 | |
| 2021-07-13 10:53:34,484 ERROR MainThread:342403 [internal.py:wandb_internal():150] Internal process interrupted. | |
| 2021-07-13 10:53:35,385 INFO WriterThread:342403 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/run-1rl2j7or.wandb | |
| 2021-07-13 10:53:35,409 INFO SenderThread:342403 [sender.py:finish():945] shutting down sender | |
| 2021-07-13 10:53:35,409 INFO SenderThread:342403 [dir_watcher.py:finish():282] shutting down directory watcher | |
| 2021-07-13 10:53:35,414 INFO HandlerThread:342403 [handler.py:finish():638] shutting down handler | |
| 2021-07-13 10:53:36,180 INFO SenderThread:342403 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files | |
| 2021-07-13 10:53:36,180 INFO SenderThread:342403 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/requirements.txt requirements.txt | |
| 2021-07-13 10:53:36,180 INFO SenderThread:342403 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log output.log | |
| 2021-07-13 10:53:36,180 INFO SenderThread:342403 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-metadata.json wandb-metadata.json | |
| 2021-07-13 10:53:36,180 INFO SenderThread:342403 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/config.yaml config.yaml | |
| 2021-07-13 10:53:36,181 INFO SenderThread:342403 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json wandb-summary.json | |
| 2021-07-13 10:53:36,181 INFO SenderThread:342403 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-13 10:53:36,181 INFO SenderThread:342403 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-13 10:53:36,622 INFO Thread-14 :342403 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/config.yaml | |
| 2021-07-13 10:53:36,624 INFO Thread-15 :342403 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json | |
| 2021-07-13 10:53:36,634 INFO Thread-13 :342403 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log | |
| 2021-07-13 10:53:36,654 INFO Thread-12 :342403 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/requirements.txt | |
| 2021-07-13 10:53:37,518 INFO MainThread:342403 [internal.py:handle_exit():78] Internal process exited | |