File size: 25,025 Bytes
f291f93 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 |
2021-07-13 10:47:45,828 INFO MainThread:342403 [internal.py:wandb_internal():88] W&B internal server running at pid: 342403, started at: 2021-07-13 10:47:45.828158
2021-07-13 10:47:45,830 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: check_version
2021-07-13 10:47:45,830 INFO WriterThread:342403 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/run-1rl2j7or.wandb
2021-07-13 10:47:45,831 DEBUG SenderThread:342403 [sender.py:send():179] send: header
2021-07-13 10:47:45,831 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: check_version
2021-07-13 10:47:45,871 DEBUG SenderThread:342403 [sender.py:send():179] send: run
2021-07-13 10:47:46,041 INFO SenderThread:342403 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files
2021-07-13 10:47:46,041 INFO SenderThread:342403 [sender.py:_start_run_threads():716] run started: 1rl2j7or with start time 1626173265
2021-07-13 10:47:46,041 DEBUG SenderThread:342403 [sender.py:send():179] send: summary
2021-07-13 10:47:46,041 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: run_start
2021-07-13 10:47:46,042 INFO SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 10:47:47,043 INFO Thread-8 :342403 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json
2021-07-13 10:47:47,215 DEBUG HandlerThread:342403 [meta.py:__init__():39] meta init
2021-07-13 10:47:47,215 DEBUG HandlerThread:342403 [meta.py:__init__():53] meta init done
2021-07-13 10:47:47,215 DEBUG HandlerThread:342403 [meta.py:probe():210] probe
2021-07-13 10:47:47,217 DEBUG HandlerThread:342403 [meta.py:_setup_git():200] setup git
2021-07-13 10:47:47,250 DEBUG HandlerThread:342403 [meta.py:_setup_git():207] setup git done
2021-07-13 10:47:47,250 DEBUG HandlerThread:342403 [meta.py:_save_pip():57] save pip
2021-07-13 10:47:47,251 DEBUG HandlerThread:342403 [meta.py:_save_pip():71] save pip done
2021-07-13 10:47:47,251 DEBUG HandlerThread:342403 [meta.py:probe():252] probe done
2021-07-13 10:47:47,255 DEBUG SenderThread:342403 [sender.py:send():179] send: files
2021-07-13 10:47:47,255 INFO SenderThread:342403 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-13 10:47:47,262 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:47:47,262 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:47:47,394 DEBUG SenderThread:342403 [sender.py:send():179] send: config
2021-07-13 10:47:47,394 DEBUG SenderThread:342403 [sender.py:send():179] send: config
2021-07-13 10:47:47,394 DEBUG SenderThread:342403 [sender.py:send():179] send: config
2021-07-13 10:47:47,719 INFO Thread-11 :342403 [upload_job.py:push():137] Uploaded file /tmp/tmpta17r5ywwandb/1f1555en-wandb-metadata.json
2021-07-13 10:47:48,042 INFO Thread-8 :342403 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-metadata.json
2021-07-13 10:47:48,042 INFO Thread-8 :342403 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/requirements.txt
2021-07-13 10:47:48,042 INFO Thread-8 :342403 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:48:02,047 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:48:02,398 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:48:02,398 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:48:04,048 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:48:15,296 DEBUG SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:48:17,054 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/config.yaml
2021-07-13 10:48:17,555 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:48:17,556 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:48:32,709 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:48:32,710 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:48:45,371 DEBUG SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:48:47,840 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:48:47,840 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:49:02,980 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:49:02,980 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:49:15,445 DEBUG SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:49:18,113 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:49:18,113 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:49:24,080 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:49:26,080 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:49:28,081 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:49:30,082 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:49:32,083 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:49:33,242 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:49:33,243 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:49:34,084 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:49:36,084 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:49:45,514 DEBUG SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:49:48,375 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:49:48,375 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:49:58,179 DEBUG SenderThread:342403 [sender.py:send():179] send: history
2021-07-13 10:49:58,180 DEBUG SenderThread:342403 [sender.py:send():179] send: summary
2021-07-13 10:49:58,180 INFO SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 10:49:59,093 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json
2021-07-13 10:50:00,093 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:02,094 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:03,510 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:50:03,510 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:50:04,095 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:15,583 DEBUG SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:50:18,643 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:50:18,643 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:50:24,102 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:28,758 DEBUG SenderThread:342403 [sender.py:send():179] send: history
2021-07-13 10:50:28,759 DEBUG SenderThread:342403 [sender.py:send():179] send: summary
2021-07-13 10:50:28,763 INFO SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 10:50:29,104 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json
2021-07-13 10:50:30,105 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:32,106 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:33,775 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:50:33,776 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:50:34,107 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:36,107 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:38,108 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:40,109 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:42,110 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:45,653 DEBUG SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:50:48,905 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:50:48,906 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:51:04,035 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:51:04,035 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:51:04,964 DEBUG SenderThread:342403 [sender.py:send():179] send: history
2021-07-13 10:51:04,964 DEBUG SenderThread:342403 [sender.py:send():179] send: summary
2021-07-13 10:51:04,964 INFO SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 10:51:05,119 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json
2021-07-13 10:51:06,119 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:08,120 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:15,726 DEBUG SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:51:19,168 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:51:19,168 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:51:24,126 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:26,127 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:34,303 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:51:34,303 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:51:35,557 DEBUG SenderThread:342403 [sender.py:send():179] send: history
2021-07-13 10:51:35,558 DEBUG SenderThread:342403 [sender.py:send():179] send: summary
2021-07-13 10:51:35,558 INFO SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 10:51:36,131 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json
2021-07-13 10:51:36,132 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:38,132 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:40,133 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:42,134 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:44,135 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:45,797 DEBUG SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:51:46,136 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:48,137 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:49,438 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:51:49,438 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:51:50,137 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:04,579 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:52:04,580 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:52:11,761 DEBUG SenderThread:342403 [sender.py:send():179] send: history
2021-07-13 10:52:11,762 DEBUG SenderThread:342403 [sender.py:send():179] send: summary
2021-07-13 10:52:11,763 INFO SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 10:52:12,146 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json
2021-07-13 10:52:14,147 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:15,867 DEBUG SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:52:19,709 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:52:19,710 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:52:24,150 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:26,151 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:34,838 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:52:34,839 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:52:42,378 DEBUG SenderThread:342403 [sender.py:send():179] send: history
2021-07-13 10:52:42,378 DEBUG SenderThread:342403 [sender.py:send():179] send: summary
2021-07-13 10:52:42,379 INFO SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 10:52:43,158 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json
2021-07-13 10:52:45,159 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:45,939 DEBUG SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:52:47,160 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:49,161 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:49,969 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:52:49,970 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:52:51,161 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:53,162 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:55,163 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:57,164 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:53:05,101 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:53:05,101 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:53:16,014 DEBUG SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:53:18,580 DEBUG SenderThread:342403 [sender.py:send():179] send: history
2021-07-13 10:53:18,580 DEBUG SenderThread:342403 [sender.py:send():179] send: summary
2021-07-13 10:53:18,580 INFO SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 10:53:19,173 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json
2021-07-13 10:53:20,233 DEBUG HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:53:20,234 DEBUG SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:53:21,173 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:53:25,175 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:53:27,176 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:53:29,177 INFO Thread-8 :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:53:34,237 WARNING MainThread:342403 [internal.py:wandb_internal():147] Internal process interrupt: 1
2021-07-13 10:53:34,484 WARNING MainThread:342403 [internal.py:wandb_internal():147] Internal process interrupt: 2
2021-07-13 10:53:34,484 ERROR MainThread:342403 [internal.py:wandb_internal():150] Internal process interrupted.
2021-07-13 10:53:35,385 INFO WriterThread:342403 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/run-1rl2j7or.wandb
2021-07-13 10:53:35,409 INFO SenderThread:342403 [sender.py:finish():945] shutting down sender
2021-07-13 10:53:35,409 INFO SenderThread:342403 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-13 10:53:35,414 INFO HandlerThread:342403 [handler.py:finish():638] shutting down handler
2021-07-13 10:53:36,180 INFO SenderThread:342403 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files
2021-07-13 10:53:36,180 INFO SenderThread:342403 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/requirements.txt requirements.txt
2021-07-13 10:53:36,180 INFO SenderThread:342403 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log output.log
2021-07-13 10:53:36,180 INFO SenderThread:342403 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-metadata.json wandb-metadata.json
2021-07-13 10:53:36,180 INFO SenderThread:342403 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/config.yaml config.yaml
2021-07-13 10:53:36,181 INFO SenderThread:342403 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json wandb-summary.json
2021-07-13 10:53:36,181 INFO SenderThread:342403 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 10:53:36,181 INFO SenderThread:342403 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 10:53:36,622 INFO Thread-14 :342403 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/config.yaml
2021-07-13 10:53:36,624 INFO Thread-15 :342403 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json
2021-07-13 10:53:36,634 INFO Thread-13 :342403 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:53:36,654 INFO Thread-12 :342403 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/requirements.txt
2021-07-13 10:53:37,518 INFO MainThread:342403 [internal.py:handle_exit():78] Internal process exited
|