File size: 47,661 Bytes
f291f93 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 |
2021-07-14 20:31:18,076 INFO MainThread:569706 [internal.py:wandb_internal():88] W&B internal server running at pid: 569706, started at: 2021-07-14 20:31:18.075968
2021-07-14 20:31:18,078 INFO WriterThread:569706 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/run-3emij6kq.wandb
2021-07-14 20:31:18,078 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 20:31:18,079 DEBUG SenderThread:569706 [sender.py:send():179] send: header
2021-07-14 20:31:18,079 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: check_version
2021-07-14 20:31:18,118 DEBUG SenderThread:569706 [sender.py:send():179] send: run
2021-07-14 20:31:18,293 INFO SenderThread:569706 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files
2021-07-14 20:31:18,293 INFO SenderThread:569706 [sender.py:_start_run_threads():716] run started: 3emij6kq with start time 1626294677
2021-07-14 20:31:18,293 DEBUG SenderThread:569706 [sender.py:send():179] send: summary
2021-07-14 20:31:18,293 INFO SenderThread:569706 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:31:18,293 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 20:31:19,296 INFO Thread-8 :569706 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/wandb-summary.json
2021-07-14 20:31:19,373 DEBUG HandlerThread:569706 [meta.py:__init__():39] meta init
2021-07-14 20:31:19,373 DEBUG HandlerThread:569706 [meta.py:__init__():53] meta init done
2021-07-14 20:31:19,373 DEBUG HandlerThread:569706 [meta.py:probe():210] probe
2021-07-14 20:31:19,374 DEBUG HandlerThread:569706 [meta.py:_setup_git():200] setup git
2021-07-14 20:31:19,403 DEBUG HandlerThread:569706 [meta.py:_setup_git():207] setup git done
2021-07-14 20:31:19,403 DEBUG HandlerThread:569706 [meta.py:_save_pip():57] save pip
2021-07-14 20:31:19,404 DEBUG HandlerThread:569706 [meta.py:_save_pip():71] save pip done
2021-07-14 20:31:19,404 DEBUG HandlerThread:569706 [meta.py:probe():252] probe done
2021-07-14 20:31:19,407 DEBUG SenderThread:569706 [sender.py:send():179] send: files
2021-07-14 20:31:19,407 INFO SenderThread:569706 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 20:31:19,413 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:31:19,413 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:31:19,541 DEBUG SenderThread:569706 [sender.py:send():179] send: config
2021-07-14 20:31:19,541 DEBUG SenderThread:569706 [sender.py:send():179] send: config
2021-07-14 20:31:19,542 DEBUG SenderThread:569706 [sender.py:send():179] send: config
2021-07-14 20:31:19,853 INFO Thread-11 :569706 [upload_job.py:push():137] Uploaded file /tmp/tmprctrqfj8wandb/151zr105-wandb-metadata.json
2021-07-14 20:31:20,294 INFO Thread-8 :569706 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/requirements.txt
2021-07-14 20:31:20,295 INFO Thread-8 :569706 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:31:20,295 INFO Thread-8 :569706 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/wandb-metadata.json
2021-07-14 20:31:34,300 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:31:34,543 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:31:34,543 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:31:36,300 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:31:40,302 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:31:47,455 DEBUG SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:31:49,306 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/config.yaml
2021-07-14 20:31:49,691 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:31:49,692 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:32:04,839 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:32:04,839 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:32:17,538 DEBUG SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:32:19,973 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:32:19,973 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:32:35,105 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:32:35,105 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:32:47,616 DEBUG SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:32:50,237 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:32:50,237 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:33:05,367 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:33:05,367 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:33:17,693 DEBUG SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:33:18,340 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:20,341 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:20,549 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:33:20,549 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:33:22,342 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:24,343 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:26,344 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:28,345 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:30,346 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:32,346 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:34,347 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:35,690 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:33:35,691 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:33:36,348 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:38,349 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:40,350 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:42,351 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:44,352 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:46,353 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:47,771 DEBUG SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:33:48,354 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:50,355 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:50,833 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:33:50,833 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:33:52,356 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:54,356 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:56,357 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:33:58,358 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:00,359 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:02,360 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:04,361 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:05,963 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:34:05,963 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:34:06,362 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:08,363 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:10,364 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:12,365 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:14,366 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:16,367 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:17,848 DEBUG SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:34:18,368 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:20,369 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:21,092 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:34:21,093 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:34:22,369 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:24,370 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:26,371 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:28,372 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:30,373 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:32,374 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:34,375 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:36,254 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:34:36,254 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:34:36,376 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:38,377 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:40,377 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:42,379 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:44,380 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:46,381 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:47,925 DEBUG SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:34:48,381 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:50,382 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:51,390 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:34:51,390 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:34:52,383 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:54,384 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:56,385 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:34:58,386 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:00,386 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:02,387 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:04,388 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:06,389 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:06,526 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:35:06,527 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:35:08,390 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:10,391 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:12,391 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:14,392 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:16,393 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:18,004 DEBUG SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:35:18,394 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:20,395 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:21,675 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:35:21,675 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:35:22,396 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:24,397 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:26,398 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:28,398 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:30,399 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:32,400 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:34,401 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:36,402 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:36,809 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:35:36,810 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:35:39,403 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:41,404 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:43,405 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:45,406 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:47,407 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:48,079 DEBUG SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:35:49,408 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:51,409 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:35:51,945 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:35:51,945 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:35:53,410 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:07,092 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:36:07,093 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:36:13,798 DEBUG SenderThread:569706 [sender.py:send():179] send: history
2021-07-14 20:36:13,798 DEBUG SenderThread:569706 [sender.py:send():179] send: summary
2021-07-14 20:36:13,800 INFO SenderThread:569706 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:36:14,418 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:14,418 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/wandb-summary.json
2021-07-14 20:36:15,419 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:17,419 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:18,159 DEBUG SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:36:19,420 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:21,421 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:22,223 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:36:22,223 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:36:23,422 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:25,423 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:27,424 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:29,425 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:31,426 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:33,427 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:35,428 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:37,354 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:36:37,354 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:36:37,429 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:39,430 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:41,431 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:43,431 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:45,432 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:47,433 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:48,232 DEBUG SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:36:49,434 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:51,435 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:52,488 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:36:52,488 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:36:53,436 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:55,437 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:57,438 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:36:59,439 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:01,440 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:03,441 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:05,442 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:07,443 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:07,644 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:37:07,644 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:37:09,444 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:11,445 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:13,446 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:15,447 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:17,447 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:18,309 DEBUG SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:37:19,448 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:21,449 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:22,825 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:37:22,825 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:37:23,450 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:25,451 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:27,452 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:29,453 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:31,454 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:33,455 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:35,456 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:37,457 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:37,956 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:37:37,956 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:37:39,458 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:41,459 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:43,460 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:45,461 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:47,462 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:48,387 DEBUG SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:37:49,463 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:51,463 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:53,090 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:37:53,091 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:37:53,464 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:55,465 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:57,466 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:37:59,467 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:01,468 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:03,469 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:05,470 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:07,470 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:08,227 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:38:08,228 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:38:09,471 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:11,472 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:13,473 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:15,474 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:17,474 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:18,464 DEBUG SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:38:19,475 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:21,476 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:23,361 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:38:23,362 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:38:23,477 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:25,478 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:27,479 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:29,480 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:31,481 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:33,482 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:35,483 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:37,484 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:38,492 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:38:38,492 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:38:39,485 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:41,486 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:43,487 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:45,488 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:47,488 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:38:48,541 DEBUG SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:38:53,658 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:38:53,658 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:39:08,787 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:39:08,788 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:39:09,733 DEBUG SenderThread:569706 [sender.py:send():179] send: history
2021-07-14 20:39:09,733 DEBUG SenderThread:569706 [sender.py:send():179] send: summary
2021-07-14 20:39:09,734 INFO SenderThread:569706 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:39:10,498 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/wandb-summary.json
2021-07-14 20:39:11,498 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:39:13,499 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:39:15,500 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:39:17,501 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:39:18,620 DEBUG SenderThread:569706 [sender.py:send():179] send: stats
2021-07-14 20:39:19,502 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:39:21,503 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:39:23,504 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:39:23,928 DEBUG HandlerThread:569706 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:39:23,928 DEBUG SenderThread:569706 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:39:25,505 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:39:27,119 WARNING MainThread:569706 [internal.py:wandb_internal():147] Internal process interrupt: 1
2021-07-14 20:39:27,506 INFO Thread-8 :569706 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log
2021-07-14 20:39:27,538 WARNING MainThread:569706 [internal.py:wandb_internal():147] Internal process interrupt: 2
2021-07-14 20:39:27,539 ERROR MainThread:569706 [internal.py:wandb_internal():150] Internal process interrupted.
2021-07-14 20:39:28,203 INFO SenderThread:569706 [sender.py:finish():945] shutting down sender
2021-07-14 20:39:28,203 INFO WriterThread:569706 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/run-3emij6kq.wandb
2021-07-14 20:39:28,204 INFO SenderThread:569706 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 20:39:28,204 INFO HandlerThread:569706 [handler.py:finish():638] shutting down handler
2021-07-14 20:39:28,507 INFO SenderThread:569706 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files
2021-07-14 20:39:28,507 INFO SenderThread:569706 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/requirements.txt requirements.txt
2021-07-14 20:39:28,507 INFO SenderThread:569706 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/output.log output.log
2021-07-14 20:39:28,507 INFO SenderThread:569706 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/wandb-metadata.json wandb-metadata.json
2021-07-14 20:39:28,507 INFO SenderThread:569706 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/config.yaml config.yaml
2021-07-14 20:39:28,508 INFO SenderThread:569706 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_203117-3emij6kq/files/wandb-summary.json wandb-summary.json
2021-07-14 20:39:28,508 INFO SenderThread:569706 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 20:39:28,508 INFO SenderThread:569706 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 20:39:28,823 INFO MainThread:569706 [internal.py:handle_exit():78] Internal process exited
|