File size: 32,915 Bytes
f291f93 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 |
2021-07-14 19:25:15,501 INFO MainThread:558812 [internal.py:wandb_internal():88] W&B internal server running at pid: 558812, started at: 2021-07-14 19:25:15.501475
2021-07-14 19:25:15,503 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 19:25:15,504 INFO WriterThread:558812 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/run-fdn39gsq.wandb
2021-07-14 19:25:15,505 DEBUG SenderThread:558812 [sender.py:send():179] send: header
2021-07-14 19:25:15,505 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: check_version
2021-07-14 19:25:15,550 DEBUG SenderThread:558812 [sender.py:send():179] send: run
2021-07-14 19:25:15,726 INFO SenderThread:558812 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files
2021-07-14 19:25:15,726 INFO SenderThread:558812 [sender.py:_start_run_threads():716] run started: fdn39gsq with start time 1626290714
2021-07-14 19:25:15,726 DEBUG SenderThread:558812 [sender.py:send():179] send: summary
2021-07-14 19:25:15,726 INFO SenderThread:558812 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 19:25:15,727 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 19:25:16,728 INFO Thread-8 :558812 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/wandb-summary.json
2021-07-14 19:25:16,843 DEBUG HandlerThread:558812 [meta.py:__init__():39] meta init
2021-07-14 19:25:16,843 DEBUG HandlerThread:558812 [meta.py:__init__():53] meta init done
2021-07-14 19:25:16,843 DEBUG HandlerThread:558812 [meta.py:probe():210] probe
2021-07-14 19:25:16,844 DEBUG HandlerThread:558812 [meta.py:_setup_git():200] setup git
2021-07-14 19:25:16,874 DEBUG HandlerThread:558812 [meta.py:_setup_git():207] setup git done
2021-07-14 19:25:16,874 DEBUG HandlerThread:558812 [meta.py:_save_pip():57] save pip
2021-07-14 19:25:16,875 DEBUG HandlerThread:558812 [meta.py:_save_pip():71] save pip done
2021-07-14 19:25:16,875 DEBUG HandlerThread:558812 [meta.py:probe():252] probe done
2021-07-14 19:25:16,878 DEBUG SenderThread:558812 [sender.py:send():179] send: files
2021-07-14 19:25:16,878 INFO SenderThread:558812 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 19:25:16,883 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:25:16,884 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:25:17,012 DEBUG SenderThread:558812 [sender.py:send():179] send: config
2021-07-14 19:25:17,013 DEBUG SenderThread:558812 [sender.py:send():179] send: config
2021-07-14 19:25:17,013 DEBUG SenderThread:558812 [sender.py:send():179] send: config
2021-07-14 19:25:17,401 INFO Thread-11 :558812 [upload_job.py:push():137] Uploaded file /tmp/tmp2t3dxezvwandb/1xrc7l9y-wandb-metadata.json
2021-07-14 19:25:17,727 INFO Thread-8 :558812 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/requirements.txt
2021-07-14 19:25:17,727 INFO Thread-8 :558812 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/wandb-metadata.json
2021-07-14 19:25:17,727 INFO Thread-8 :558812 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/output.log
2021-07-14 19:25:32,014 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:25:32,014 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:25:35,734 INFO Thread-8 :558812 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/output.log
2021-07-14 19:25:44,927 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:25:46,738 INFO Thread-8 :558812 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/config.yaml
2021-07-14 19:25:47,145 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:25:47,146 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:26:02,275 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:26:02,276 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:26:14,992 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:26:17,407 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:26:17,407 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:26:32,541 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:26:32,541 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:26:45,071 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:26:47,675 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:26:47,676 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:27:02,817 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:27:02,817 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:27:15,150 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:27:17,954 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:27:17,955 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:27:33,130 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:27:33,130 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:27:45,227 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:27:48,264 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:27:48,264 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:28:03,396 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:28:03,397 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:28:15,303 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:28:18,531 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:28:18,532 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:28:33,662 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:28:33,662 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:28:45,377 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:28:48,794 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:28:48,795 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:29:03,929 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:29:03,929 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:29:15,441 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:29:19,068 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:29:19,068 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:29:34,207 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:29:34,207 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:29:45,507 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:29:49,342 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:29:49,342 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:30:04,484 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:30:04,485 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:30:15,573 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:30:19,618 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:30:19,619 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:30:34,752 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:30:34,753 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:30:45,637 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:30:49,884 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:30:49,885 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:31:05,016 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:31:05,017 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:31:15,702 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:31:20,150 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:31:20,150 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:31:35,285 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:31:35,285 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:31:45,768 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:31:50,417 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:31:50,418 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:32:05,556 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:32:05,557 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:32:15,834 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:32:20,694 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:32:20,695 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:32:35,831 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:32:35,831 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:32:45,903 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:32:50,966 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:32:50,967 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:33:06,100 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:33:06,100 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:33:15,972 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:33:21,242 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:33:21,242 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:33:36,374 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:33:36,374 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:33:46,047 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:33:51,520 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:33:51,520 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:34:06,655 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:34:06,655 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:34:16,121 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:34:21,789 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:34:21,790 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:34:36,924 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:34:36,925 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:34:46,200 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:34:52,186 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:34:52,187 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:35:07,323 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:35:07,323 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:35:16,277 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:35:22,455 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:35:22,455 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:35:37,590 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:35:37,590 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:35:46,355 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:35:52,727 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:35:52,727 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:36:07,858 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:36:07,858 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:36:16,432 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:36:22,989 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:36:22,989 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:36:38,123 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:36:38,124 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:36:46,506 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:36:53,255 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:36:53,255 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:37:08,418 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:37:08,419 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:37:16,581 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:37:23,567 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:37:23,567 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:37:38,702 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:37:38,702 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:37:46,660 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:37:53,835 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:37:53,835 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:38:08,972 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:38:08,972 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:38:16,734 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:38:24,103 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:38:24,103 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:38:39,236 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:38:39,236 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:38:46,811 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:38:54,369 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:38:54,369 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:39:09,502 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:39:09,503 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:39:16,888 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:39:24,635 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:39:24,635 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:39:39,766 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:39:39,766 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:39:46,967 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:39:54,898 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:39:54,899 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:40:10,033 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:40:10,033 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:40:17,045 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:40:25,168 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:40:25,168 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:40:40,300 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:40:40,300 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:40:47,122 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:40:49,097 INFO Thread-8 :558812 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/output.log
2021-07-14 19:40:50,137 DEBUG SenderThread:558812 [sender.py:send():179] send: telemetry
2021-07-14 19:40:50,137 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:50,137 DEBUG SenderThread:558812 [sender.py:send():179] send: exit
2021-07-14 19:40:50,137 INFO SenderThread:558812 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 19:40:50,138 INFO SenderThread:558812 [sender.py:send_exit():295] send defer
2021-07-14 19:40:50,138 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:50,138 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:50,139 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 19:40:50,139 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:50,139 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 19:40:50,139 INFO SenderThread:558812 [sender.py:transition_state():308] send defer: 1
2021-07-14 19:40:50,139 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:50,139 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 19:40:50,231 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:50,231 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 19:40:50,231 INFO SenderThread:558812 [sender.py:transition_state():308] send defer: 2
2021-07-14 19:40:50,231 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:40:50,232 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:50,232 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 19:40:50,232 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:50,233 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 19:40:50,233 INFO SenderThread:558812 [sender.py:transition_state():308] send defer: 3
2021-07-14 19:40:50,233 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:50,233 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 19:40:50,233 DEBUG SenderThread:558812 [sender.py:send():179] send: summary
2021-07-14 19:40:50,234 INFO SenderThread:558812 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 19:40:50,234 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:50,234 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 19:40:50,234 INFO SenderThread:558812 [sender.py:transition_state():308] send defer: 4
2021-07-14 19:40:50,234 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:50,234 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 19:40:50,234 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:50,235 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 19:40:50,240 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:50,432 INFO SenderThread:558812 [sender.py:transition_state():308] send defer: 5
2021-07-14 19:40:50,432 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:50,433 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:50,433 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 19:40:50,433 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:50,433 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 19:40:50,433 INFO SenderThread:558812 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 19:40:50,534 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:51,098 INFO Thread-8 :558812 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/wandb-summary.json
2021-07-14 19:40:51,099 INFO SenderThread:558812 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/output.log
2021-07-14 19:40:51,099 INFO SenderThread:558812 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/config.yaml
2021-07-14 19:40:51,099 INFO SenderThread:558812 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files
2021-07-14 19:40:51,099 INFO SenderThread:558812 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/requirements.txt requirements.txt
2021-07-14 19:40:51,100 INFO SenderThread:558812 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/output.log output.log
2021-07-14 19:40:51,100 INFO SenderThread:558812 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/wandb-metadata.json wandb-metadata.json
2021-07-14 19:40:51,100 INFO SenderThread:558812 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/config.yaml config.yaml
2021-07-14 19:40:51,101 INFO SenderThread:558812 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/wandb-summary.json wandb-summary.json
2021-07-14 19:40:51,101 INFO SenderThread:558812 [sender.py:transition_state():308] send defer: 6
2021-07-14 19:40:51,104 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:51,110 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:51,110 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 19:40:51,113 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:51,114 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 19:40:51,114 INFO SenderThread:558812 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 19:40:51,206 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:51,206 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:51,309 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:51,309 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:51,411 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:51,411 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:51,512 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:51,513 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:51,533 INFO Thread-14 :558812 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/config.yaml
2021-07-14 19:40:51,534 INFO Thread-12 :558812 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/requirements.txt
2021-07-14 19:40:51,535 INFO Thread-15 :558812 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/wandb-summary.json
2021-07-14 19:40:51,537 INFO Thread-13 :558812 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/output.log
2021-07-14 19:40:51,614 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:51,615 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:51,716 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:51,717 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:51,737 INFO Thread-7 :558812 [sender.py:transition_state():308] send defer: 7
2021-07-14 19:40:51,738 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:51,738 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 19:40:51,738 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:51,738 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 19:40:51,818 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:52,030 INFO SenderThread:558812 [sender.py:transition_state():308] send defer: 8
2021-07-14 19:40:52,030 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:52,031 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:52,031 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 19:40:52,031 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:52,031 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 19:40:52,031 INFO SenderThread:558812 [sender.py:transition_state():308] send defer: 9
2021-07-14 19:40:52,032 DEBUG SenderThread:558812 [sender.py:send():179] send: final
2021-07-14 19:40:52,032 DEBUG SenderThread:558812 [sender.py:send():179] send: footer
2021-07-14 19:40:52,032 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:52,032 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 19:40:52,032 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:52,032 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 19:40:52,132 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:52,133 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:52,133 INFO SenderThread:558812 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 19:40:52,134 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 19:40:52,135 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 19:40:52,135 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 19:40:52,135 INFO HandlerThread:558812 [handler.py:finish():638] shutting down handler
2021-07-14 19:40:53,033 INFO WriterThread:558812 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/run-fdn39gsq.wandb
2021-07-14 19:40:53,133 INFO SenderThread:558812 [sender.py:finish():945] shutting down sender
2021-07-14 19:40:53,133 INFO SenderThread:558812 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 19:40:53,133 INFO SenderThread:558812 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 19:40:53,136 INFO MainThread:558812 [internal.py:handle_exit():78] Internal process exited
|