File size: 32,630 Bytes
f291f93 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 |
2021-07-14 18:55:37,982 INFO MainThread:555087 [internal.py:wandb_internal():88] W&B internal server running at pid: 555087, started at: 2021-07-14 18:55:37.982262
2021-07-14 18:55:37,984 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 18:55:37,985 INFO WriterThread:555087 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/run-379h21h2.wandb
2021-07-14 18:55:37,986 DEBUG SenderThread:555087 [sender.py:send():179] send: header
2021-07-14 18:55:37,986 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: check_version
2021-07-14 18:55:38,025 DEBUG SenderThread:555087 [sender.py:send():179] send: run
2021-07-14 18:55:38,206 INFO SenderThread:555087 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files
2021-07-14 18:55:38,206 INFO SenderThread:555087 [sender.py:_start_run_threads():716] run started: 379h21h2 with start time 1626288937
2021-07-14 18:55:38,206 DEBUG SenderThread:555087 [sender.py:send():179] send: summary
2021-07-14 18:55:38,206 INFO SenderThread:555087 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 18:55:38,207 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 18:55:39,207 INFO Thread-8 :555087 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/wandb-summary.json
2021-07-14 18:55:39,338 DEBUG HandlerThread:555087 [meta.py:__init__():39] meta init
2021-07-14 18:55:39,338 DEBUG HandlerThread:555087 [meta.py:__init__():53] meta init done
2021-07-14 18:55:39,338 DEBUG HandlerThread:555087 [meta.py:probe():210] probe
2021-07-14 18:55:39,340 DEBUG HandlerThread:555087 [meta.py:_setup_git():200] setup git
2021-07-14 18:55:39,371 DEBUG HandlerThread:555087 [meta.py:_setup_git():207] setup git done
2021-07-14 18:55:39,371 DEBUG HandlerThread:555087 [meta.py:_save_pip():57] save pip
2021-07-14 18:55:39,372 DEBUG HandlerThread:555087 [meta.py:_save_pip():71] save pip done
2021-07-14 18:55:39,372 DEBUG HandlerThread:555087 [meta.py:probe():252] probe done
2021-07-14 18:55:39,397 DEBUG SenderThread:555087 [sender.py:send():179] send: files
2021-07-14 18:55:39,397 INFO SenderThread:555087 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 18:55:39,404 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:55:39,405 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:55:39,533 DEBUG SenderThread:555087 [sender.py:send():179] send: config
2021-07-14 18:55:39,534 DEBUG SenderThread:555087 [sender.py:send():179] send: config
2021-07-14 18:55:39,534 DEBUG SenderThread:555087 [sender.py:send():179] send: config
2021-07-14 18:55:39,858 INFO Thread-11 :555087 [upload_job.py:push():137] Uploaded file /tmp/tmpywmfy6u4wandb/302tickz-wandb-metadata.json
2021-07-14 18:55:40,207 INFO Thread-8 :555087 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/output.log
2021-07-14 18:55:40,207 INFO Thread-8 :555087 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/wandb-metadata.json
2021-07-14 18:55:40,207 INFO Thread-8 :555087 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/requirements.txt
2021-07-14 18:55:54,543 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:55:54,543 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:55:58,216 INFO Thread-8 :555087 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/output.log
2021-07-14 18:56:07,426 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 18:56:09,220 INFO Thread-8 :555087 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/config.yaml
2021-07-14 18:56:09,675 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:56:09,675 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:56:24,806 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:56:24,806 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:56:37,502 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 18:56:39,942 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:56:39,942 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:56:55,073 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:56:55,073 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:57:07,579 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 18:57:10,210 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:57:10,210 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:57:25,342 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:57:25,343 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:57:37,658 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 18:57:40,474 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:57:40,475 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:57:55,605 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:57:55,606 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:58:07,732 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 18:58:10,737 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:58:10,737 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:58:25,870 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:58:25,870 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:58:37,810 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 18:58:41,005 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:58:41,005 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:58:56,137 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:58:56,138 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:59:07,886 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 18:59:11,274 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:59:11,274 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:59:26,406 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:59:26,406 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:59:37,963 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 18:59:41,543 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:59:41,543 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:59:56,673 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:59:56,674 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:00:08,040 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:00:11,805 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:00:11,806 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:00:26,938 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:00:26,938 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:00:38,115 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:00:42,071 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:00:42,073 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:00:57,206 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:00:57,206 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:01:08,189 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:01:12,338 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:01:12,338 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:01:27,470 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:01:27,470 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:01:38,253 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:01:42,602 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:01:42,602 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:01:57,736 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:01:57,736 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:02:08,323 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:02:12,869 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:02:12,869 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:02:28,002 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:02:28,002 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:02:38,399 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:02:43,167 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:02:43,167 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:02:58,298 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:02:58,299 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:03:08,473 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:03:13,435 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:03:13,435 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:03:28,570 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:03:28,570 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:03:38,550 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:03:43,702 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:03:43,702 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:03:58,838 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:03:58,838 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:04:08,624 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:04:13,976 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:04:13,976 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:04:29,120 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:04:29,121 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:04:38,701 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:04:44,257 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:04:44,258 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:04:59,397 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:04:59,397 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:05:08,777 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:05:14,531 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:05:14,532 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:05:29,687 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:05:29,688 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:05:38,851 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:05:44,819 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:05:44,820 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:05:59,950 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:05:59,951 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:06:08,925 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:06:15,090 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:06:15,091 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:06:30,226 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:06:30,226 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:06:39,001 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:06:45,360 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:06:45,360 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:07:00,496 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:07:00,496 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:07:09,079 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:07:15,632 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:07:15,633 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:07:30,767 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:07:30,767 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:07:39,157 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:07:45,903 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:07:45,903 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:08:01,041 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:08:01,041 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:08:09,235 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:08:16,178 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:08:16,179 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:08:31,336 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:08:31,336 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:08:39,312 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:08:46,469 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:08:46,470 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:09:01,600 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:09:01,601 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:09:09,391 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:09:16,736 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:09:16,736 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:09:31,869 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:09:31,870 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:09:39,454 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:09:47,008 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:09:47,008 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:10:02,139 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:10:02,140 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:10:09,532 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:10:17,277 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:10:17,277 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:10:32,410 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:10:32,410 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:10:39,609 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:10:47,545 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:10:47,545 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:11:01,425 DEBUG SenderThread:555087 [sender.py:send():179] send: telemetry
2021-07-14 19:11:01,425 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:01,425 DEBUG SenderThread:555087 [sender.py:send():179] send: exit
2021-07-14 19:11:01,426 INFO SenderThread:555087 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 19:11:01,426 INFO SenderThread:555087 [sender.py:send_exit():295] send defer
2021-07-14 19:11:01,427 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:01,427 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:01,427 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 19:11:01,428 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:01,428 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 19:11:01,428 INFO SenderThread:555087 [sender.py:transition_state():308] send defer: 1
2021-07-14 19:11:01,428 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:01,428 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 19:11:01,460 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:01,460 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 19:11:01,460 INFO SenderThread:555087 [sender.py:transition_state():308] send defer: 2
2021-07-14 19:11:01,460 DEBUG SenderThread:555087 [sender.py:send():179] send: stats
2021-07-14 19:11:01,461 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:01,461 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 19:11:01,461 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:01,461 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 19:11:01,461 INFO SenderThread:555087 [sender.py:transition_state():308] send defer: 3
2021-07-14 19:11:01,461 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:01,462 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 19:11:01,462 DEBUG SenderThread:555087 [sender.py:send():179] send: summary
2021-07-14 19:11:01,462 INFO SenderThread:555087 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 19:11:01,462 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:01,462 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 19:11:01,462 INFO SenderThread:555087 [sender.py:transition_state():308] send defer: 4
2021-07-14 19:11:01,463 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:01,463 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 19:11:01,463 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:01,463 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 19:11:01,529 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:01,566 INFO Thread-8 :555087 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/output.log
2021-07-14 19:11:01,566 INFO Thread-8 :555087 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/wandb-summary.json
2021-07-14 19:11:01,667 INFO SenderThread:555087 [sender.py:transition_state():308] send defer: 5
2021-07-14 19:11:01,667 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:01,668 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:01,668 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 19:11:01,668 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:01,668 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 19:11:01,668 INFO SenderThread:555087 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 19:11:01,769 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:02,566 INFO Thread-8 :555087 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/config.yaml
2021-07-14 19:11:02,567 INFO SenderThread:555087 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files
2021-07-14 19:11:02,567 INFO SenderThread:555087 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/requirements.txt requirements.txt
2021-07-14 19:11:02,567 INFO SenderThread:555087 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/output.log output.log
2021-07-14 19:11:02,567 INFO SenderThread:555087 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/wandb-metadata.json wandb-metadata.json
2021-07-14 19:11:02,567 INFO SenderThread:555087 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/config.yaml config.yaml
2021-07-14 19:11:02,568 INFO SenderThread:555087 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/wandb-summary.json wandb-summary.json
2021-07-14 19:11:02,568 INFO SenderThread:555087 [sender.py:transition_state():308] send defer: 6
2021-07-14 19:11:02,569 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:02,572 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:02,572 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 19:11:02,575 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:02,575 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 19:11:02,575 INFO SenderThread:555087 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 19:11:02,671 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:02,671 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:02,773 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:02,774 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:02,876 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:02,876 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:02,977 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:02,978 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:03,031 INFO Thread-14 :555087 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/config.yaml
2021-07-14 19:11:03,066 INFO Thread-13 :555087 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/output.log
2021-07-14 19:11:03,074 INFO Thread-12 :555087 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/requirements.txt
2021-07-14 19:11:03,079 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:03,079 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:03,088 INFO Thread-15 :555087 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/files/wandb-summary.json
2021-07-14 19:11:03,181 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:03,181 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:03,283 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:03,283 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:03,288 INFO Thread-7 :555087 [sender.py:transition_state():308] send defer: 7
2021-07-14 19:11:03,288 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:03,289 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 19:11:03,289 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:03,289 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 19:11:03,385 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:03,599 INFO SenderThread:555087 [sender.py:transition_state():308] send defer: 8
2021-07-14 19:11:03,600 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:03,600 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:03,600 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 19:11:03,600 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:03,600 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 19:11:03,601 INFO SenderThread:555087 [sender.py:transition_state():308] send defer: 9
2021-07-14 19:11:03,601 DEBUG SenderThread:555087 [sender.py:send():179] send: final
2021-07-14 19:11:03,601 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:11:03,601 INFO HandlerThread:555087 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 19:11:03,601 DEBUG SenderThread:555087 [sender.py:send():179] send: footer
2021-07-14 19:11:03,602 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: defer
2021-07-14 19:11:03,602 INFO SenderThread:555087 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 19:11:03,702 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:11:03,702 DEBUG SenderThread:555087 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:11:03,702 INFO SenderThread:555087 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 19:11:03,704 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 19:11:03,704 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 19:11:03,705 DEBUG HandlerThread:555087 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 19:11:03,705 INFO HandlerThread:555087 [handler.py:finish():638] shutting down handler
2021-07-14 19:11:04,602 INFO WriterThread:555087 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_185537-379h21h2/run-379h21h2.wandb
2021-07-14 19:11:04,703 INFO SenderThread:555087 [sender.py:finish():945] shutting down sender
2021-07-14 19:11:04,703 INFO SenderThread:555087 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 19:11:04,703 INFO SenderThread:555087 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 19:11:04,706 INFO MainThread:555087 [internal.py:handle_exit():78] Internal process exited
|