File size: 23,414 Bytes
731244f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 |
2021-07-15 01:00:39,018 INFO MainThread:626669 [internal.py:wandb_internal():88] W&B internal server running at pid: 626669, started at: 2021-07-15 01:00:39.017812
2021-07-15 01:00:39,020 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: check_version
2021-07-15 01:00:39,020 INFO WriterThread:626669 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/run-2gdygcu4.wandb
2021-07-15 01:00:39,021 DEBUG SenderThread:626669 [sender.py:send():179] send: header
2021-07-15 01:00:39,021 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: check_version
2021-07-15 01:00:39,055 DEBUG SenderThread:626669 [sender.py:send():179] send: run
2021-07-15 01:00:39,219 INFO SenderThread:626669 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files
2021-07-15 01:00:39,219 INFO SenderThread:626669 [sender.py:_start_run_threads():716] run started: 2gdygcu4 with start time 1626310838
2021-07-15 01:00:39,220 DEBUG SenderThread:626669 [sender.py:send():179] send: summary
2021-07-15 01:00:39,220 INFO SenderThread:626669 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 01:00:39,220 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: run_start
2021-07-15 01:00:40,222 INFO Thread-8 :626669 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/wandb-summary.json
2021-07-15 01:00:40,314 DEBUG HandlerThread:626669 [meta.py:__init__():39] meta init
2021-07-15 01:00:40,314 DEBUG HandlerThread:626669 [meta.py:__init__():53] meta init done
2021-07-15 01:00:40,314 DEBUG HandlerThread:626669 [meta.py:probe():210] probe
2021-07-15 01:00:40,316 DEBUG HandlerThread:626669 [meta.py:_setup_git():200] setup git
2021-07-15 01:00:40,345 DEBUG HandlerThread:626669 [meta.py:_setup_git():207] setup git done
2021-07-15 01:00:40,345 DEBUG HandlerThread:626669 [meta.py:_save_pip():57] save pip
2021-07-15 01:00:40,346 DEBUG HandlerThread:626669 [meta.py:_save_pip():71] save pip done
2021-07-15 01:00:40,346 DEBUG HandlerThread:626669 [meta.py:probe():252] probe done
2021-07-15 01:00:40,349 DEBUG SenderThread:626669 [sender.py:send():179] send: files
2021-07-15 01:00:40,349 INFO SenderThread:626669 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-15 01:00:40,354 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:00:40,355 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:00:40,481 DEBUG SenderThread:626669 [sender.py:send():179] send: config
2021-07-15 01:00:40,482 DEBUG SenderThread:626669 [sender.py:send():179] send: config
2021-07-15 01:00:40,482 DEBUG SenderThread:626669 [sender.py:send():179] send: config
2021-07-15 01:00:40,774 INFO Thread-11 :626669 [upload_job.py:push():137] Uploaded file /tmp/tmpwx_o6qsuwandb/316agsp6-wandb-metadata.json
2021-07-15 01:00:41,221 INFO Thread-8 :626669 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/requirements.txt
2021-07-15 01:00:41,221 INFO Thread-8 :626669 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/wandb-metadata.json
2021-07-15 01:00:41,221 INFO Thread-8 :626669 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/output.log
2021-07-15 01:00:55,227 INFO Thread-8 :626669 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/output.log
2021-07-15 01:00:55,483 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:00:55,484 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:01:08,399 DEBUG SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:01:10,233 INFO Thread-8 :626669 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/config.yaml
2021-07-15 01:01:10,612 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:01:10,613 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:01:25,743 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:01:25,744 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:01:38,482 DEBUG SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:01:40,875 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:01:40,875 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:01:56,006 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:01:56,006 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:02:08,558 DEBUG SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:02:11,138 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:02:11,138 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:02:26,268 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:02:26,268 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:02:38,636 DEBUG SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:02:41,402 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:02:41,402 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:02:56,534 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:02:56,534 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:03:08,712 DEBUG SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:03:11,667 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:03:11,668 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:03:26,799 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:03:26,800 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:03:38,780 DEBUG SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:03:41,932 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:03:41,932 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:03:57,065 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:03:57,065 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:04:08,851 DEBUG SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:04:12,197 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:04:12,197 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:04:27,326 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:04:27,327 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:04:38,928 DEBUG SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:04:42,462 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:04:42,463 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:04:57,592 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:04:57,593 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:05:08,987 DEBUG SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:05:12,723 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:05:12,723 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:05:27,851 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:05:27,852 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:05:39,058 DEBUG SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:05:39,359 INFO Thread-8 :626669 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/output.log
2021-07-15 01:05:43,032 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:05:43,032 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:05:58,189 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:05:58,190 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:06:09,137 DEBUG SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:06:13,331 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:06:13,331 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:06:28,461 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:06:28,461 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:06:39,220 DEBUG SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:06:43,596 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:06:43,596 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:06:58,728 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:06:58,728 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:07:05,299 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:05,299 DEBUG SenderThread:626669 [sender.py:send():179] send: telemetry
2021-07-15 01:07:05,299 DEBUG SenderThread:626669 [sender.py:send():179] send: exit
2021-07-15 01:07:05,299 INFO SenderThread:626669 [sender.py:send_exit():287] handling exit code: 1
2021-07-15 01:07:05,299 INFO SenderThread:626669 [sender.py:send_exit():295] send defer
2021-07-15 01:07:05,299 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:05,300 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:05,300 INFO HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-15 01:07:05,300 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:05,300 INFO SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-15 01:07:05,300 INFO SenderThread:626669 [sender.py:transition_state():308] send defer: 1
2021-07-15 01:07:05,300 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:05,301 INFO HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-15 01:07:05,385 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:05,385 INFO SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-15 01:07:05,385 INFO SenderThread:626669 [sender.py:transition_state():308] send defer: 2
2021-07-15 01:07:05,385 DEBUG SenderThread:626669 [sender.py:send():179] send: stats
2021-07-15 01:07:05,386 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:05,386 INFO HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-15 01:07:05,386 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:05,386 INFO SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-15 01:07:05,386 INFO SenderThread:626669 [sender.py:transition_state():308] send defer: 3
2021-07-15 01:07:05,386 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:05,386 INFO HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-15 01:07:05,386 DEBUG SenderThread:626669 [sender.py:send():179] send: summary
2021-07-15 01:07:05,387 INFO SenderThread:626669 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 01:07:05,387 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:05,387 INFO SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-15 01:07:05,387 INFO SenderThread:626669 [sender.py:transition_state():308] send defer: 4
2021-07-15 01:07:05,387 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:05,387 INFO HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-15 01:07:05,387 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:05,387 INFO SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-15 01:07:05,393 INFO Thread-8 :626669 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/wandb-summary.json
2021-07-15 01:07:05,393 INFO Thread-8 :626669 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/output.log
2021-07-15 01:07:05,401 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:05,571 INFO SenderThread:626669 [sender.py:transition_state():308] send defer: 5
2021-07-15 01:07:05,571 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:05,572 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:05,572 INFO HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-15 01:07:05,572 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:05,572 INFO SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-15 01:07:05,572 INFO SenderThread:626669 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-15 01:07:05,673 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:06,394 INFO Thread-8 :626669 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/config.yaml
2021-07-15 01:07:06,394 INFO SenderThread:626669 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files
2021-07-15 01:07:06,395 INFO SenderThread:626669 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/requirements.txt requirements.txt
2021-07-15 01:07:06,395 INFO SenderThread:626669 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/output.log output.log
2021-07-15 01:07:06,395 INFO SenderThread:626669 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/wandb-metadata.json wandb-metadata.json
2021-07-15 01:07:06,395 INFO SenderThread:626669 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/config.yaml config.yaml
2021-07-15 01:07:06,395 INFO SenderThread:626669 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/wandb-summary.json wandb-summary.json
2021-07-15 01:07:06,395 INFO SenderThread:626669 [sender.py:transition_state():308] send defer: 6
2021-07-15 01:07:06,396 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:06,399 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:06,399 INFO HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-15 01:07:06,399 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:06,400 INFO SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-15 01:07:06,400 INFO SenderThread:626669 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 01:07:06,501 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:06,501 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:06,603 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:06,604 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:06,706 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:06,706 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:06,808 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:06,808 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:06,857 INFO Thread-12 :626669 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/requirements.txt
2021-07-15 01:07:06,884 INFO Thread-15 :626669 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/wandb-summary.json
2021-07-15 01:07:06,910 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:06,910 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:06,920 INFO Thread-14 :626669 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/config.yaml
2021-07-15 01:07:06,968 INFO Thread-13 :626669 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/files/output.log
2021-07-15 01:07:07,012 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:07,012 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:07,115 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:07,115 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:07,169 INFO Thread-7 :626669 [sender.py:transition_state():308] send defer: 7
2021-07-15 01:07:07,169 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:07,169 INFO HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-15 01:07:07,170 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:07,170 INFO SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-15 01:07:07,217 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:07,450 INFO SenderThread:626669 [sender.py:transition_state():308] send defer: 8
2021-07-15 01:07:07,450 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:07,451 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:07,451 INFO HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-15 01:07:07,451 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:07,451 INFO SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-15 01:07:07,451 INFO SenderThread:626669 [sender.py:transition_state():308] send defer: 9
2021-07-15 01:07:07,452 DEBUG SenderThread:626669 [sender.py:send():179] send: final
2021-07-15 01:07:07,452 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:07:07,452 INFO HandlerThread:626669 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-15 01:07:07,452 DEBUG SenderThread:626669 [sender.py:send():179] send: footer
2021-07-15 01:07:07,452 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: defer
2021-07-15 01:07:07,452 INFO SenderThread:626669 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-15 01:07:07,552 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:07:07,552 DEBUG SenderThread:626669 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:07:07,553 INFO SenderThread:626669 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 01:07:07,554 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: get_summary
2021-07-15 01:07:07,555 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-15 01:07:07,555 DEBUG HandlerThread:626669 [handler.py:handle_request():124] handle_request: shutdown
2021-07-15 01:07:07,555 INFO HandlerThread:626669 [handler.py:finish():638] shutting down handler
2021-07-15 01:07:08,453 INFO WriterThread:626669 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210715_010038-2gdygcu4/run-2gdygcu4.wandb
2021-07-15 01:07:08,553 INFO SenderThread:626669 [sender.py:finish():945] shutting down sender
2021-07-15 01:07:08,553 INFO SenderThread:626669 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 01:07:08,553 INFO SenderThread:626669 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 01:07:08,555 INFO MainThread:626669 [internal.py:handle_exit():78] Internal process exited
|