File size: 24,236 Bytes
731244f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 |
2021-07-15 00:52:29,330 INFO MainThread:624595 [internal.py:wandb_internal():88] W&B internal server running at pid: 624595, started at: 2021-07-15 00:52:29.329937
2021-07-15 00:52:29,332 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: check_version
2021-07-15 00:52:29,333 INFO WriterThread:624595 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/run-361cvuow.wandb
2021-07-15 00:52:29,333 DEBUG SenderThread:624595 [sender.py:send():179] send: header
2021-07-15 00:52:29,333 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: check_version
2021-07-15 00:52:29,368 DEBUG SenderThread:624595 [sender.py:send():179] send: run
2021-07-15 00:52:29,550 INFO SenderThread:624595 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files
2021-07-15 00:52:29,550 INFO SenderThread:624595 [sender.py:_start_run_threads():716] run started: 361cvuow with start time 1626310348
2021-07-15 00:52:29,550 DEBUG SenderThread:624595 [sender.py:send():179] send: summary
2021-07-15 00:52:29,551 INFO SenderThread:624595 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 00:52:29,551 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: run_start
2021-07-15 00:52:30,553 INFO Thread-8 :624595 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/wandb-summary.json
2021-07-15 00:52:30,640 DEBUG HandlerThread:624595 [meta.py:__init__():39] meta init
2021-07-15 00:52:30,640 DEBUG HandlerThread:624595 [meta.py:__init__():53] meta init done
2021-07-15 00:52:30,640 DEBUG HandlerThread:624595 [meta.py:probe():210] probe
2021-07-15 00:52:30,641 DEBUG HandlerThread:624595 [meta.py:_setup_git():200] setup git
2021-07-15 00:52:30,670 DEBUG HandlerThread:624595 [meta.py:_setup_git():207] setup git done
2021-07-15 00:52:30,670 DEBUG HandlerThread:624595 [meta.py:_save_pip():57] save pip
2021-07-15 00:52:30,671 DEBUG HandlerThread:624595 [meta.py:_save_pip():71] save pip done
2021-07-15 00:52:30,671 DEBUG HandlerThread:624595 [meta.py:probe():252] probe done
2021-07-15 00:52:30,674 DEBUG SenderThread:624595 [sender.py:send():179] send: files
2021-07-15 00:52:30,674 INFO SenderThread:624595 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-15 00:52:30,680 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:52:30,680 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:52:30,812 DEBUG SenderThread:624595 [sender.py:send():179] send: config
2021-07-15 00:52:30,812 DEBUG SenderThread:624595 [sender.py:send():179] send: config
2021-07-15 00:52:30,813 DEBUG SenderThread:624595 [sender.py:send():179] send: config
2021-07-15 00:52:31,126 INFO Thread-11 :624595 [upload_job.py:push():137] Uploaded file /tmp/tmp7sk_pd_0wandb/2r26ralk-wandb-metadata.json
2021-07-15 00:52:31,551 INFO Thread-8 :624595 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/wandb-metadata.json
2021-07-15 00:52:31,552 INFO Thread-8 :624595 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/requirements.txt
2021-07-15 00:52:31,552 INFO Thread-8 :624595 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log
2021-07-15 00:52:45,557 INFO Thread-8 :624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log
2021-07-15 00:52:45,814 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:52:45,814 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:52:58,723 DEBUG SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:53:00,564 INFO Thread-8 :624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/config.yaml
2021-07-15 00:53:00,947 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:53:00,947 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:53:16,080 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:53:16,080 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:53:28,803 DEBUG SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:53:31,216 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:53:31,216 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:53:46,347 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:53:46,347 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:53:58,879 DEBUG SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:54:01,496 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:54:01,496 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:54:16,630 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:54:16,630 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:54:28,953 DEBUG SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:54:31,760 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:54:31,761 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:54:46,892 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:54:46,893 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:54:59,029 DEBUG SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:55:02,026 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:55:02,027 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:55:17,159 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:55:17,160 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:55:29,107 DEBUG SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:55:32,292 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:55:32,292 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:55:47,422 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:55:47,422 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:55:59,186 DEBUG SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:56:02,555 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:56:02,556 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:56:17,688 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:56:17,688 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:56:29,261 DEBUG SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:56:32,824 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:56:32,824 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:56:47,955 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:56:47,955 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:56:59,339 DEBUG SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:57:03,086 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:57:03,086 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:57:18,219 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:57:18,219 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:57:29,416 DEBUG SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:57:33,350 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:57:33,350 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:57:48,482 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:57:48,483 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:57:59,490 DEBUG SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:58:03,614 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:58:03,614 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:58:17,674 INFO Thread-8 :624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log
2021-07-15 00:58:18,763 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:58:18,763 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:58:29,567 DEBUG SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:58:33,909 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:58:33,910 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:58:49,058 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:58:49,059 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:58:59,648 DEBUG SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:59:04,188 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:59:04,188 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:59:19,321 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:59:19,322 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:59:29,729 DEBUG SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:59:34,455 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:59:34,455 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:59:41,706 INFO Thread-8 :624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log
2021-07-15 00:59:43,394 DEBUG SenderThread:624595 [sender.py:send():179] send: telemetry
2021-07-15 00:59:43,394 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:43,395 DEBUG SenderThread:624595 [sender.py:send():179] send: exit
2021-07-15 00:59:43,395 INFO SenderThread:624595 [sender.py:send_exit():287] handling exit code: 1
2021-07-15 00:59:43,395 INFO SenderThread:624595 [sender.py:send_exit():295] send defer
2021-07-15 00:59:43,395 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:43,396 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,396 INFO HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-15 00:59:43,396 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,396 INFO SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-15 00:59:43,396 INFO SenderThread:624595 [sender.py:transition_state():308] send defer: 1
2021-07-15 00:59:43,396 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,396 INFO HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-15 00:59:43,467 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,467 INFO SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-15 00:59:43,467 INFO SenderThread:624595 [sender.py:transition_state():308] send defer: 2
2021-07-15 00:59:43,467 DEBUG SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:59:43,467 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,467 INFO HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-15 00:59:43,468 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,468 INFO SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-15 00:59:43,468 INFO SenderThread:624595 [sender.py:transition_state():308] send defer: 3
2021-07-15 00:59:43,468 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,468 INFO HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-15 00:59:43,468 DEBUG SenderThread:624595 [sender.py:send():179] send: summary
2021-07-15 00:59:43,468 INFO SenderThread:624595 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 00:59:43,468 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,469 INFO SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-15 00:59:43,469 INFO SenderThread:624595 [sender.py:transition_state():308] send defer: 4
2021-07-15 00:59:43,469 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,469 INFO HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-15 00:59:43,469 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,469 INFO SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-15 00:59:43,498 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:43,675 INFO SenderThread:624595 [sender.py:transition_state():308] send defer: 5
2021-07-15 00:59:43,675 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:43,676 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,676 INFO HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-15 00:59:43,676 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,676 INFO SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-15 00:59:43,676 INFO SenderThread:624595 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-15 00:59:43,707 INFO SenderThread:624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/config.yaml
2021-07-15 00:59:43,708 INFO SenderThread:624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log
2021-07-15 00:59:43,708 INFO SenderThread:624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/wandb-summary.json
2021-07-15 00:59:43,708 INFO SenderThread:624595 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files
2021-07-15 00:59:43,708 INFO SenderThread:624595 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/requirements.txt requirements.txt
2021-07-15 00:59:43,708 INFO SenderThread:624595 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log output.log
2021-07-15 00:59:43,708 INFO SenderThread:624595 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/wandb-metadata.json wandb-metadata.json
2021-07-15 00:59:43,708 INFO SenderThread:624595 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/config.yaml config.yaml
2021-07-15 00:59:43,708 INFO SenderThread:624595 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/wandb-summary.json wandb-summary.json
2021-07-15 00:59:43,709 INFO SenderThread:624595 [sender.py:transition_state():308] send defer: 6
2021-07-15 00:59:43,709 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,715 INFO HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-15 00:59:43,720 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,720 INFO SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-15 00:59:43,721 INFO SenderThread:624595 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 00:59:43,778 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:43,778 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:43,880 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:43,880 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:43,982 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:43,982 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,084 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:44,084 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,146 INFO Thread-13 :624595 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log
2021-07-15 00:59:44,161 INFO Thread-15 :624595 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/wandb-summary.json
2021-07-15 00:59:44,186 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:44,187 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,194 INFO Thread-12 :624595 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/requirements.txt
2021-07-15 00:59:44,198 INFO Thread-14 :624595 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/config.yaml
2021-07-15 00:59:44,289 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:44,289 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,391 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:44,391 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,399 INFO Thread-7 :624595 [sender.py:transition_state():308] send defer: 7
2021-07-15 00:59:44,399 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:44,399 INFO HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-15 00:59:44,399 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:44,400 INFO SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-15 00:59:44,493 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:44,686 INFO SenderThread:624595 [sender.py:transition_state():308] send defer: 8
2021-07-15 00:59:44,687 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,687 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:44,687 INFO HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-15 00:59:44,687 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:44,688 INFO SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-15 00:59:44,688 INFO SenderThread:624595 [sender.py:transition_state():308] send defer: 9
2021-07-15 00:59:44,688 DEBUG SenderThread:624595 [sender.py:send():179] send: final
2021-07-15 00:59:44,688 DEBUG SenderThread:624595 [sender.py:send():179] send: footer
2021-07-15 00:59:44,688 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:44,689 INFO HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-15 00:59:44,689 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:44,689 INFO SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-15 00:59:44,789 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:44,789 DEBUG SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,789 INFO SenderThread:624595 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 00:59:44,791 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: get_summary
2021-07-15 00:59:44,791 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-15 00:59:44,792 DEBUG HandlerThread:624595 [handler.py:handle_request():124] handle_request: shutdown
2021-07-15 00:59:44,792 INFO HandlerThread:624595 [handler.py:finish():638] shutting down handler
2021-07-15 00:59:45,689 INFO WriterThread:624595 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/run-361cvuow.wandb
2021-07-15 00:59:45,789 INFO SenderThread:624595 [sender.py:finish():945] shutting down sender
2021-07-15 00:59:45,790 INFO SenderThread:624595 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 00:59:45,790 INFO SenderThread:624595 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 00:59:45,792 INFO MainThread:624595 [internal.py:handle_exit():78] Internal process exited
|