File size: 22,405 Bytes
731244f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 |
2021-07-15 01:08:38,704 INFO MainThread:628228 [internal.py:wandb_internal():88] W&B internal server running at pid: 628228, started at: 2021-07-15 01:08:38.703926
2021-07-15 01:08:38,706 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: check_version
2021-07-15 01:08:38,706 INFO WriterThread:628228 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/run-hepp3r83.wandb
2021-07-15 01:08:38,707 DEBUG SenderThread:628228 [sender.py:send():179] send: header
2021-07-15 01:08:38,707 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: check_version
2021-07-15 01:08:38,744 DEBUG SenderThread:628228 [sender.py:send():179] send: run
2021-07-15 01:08:38,910 INFO SenderThread:628228 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files
2021-07-15 01:08:38,910 INFO SenderThread:628228 [sender.py:_start_run_threads():716] run started: hepp3r83 with start time 1626311318
2021-07-15 01:08:38,911 DEBUG SenderThread:628228 [sender.py:send():179] send: summary
2021-07-15 01:08:38,911 INFO SenderThread:628228 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 01:08:38,911 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: run_start
2021-07-15 01:08:39,914 INFO Thread-8 :628228 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/wandb-summary.json
2021-07-15 01:08:40,001 DEBUG HandlerThread:628228 [meta.py:__init__():39] meta init
2021-07-15 01:08:40,001 DEBUG HandlerThread:628228 [meta.py:__init__():53] meta init done
2021-07-15 01:08:40,001 DEBUG HandlerThread:628228 [meta.py:probe():210] probe
2021-07-15 01:08:40,002 DEBUG HandlerThread:628228 [meta.py:_setup_git():200] setup git
2021-07-15 01:08:40,031 DEBUG HandlerThread:628228 [meta.py:_setup_git():207] setup git done
2021-07-15 01:08:40,032 DEBUG HandlerThread:628228 [meta.py:_save_pip():57] save pip
2021-07-15 01:08:40,032 DEBUG HandlerThread:628228 [meta.py:_save_pip():71] save pip done
2021-07-15 01:08:40,032 DEBUG HandlerThread:628228 [meta.py:probe():252] probe done
2021-07-15 01:08:40,035 DEBUG SenderThread:628228 [sender.py:send():179] send: files
2021-07-15 01:08:40,035 INFO SenderThread:628228 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-15 01:08:40,041 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:08:40,041 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:08:40,174 DEBUG SenderThread:628228 [sender.py:send():179] send: config
2021-07-15 01:08:40,174 DEBUG SenderThread:628228 [sender.py:send():179] send: config
2021-07-15 01:08:40,175 DEBUG SenderThread:628228 [sender.py:send():179] send: config
2021-07-15 01:08:40,469 INFO Thread-11 :628228 [upload_job.py:push():137] Uploaded file /tmp/tmpdiv10f4rwandb/1ladryaq-wandb-metadata.json
2021-07-15 01:08:40,913 INFO Thread-8 :628228 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/output.log
2021-07-15 01:08:40,913 INFO Thread-8 :628228 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/wandb-metadata.json
2021-07-15 01:08:40,913 INFO Thread-8 :628228 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/requirements.txt
2021-07-15 01:08:54,918 INFO Thread-8 :628228 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/output.log
2021-07-15 01:08:55,176 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:08:55,177 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:09:08,085 DEBUG SenderThread:628228 [sender.py:send():179] send: stats
2021-07-15 01:09:09,924 INFO Thread-8 :628228 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/config.yaml
2021-07-15 01:09:10,307 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:09:10,308 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:09:25,440 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:09:25,440 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:09:38,165 DEBUG SenderThread:628228 [sender.py:send():179] send: stats
2021-07-15 01:09:40,570 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:09:40,571 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:09:55,703 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:09:55,703 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:10:08,241 DEBUG SenderThread:628228 [sender.py:send():179] send: stats
2021-07-15 01:10:10,835 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:10:10,836 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:10:25,968 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:10:25,968 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:10:38,315 DEBUG SenderThread:628228 [sender.py:send():179] send: stats
2021-07-15 01:10:41,102 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:10:41,102 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:10:56,235 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:10:56,235 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:11:08,383 DEBUG SenderThread:628228 [sender.py:send():179] send: stats
2021-07-15 01:11:11,368 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:11:11,368 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:11:26,499 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:11:26,500 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:11:38,459 DEBUG SenderThread:628228 [sender.py:send():179] send: stats
2021-07-15 01:11:41,634 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:11:41,634 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:11:56,768 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:11:56,769 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:12:08,533 DEBUG SenderThread:628228 [sender.py:send():179] send: stats
2021-07-15 01:12:11,900 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:12:11,901 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:12:27,032 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:12:27,032 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:12:38,609 DEBUG SenderThread:628228 [sender.py:send():179] send: stats
2021-07-15 01:12:42,164 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:12:42,165 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:12:47,008 INFO Thread-8 :628228 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/output.log
2021-07-15 01:12:57,312 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:12:57,312 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:13:08,688 DEBUG SenderThread:628228 [sender.py:send():179] send: stats
2021-07-15 01:13:12,457 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:13:12,457 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:13:27,589 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:13:27,589 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:13:38,769 DEBUG SenderThread:628228 [sender.py:send():179] send: stats
2021-07-15 01:13:42,720 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:13:42,721 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:13:57,852 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 01:13:57,852 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: stop_status
2021-07-15 01:14:08,848 DEBUG SenderThread:628228 [sender.py:send():179] send: stats
2021-07-15 01:14:13,043 INFO Thread-8 :628228 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/output.log
2021-07-15 01:14:13,314 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:14:13,314 DEBUG SenderThread:628228 [sender.py:send():179] send: telemetry
2021-07-15 01:14:13,315 DEBUG SenderThread:628228 [sender.py:send():179] send: exit
2021-07-15 01:14:13,315 INFO SenderThread:628228 [sender.py:send_exit():287] handling exit code: 1
2021-07-15 01:14:13,315 INFO SenderThread:628228 [sender.py:send_exit():295] send defer
2021-07-15 01:14:13,315 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:14:13,316 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:14:13,316 INFO HandlerThread:628228 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-15 01:14:13,316 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: defer
2021-07-15 01:14:13,316 INFO SenderThread:628228 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-15 01:14:13,316 INFO SenderThread:628228 [sender.py:transition_state():308] send defer: 1
2021-07-15 01:14:13,316 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:14:13,316 INFO HandlerThread:628228 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-15 01:14:13,362 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: defer
2021-07-15 01:14:13,362 INFO SenderThread:628228 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-15 01:14:13,362 INFO SenderThread:628228 [sender.py:transition_state():308] send defer: 2
2021-07-15 01:14:13,362 DEBUG SenderThread:628228 [sender.py:send():179] send: stats
2021-07-15 01:14:13,363 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:14:13,363 INFO HandlerThread:628228 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-15 01:14:13,363 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: defer
2021-07-15 01:14:13,363 INFO SenderThread:628228 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-15 01:14:13,363 INFO SenderThread:628228 [sender.py:transition_state():308] send defer: 3
2021-07-15 01:14:13,363 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:14:13,363 INFO HandlerThread:628228 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-15 01:14:13,363 DEBUG SenderThread:628228 [sender.py:send():179] send: summary
2021-07-15 01:14:13,364 INFO SenderThread:628228 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 01:14:13,364 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: defer
2021-07-15 01:14:13,364 INFO SenderThread:628228 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-15 01:14:13,364 INFO SenderThread:628228 [sender.py:transition_state():308] send defer: 4
2021-07-15 01:14:13,364 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:14:13,364 INFO HandlerThread:628228 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-15 01:14:13,364 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: defer
2021-07-15 01:14:13,364 INFO SenderThread:628228 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-15 01:14:13,417 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:14:13,544 INFO SenderThread:628228 [sender.py:transition_state():308] send defer: 5
2021-07-15 01:14:13,545 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:14:13,545 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:14:13,545 INFO HandlerThread:628228 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-15 01:14:13,545 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: defer
2021-07-15 01:14:13,545 INFO SenderThread:628228 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-15 01:14:13,545 INFO SenderThread:628228 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-15 01:14:13,647 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:14:14,043 INFO Thread-8 :628228 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/config.yaml
2021-07-15 01:14:14,044 INFO SenderThread:628228 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/output.log
2021-07-15 01:14:14,044 INFO SenderThread:628228 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/wandb-summary.json
2021-07-15 01:14:14,044 INFO SenderThread:628228 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files
2021-07-15 01:14:14,045 INFO SenderThread:628228 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/requirements.txt requirements.txt
2021-07-15 01:14:14,045 INFO SenderThread:628228 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/output.log output.log
2021-07-15 01:14:14,045 INFO SenderThread:628228 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/wandb-metadata.json wandb-metadata.json
2021-07-15 01:14:14,045 INFO SenderThread:628228 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/config.yaml config.yaml
2021-07-15 01:14:14,045 INFO SenderThread:628228 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/wandb-summary.json wandb-summary.json
2021-07-15 01:14:14,045 INFO SenderThread:628228 [sender.py:transition_state():308] send defer: 6
2021-07-15 01:14:14,046 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:14:14,052 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:14:14,052 INFO HandlerThread:628228 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-15 01:14:14,052 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: defer
2021-07-15 01:14:14,052 INFO SenderThread:628228 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-15 01:14:14,052 INFO SenderThread:628228 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 01:14:14,151 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:14:14,151 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:14:14,253 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:14:14,253 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:14:14,355 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:14:14,355 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:14:14,457 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:14:14,457 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:14:14,483 INFO Thread-12 :628228 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/requirements.txt
2021-07-15 01:14:14,487 INFO Thread-15 :628228 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/wandb-summary.json
2021-07-15 01:14:14,535 INFO Thread-14 :628228 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/config.yaml
2021-07-15 01:14:14,545 INFO Thread-13 :628228 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/files/output.log
2021-07-15 01:14:14,559 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:14:14,559 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:14:14,661 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:14:14,661 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:14:14,745 INFO Thread-7 :628228 [sender.py:transition_state():308] send defer: 7
2021-07-15 01:14:14,745 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:14:14,746 INFO HandlerThread:628228 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-15 01:14:14,746 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: defer
2021-07-15 01:14:14,746 INFO SenderThread:628228 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-15 01:14:14,763 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:14:15,027 INFO SenderThread:628228 [sender.py:transition_state():308] send defer: 8
2021-07-15 01:14:15,028 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:14:15,028 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:14:15,028 INFO HandlerThread:628228 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-15 01:14:15,028 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: defer
2021-07-15 01:14:15,028 INFO SenderThread:628228 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-15 01:14:15,028 INFO SenderThread:628228 [sender.py:transition_state():308] send defer: 9
2021-07-15 01:14:15,029 DEBUG SenderThread:628228 [sender.py:send():179] send: final
2021-07-15 01:14:15,029 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: defer
2021-07-15 01:14:15,029 DEBUG SenderThread:628228 [sender.py:send():179] send: footer
2021-07-15 01:14:15,029 INFO HandlerThread:628228 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-15 01:14:15,030 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: defer
2021-07-15 01:14:15,030 INFO SenderThread:628228 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-15 01:14:15,130 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 01:14:15,130 DEBUG SenderThread:628228 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 01:14:15,130 INFO SenderThread:628228 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 01:14:15,131 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: get_summary
2021-07-15 01:14:15,132 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-15 01:14:15,132 DEBUG HandlerThread:628228 [handler.py:handle_request():124] handle_request: shutdown
2021-07-15 01:14:15,133 INFO HandlerThread:628228 [handler.py:finish():638] shutting down handler
2021-07-15 01:14:16,030 INFO WriterThread:628228 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210715_010838-hepp3r83/run-hepp3r83.wandb
2021-07-15 01:14:16,130 INFO SenderThread:628228 [sender.py:finish():945] shutting down sender
2021-07-15 01:14:16,131 INFO SenderThread:628228 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 01:14:16,131 INFO SenderThread:628228 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 01:14:16,133 INFO MainThread:628228 [internal.py:handle_exit():78] Internal process exited
|