File size: 16,773 Bytes
d725b93 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 |
2021-07-14 21:35:38,058 INFO MainThread:582612 [internal.py:wandb_internal():88] W&B internal server running at pid: 582612, started at: 2021-07-14 21:35:38.058075
2021-07-14 21:35:38,060 INFO WriterThread:582612 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/run-20l16od8.wandb
2021-07-14 21:35:38,060 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 21:35:38,061 DEBUG SenderThread:582612 [sender.py:send():179] send: header
2021-07-14 21:35:38,061 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: check_version
2021-07-14 21:35:38,099 DEBUG SenderThread:582612 [sender.py:send():179] send: run
2021-07-14 21:35:38,275 INFO SenderThread:582612 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files
2021-07-14 21:35:38,275 INFO SenderThread:582612 [sender.py:_start_run_threads():716] run started: 20l16od8 with start time 1626298537
2021-07-14 21:35:38,275 DEBUG SenderThread:582612 [sender.py:send():179] send: summary
2021-07-14 21:35:38,275 INFO SenderThread:582612 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:35:38,276 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 21:35:39,277 INFO Thread-8 :582612 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/wandb-summary.json
2021-07-14 21:35:39,374 DEBUG HandlerThread:582612 [meta.py:__init__():39] meta init
2021-07-14 21:35:39,374 DEBUG HandlerThread:582612 [meta.py:__init__():53] meta init done
2021-07-14 21:35:39,374 DEBUG HandlerThread:582612 [meta.py:probe():210] probe
2021-07-14 21:35:39,375 DEBUG HandlerThread:582612 [meta.py:_setup_git():200] setup git
2021-07-14 21:35:39,404 DEBUG HandlerThread:582612 [meta.py:_setup_git():207] setup git done
2021-07-14 21:35:39,405 DEBUG HandlerThread:582612 [meta.py:_save_pip():57] save pip
2021-07-14 21:35:39,405 DEBUG HandlerThread:582612 [meta.py:_save_pip():71] save pip done
2021-07-14 21:35:39,405 DEBUG HandlerThread:582612 [meta.py:probe():252] probe done
2021-07-14 21:35:39,408 DEBUG SenderThread:582612 [sender.py:send():179] send: files
2021-07-14 21:35:39,408 INFO SenderThread:582612 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 21:35:39,414 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:35:39,415 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:35:39,542 DEBUG SenderThread:582612 [sender.py:send():179] send: config
2021-07-14 21:35:39,542 DEBUG SenderThread:582612 [sender.py:send():179] send: config
2021-07-14 21:35:39,542 DEBUG SenderThread:582612 [sender.py:send():179] send: config
2021-07-14 21:35:39,845 INFO Thread-11 :582612 [upload_job.py:push():137] Uploaded file /tmp/tmp81cfdra0wandb/3f4lkdmt-wandb-metadata.json
2021-07-14 21:35:40,276 INFO Thread-8 :582612 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/requirements.txt
2021-07-14 21:35:40,276 INFO Thread-8 :582612 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/wandb-metadata.json
2021-07-14 21:35:40,276 INFO Thread-8 :582612 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/output.log
2021-07-14 21:35:54,281 INFO Thread-8 :582612 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/output.log
2021-07-14 21:35:54,573 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:35:54,574 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:35:56,282 INFO Thread-8 :582612 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/output.log
2021-07-14 21:35:57,283 INFO Thread-8 :582612 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/output.log
2021-07-14 21:35:58,079 DEBUG SenderThread:582612 [sender.py:send():179] send: telemetry
2021-07-14 21:35:58,080 DEBUG SenderThread:582612 [sender.py:send():179] send: exit
2021-07-14 21:35:58,080 INFO SenderThread:582612 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 21:35:58,080 INFO SenderThread:582612 [sender.py:send_exit():295] send defer
2021-07-14 21:35:58,080 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:35:58,080 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:35:58,081 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:35:58,081 INFO HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 21:35:58,081 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:35:58,081 INFO SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 21:35:58,081 INFO SenderThread:582612 [sender.py:transition_state():308] send defer: 1
2021-07-14 21:35:58,081 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:35:58,081 INFO HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 21:35:58,128 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:35:58,128 INFO SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 21:35:58,128 INFO SenderThread:582612 [sender.py:transition_state():308] send defer: 2
2021-07-14 21:35:58,128 DEBUG SenderThread:582612 [sender.py:send():179] send: stats
2021-07-14 21:35:58,128 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:35:58,129 INFO HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 21:35:58,129 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:35:58,129 INFO SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 21:35:58,129 INFO SenderThread:582612 [sender.py:transition_state():308] send defer: 3
2021-07-14 21:35:58,129 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:35:58,129 INFO HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 21:35:58,129 DEBUG SenderThread:582612 [sender.py:send():179] send: summary
2021-07-14 21:35:58,130 INFO SenderThread:582612 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:35:58,130 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:35:58,130 INFO SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 21:35:58,130 INFO SenderThread:582612 [sender.py:transition_state():308] send defer: 4
2021-07-14 21:35:58,130 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:35:58,130 INFO HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 21:35:58,130 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:35:58,130 INFO SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 21:35:58,183 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:35:58,299 INFO SenderThread:582612 [sender.py:transition_state():308] send defer: 5
2021-07-14 21:35:58,299 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:35:58,300 INFO Thread-8 :582612 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/output.log
2021-07-14 21:35:58,300 INFO Thread-8 :582612 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/wandb-summary.json
2021-07-14 21:35:58,300 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:35:58,300 INFO HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 21:35:58,300 INFO Thread-8 :582612 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/config.yaml
2021-07-14 21:35:58,301 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:35:58,301 INFO SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 21:35:58,301 INFO SenderThread:582612 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 21:35:58,402 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:35:59,301 INFO SenderThread:582612 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files
2021-07-14 21:35:59,302 INFO SenderThread:582612 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/requirements.txt requirements.txt
2021-07-14 21:35:59,302 INFO SenderThread:582612 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/output.log output.log
2021-07-14 21:35:59,302 INFO SenderThread:582612 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/wandb-metadata.json wandb-metadata.json
2021-07-14 21:35:59,302 INFO SenderThread:582612 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/config.yaml config.yaml
2021-07-14 21:35:59,302 INFO SenderThread:582612 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/wandb-summary.json wandb-summary.json
2021-07-14 21:35:59,302 INFO SenderThread:582612 [sender.py:transition_state():308] send defer: 6
2021-07-14 21:35:59,303 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:35:59,303 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:35:59,303 INFO HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 21:35:59,307 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:35:59,307 INFO SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 21:35:59,307 INFO SenderThread:582612 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 21:35:59,405 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:35:59,405 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:35:59,507 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:35:59,507 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:35:59,609 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:35:59,609 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:35:59,711 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:35:59,711 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:35:59,738 INFO Thread-15 :582612 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/wandb-summary.json
2021-07-14 21:35:59,759 INFO Thread-13 :582612 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/output.log
2021-07-14 21:35:59,764 INFO Thread-14 :582612 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/config.yaml
2021-07-14 21:35:59,779 INFO Thread-12 :582612 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/requirements.txt
2021-07-14 21:35:59,813 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:35:59,813 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:35:59,915 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:35:59,915 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:35:59,979 INFO Thread-7 :582612 [sender.py:transition_state():308] send defer: 7
2021-07-14 21:35:59,980 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:35:59,980 INFO HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 21:35:59,980 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:35:59,980 INFO SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 21:36:00,016 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:36:00,132 INFO SenderThread:582612 [sender.py:transition_state():308] send defer: 8
2021-07-14 21:36:00,132 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:36:00,133 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:36:00,133 INFO HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 21:36:00,133 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:36:00,133 INFO SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 21:36:00,133 INFO SenderThread:582612 [sender.py:transition_state():308] send defer: 9
2021-07-14 21:36:00,134 DEBUG SenderThread:582612 [sender.py:send():179] send: final
2021-07-14 21:36:00,134 DEBUG SenderThread:582612 [sender.py:send():179] send: footer
2021-07-14 21:36:00,134 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:36:00,134 INFO HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 21:36:00,134 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:36:00,134 INFO SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 21:36:00,234 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:36:00,235 DEBUG SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:36:00,235 INFO SenderThread:582612 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 21:36:00,236 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 21:36:00,237 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 21:36:00,237 DEBUG HandlerThread:582612 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 21:36:00,237 INFO HandlerThread:582612 [handler.py:finish():638] shutting down handler
2021-07-14 21:36:01,134 INFO WriterThread:582612 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/run-20l16od8.wandb
2021-07-14 21:36:01,235 INFO SenderThread:582612 [sender.py:finish():945] shutting down sender
2021-07-14 21:36:01,235 INFO SenderThread:582612 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 21:36:01,235 INFO SenderThread:582612 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 21:36:01,237 INFO MainThread:582612 [internal.py:handle_exit():78] Internal process exited
|