File size: 16,393 Bytes
f291f93 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 |
2021-07-14 20:01:08,001 INFO MainThread:564646 [internal.py:wandb_internal():88] W&B internal server running at pid: 564646, started at: 2021-07-14 20:01:08.000744
2021-07-14 20:01:08,003 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 20:01:08,003 INFO WriterThread:564646 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/run-3qnuldhi.wandb
2021-07-14 20:01:08,004 DEBUG SenderThread:564646 [sender.py:send():179] send: header
2021-07-14 20:01:08,004 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: check_version
2021-07-14 20:01:08,042 DEBUG SenderThread:564646 [sender.py:send():179] send: run
2021-07-14 20:01:08,213 INFO SenderThread:564646 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files
2021-07-14 20:01:08,213 INFO SenderThread:564646 [sender.py:_start_run_threads():716] run started: 3qnuldhi with start time 1626292867
2021-07-14 20:01:08,213 DEBUG SenderThread:564646 [sender.py:send():179] send: summary
2021-07-14 20:01:08,214 INFO SenderThread:564646 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:01:08,214 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 20:01:09,216 INFO Thread-8 :564646 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/wandb-summary.json
2021-07-14 20:01:09,340 DEBUG HandlerThread:564646 [meta.py:__init__():39] meta init
2021-07-14 20:01:09,340 DEBUG HandlerThread:564646 [meta.py:__init__():53] meta init done
2021-07-14 20:01:09,340 DEBUG HandlerThread:564646 [meta.py:probe():210] probe
2021-07-14 20:01:09,342 DEBUG HandlerThread:564646 [meta.py:_setup_git():200] setup git
2021-07-14 20:01:09,376 DEBUG HandlerThread:564646 [meta.py:_setup_git():207] setup git done
2021-07-14 20:01:09,376 DEBUG HandlerThread:564646 [meta.py:_save_pip():57] save pip
2021-07-14 20:01:09,377 DEBUG HandlerThread:564646 [meta.py:_save_pip():71] save pip done
2021-07-14 20:01:09,377 DEBUG HandlerThread:564646 [meta.py:probe():252] probe done
2021-07-14 20:01:09,380 DEBUG SenderThread:564646 [sender.py:send():179] send: files
2021-07-14 20:01:09,381 INFO SenderThread:564646 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 20:01:09,388 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:01:09,388 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:01:09,525 DEBUG SenderThread:564646 [sender.py:send():179] send: config
2021-07-14 20:01:09,526 DEBUG SenderThread:564646 [sender.py:send():179] send: config
2021-07-14 20:01:09,526 DEBUG SenderThread:564646 [sender.py:send():179] send: config
2021-07-14 20:01:09,828 INFO Thread-11 :564646 [upload_job.py:push():137] Uploaded file /tmp/tmp6zvg4jijwandb/288ietle-wandb-metadata.json
2021-07-14 20:01:10,214 INFO Thread-8 :564646 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/requirements.txt
2021-07-14 20:01:10,215 INFO Thread-8 :564646 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/output.log
2021-07-14 20:01:10,215 INFO Thread-8 :564646 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/wandb-metadata.json
2021-07-14 20:01:24,220 INFO Thread-8 :564646 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/output.log
2021-07-14 20:01:24,538 DEBUG SenderThread:564646 [sender.py:send():179] send: telemetry
2021-07-14 20:01:24,539 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:24,539 DEBUG SenderThread:564646 [sender.py:send():179] send: exit
2021-07-14 20:01:24,539 INFO SenderThread:564646 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 20:01:24,539 INFO SenderThread:564646 [sender.py:send_exit():295] send defer
2021-07-14 20:01:24,540 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:24,540 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:24,540 INFO HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 20:01:24,541 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:24,541 INFO SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 20:01:24,541 INFO SenderThread:564646 [sender.py:transition_state():308] send defer: 1
2021-07-14 20:01:24,541 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:24,541 INFO HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 20:01:24,590 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:24,590 INFO SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 20:01:24,590 INFO SenderThread:564646 [sender.py:transition_state():308] send defer: 2
2021-07-14 20:01:24,590 DEBUG SenderThread:564646 [sender.py:send():179] send: stats
2021-07-14 20:01:24,591 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:24,591 INFO HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 20:01:24,591 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:24,591 INFO SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 20:01:24,591 INFO SenderThread:564646 [sender.py:transition_state():308] send defer: 3
2021-07-14 20:01:24,591 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:24,591 INFO HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 20:01:24,592 DEBUG SenderThread:564646 [sender.py:send():179] send: summary
2021-07-14 20:01:24,592 INFO SenderThread:564646 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:01:24,592 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:24,592 INFO SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 20:01:24,592 INFO SenderThread:564646 [sender.py:transition_state():308] send defer: 4
2021-07-14 20:01:24,593 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:24,593 INFO HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 20:01:24,593 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:24,593 INFO SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 20:01:24,642 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:24,785 INFO SenderThread:564646 [sender.py:transition_state():308] send defer: 5
2021-07-14 20:01:24,786 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:24,786 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:24,786 INFO HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 20:01:24,787 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:24,787 INFO SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 20:01:24,787 INFO SenderThread:564646 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 20:01:24,888 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:25,221 INFO Thread-8 :564646 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/wandb-summary.json
2021-07-14 20:01:25,221 INFO SenderThread:564646 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/config.yaml
2021-07-14 20:01:25,222 INFO SenderThread:564646 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/output.log
2021-07-14 20:01:25,222 INFO SenderThread:564646 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files
2021-07-14 20:01:25,222 INFO SenderThread:564646 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/requirements.txt requirements.txt
2021-07-14 20:01:25,222 INFO SenderThread:564646 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/output.log output.log
2021-07-14 20:01:25,223 INFO SenderThread:564646 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/wandb-metadata.json wandb-metadata.json
2021-07-14 20:01:25,223 INFO SenderThread:564646 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/config.yaml config.yaml
2021-07-14 20:01:25,223 INFO SenderThread:564646 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/wandb-summary.json wandb-summary.json
2021-07-14 20:01:25,223 INFO SenderThread:564646 [sender.py:transition_state():308] send defer: 6
2021-07-14 20:01:25,223 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:25,227 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:25,227 INFO HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 20:01:25,230 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:25,230 INFO SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 20:01:25,230 INFO SenderThread:564646 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 20:01:25,328 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:25,328 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:25,430 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:25,431 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:25,533 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:25,533 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:25,635 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:25,635 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:25,672 INFO Thread-14 :564646 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/config.yaml
2021-07-14 20:01:25,678 INFO Thread-15 :564646 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/wandb-summary.json
2021-07-14 20:01:25,692 INFO Thread-13 :564646 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/output.log
2021-07-14 20:01:25,737 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:25,737 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:25,743 INFO Thread-12 :564646 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/requirements.txt
2021-07-14 20:01:25,839 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:25,839 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:25,941 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:25,941 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:25,943 INFO Thread-7 :564646 [sender.py:transition_state():308] send defer: 7
2021-07-14 20:01:25,944 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:25,944 INFO HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 20:01:25,944 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:25,944 INFO SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 20:01:26,043 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:26,074 INFO SenderThread:564646 [sender.py:transition_state():308] send defer: 8
2021-07-14 20:01:26,074 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:26,075 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:26,075 INFO HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 20:01:26,075 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:26,075 INFO SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 20:01:26,075 INFO SenderThread:564646 [sender.py:transition_state():308] send defer: 9
2021-07-14 20:01:26,076 DEBUG SenderThread:564646 [sender.py:send():179] send: final
2021-07-14 20:01:26,076 DEBUG SenderThread:564646 [sender.py:send():179] send: footer
2021-07-14 20:01:26,076 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:26,076 INFO HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 20:01:26,076 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:26,076 INFO SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 20:01:26,176 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:26,177 DEBUG SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:26,177 INFO SenderThread:564646 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 20:01:26,178 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 20:01:26,179 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 20:01:26,180 DEBUG HandlerThread:564646 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 20:01:26,180 INFO HandlerThread:564646 [handler.py:finish():638] shutting down handler
2021-07-14 20:01:27,076 INFO WriterThread:564646 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/run-3qnuldhi.wandb
2021-07-14 20:01:27,177 INFO SenderThread:564646 [sender.py:finish():945] shutting down sender
2021-07-14 20:01:27,177 INFO SenderThread:564646 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 20:01:27,177 INFO SenderThread:564646 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 20:01:27,180 INFO MainThread:564646 [internal.py:handle_exit():78] Internal process exited
|