File size: 16,000 Bytes
f291f93 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 |
2021-07-14 21:00:58,647 INFO MainThread:574948 [internal.py:wandb_internal():88] W&B internal server running at pid: 574948, started at: 2021-07-14 21:00:58.647052
2021-07-14 21:00:58,649 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 21:00:58,649 INFO WriterThread:574948 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/run-dn4954s5.wandb
2021-07-14 21:00:58,650 DEBUG SenderThread:574948 [sender.py:send():179] send: header
2021-07-14 21:00:58,650 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: check_version
2021-07-14 21:00:58,686 DEBUG SenderThread:574948 [sender.py:send():179] send: run
2021-07-14 21:00:58,856 INFO SenderThread:574948 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files
2021-07-14 21:00:58,856 INFO SenderThread:574948 [sender.py:_start_run_threads():716] run started: dn4954s5 with start time 1626296458
2021-07-14 21:00:58,856 DEBUG SenderThread:574948 [sender.py:send():179] send: summary
2021-07-14 21:00:58,857 INFO SenderThread:574948 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:00:58,857 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 21:00:59,858 INFO Thread-8 :574948 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/wandb-summary.json
2021-07-14 21:00:59,959 DEBUG HandlerThread:574948 [meta.py:__init__():39] meta init
2021-07-14 21:00:59,959 DEBUG HandlerThread:574948 [meta.py:__init__():53] meta init done
2021-07-14 21:00:59,959 DEBUG HandlerThread:574948 [meta.py:probe():210] probe
2021-07-14 21:00:59,960 DEBUG HandlerThread:574948 [meta.py:_setup_git():200] setup git
2021-07-14 21:00:59,989 DEBUG HandlerThread:574948 [meta.py:_setup_git():207] setup git done
2021-07-14 21:00:59,989 DEBUG HandlerThread:574948 [meta.py:_save_pip():57] save pip
2021-07-14 21:00:59,989 DEBUG HandlerThread:574948 [meta.py:_save_pip():71] save pip done
2021-07-14 21:00:59,990 DEBUG HandlerThread:574948 [meta.py:probe():252] probe done
2021-07-14 21:00:59,992 DEBUG SenderThread:574948 [sender.py:send():179] send: files
2021-07-14 21:00:59,993 INFO SenderThread:574948 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 21:00:59,998 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:00:59,999 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:01:00,132 DEBUG SenderThread:574948 [sender.py:send():179] send: config
2021-07-14 21:01:00,133 DEBUG SenderThread:574948 [sender.py:send():179] send: config
2021-07-14 21:01:00,133 DEBUG SenderThread:574948 [sender.py:send():179] send: config
2021-07-14 21:01:00,445 INFO Thread-11 :574948 [upload_job.py:push():137] Uploaded file /tmp/tmpqq6ketpgwandb/y90ikahm-wandb-metadata.json
2021-07-14 21:01:00,857 INFO Thread-8 :574948 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log
2021-07-14 21:01:00,857 INFO Thread-8 :574948 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/requirements.txt
2021-07-14 21:01:00,857 INFO Thread-8 :574948 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/wandb-metadata.json
2021-07-14 21:01:14,863 INFO Thread-8 :574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log
2021-07-14 21:01:15,134 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:01:15,135 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:01:20,865 INFO Thread-8 :574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log
2021-07-14 21:01:28,042 DEBUG SenderThread:574948 [sender.py:send():179] send: stats
2021-07-14 21:01:29,869 INFO Thread-8 :574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/config.yaml
2021-07-14 21:01:30,286 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:01:30,287 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:01:45,439 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:01:45,440 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:01:58,125 DEBUG SenderThread:574948 [sender.py:send():179] send: stats
2021-07-14 21:02:00,570 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:02:00,571 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:02:15,704 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:02:15,704 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:02:28,204 DEBUG SenderThread:574948 [sender.py:send():179] send: stats
2021-07-14 21:02:30,841 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:02:30,841 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:02:32,894 INFO Thread-8 :574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log
2021-07-14 21:02:33,493 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:02:33,493 DEBUG SenderThread:574948 [sender.py:send():179] send: telemetry
2021-07-14 21:02:33,493 DEBUG SenderThread:574948 [sender.py:send():179] send: exit
2021-07-14 21:02:33,493 INFO SenderThread:574948 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 21:02:33,494 INFO SenderThread:574948 [sender.py:send_exit():295] send defer
2021-07-14 21:02:33,494 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:02:33,494 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,494 INFO HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 21:02:33,495 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,495 INFO SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 21:02:33,495 INFO SenderThread:574948 [sender.py:transition_state():308] send defer: 1
2021-07-14 21:02:33,495 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,495 INFO HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 21:02:33,518 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,518 INFO SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 21:02:33,518 INFO SenderThread:574948 [sender.py:transition_state():308] send defer: 2
2021-07-14 21:02:33,518 DEBUG SenderThread:574948 [sender.py:send():179] send: stats
2021-07-14 21:02:33,519 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,519 INFO HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 21:02:33,519 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,519 INFO SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 21:02:33,519 INFO SenderThread:574948 [sender.py:transition_state():308] send defer: 3
2021-07-14 21:02:33,520 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,520 INFO HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 21:02:33,520 DEBUG SenderThread:574948 [sender.py:send():179] send: summary
2021-07-14 21:02:33,520 INFO SenderThread:574948 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:02:33,520 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,520 INFO SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 21:02:33,520 INFO SenderThread:574948 [sender.py:transition_state():308] send defer: 4
2021-07-14 21:02:33,521 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,521 INFO HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 21:02:33,521 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,521 INFO SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 21:02:33,596 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:02:33,707 INFO SenderThread:574948 [sender.py:transition_state():308] send defer: 5
2021-07-14 21:02:33,708 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:02:33,708 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,708 INFO HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 21:02:33,709 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,709 INFO SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 21:02:33,709 INFO SenderThread:574948 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 21:02:33,810 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:02:33,895 INFO Thread-8 :574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/config.yaml
2021-07-14 21:02:33,895 INFO SenderThread:574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log
2021-07-14 21:02:33,895 INFO SenderThread:574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/wandb-summary.json
2021-07-14 21:02:33,895 INFO SenderThread:574948 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files
2021-07-14 21:02:33,896 INFO SenderThread:574948 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/requirements.txt requirements.txt
2021-07-14 21:02:33,896 INFO SenderThread:574948 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log output.log
2021-07-14 21:02:33,896 INFO SenderThread:574948 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/wandb-metadata.json wandb-metadata.json
2021-07-14 21:02:33,896 INFO SenderThread:574948 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/config.yaml config.yaml
2021-07-14 21:02:33,896 INFO SenderThread:574948 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/wandb-summary.json wandb-summary.json
2021-07-14 21:02:33,896 INFO SenderThread:574948 [sender.py:transition_state():308] send defer: 6
2021-07-14 21:02:33,897 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:02:33,908 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,908 INFO HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 21:02:33,909 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,909 INFO SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 21:02:33,909 INFO SenderThread:574948 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 21:02:34,002 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:02:34,002 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:02:34,104 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:02:34,104 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:02:34,203 WARNING MainThread:574948 [internal.py:wandb_internal():147] Internal process interrupt: 1
2021-07-14 21:02:34,354 INFO Thread-14 :574948 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/config.yaml
2021-07-14 21:02:34,366 INFO Thread-13 :574948 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log
2021-07-14 21:02:34,377 INFO Thread-12 :574948 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/requirements.txt
2021-07-14 21:02:34,379 INFO Thread-15 :574948 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/wandb-summary.json
2021-07-14 21:02:34,579 INFO Thread-7 :574948 [sender.py:transition_state():308] send defer: 7
2021-07-14 21:02:34,580 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:34,580 INFO HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 21:02:34,580 DEBUG SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:34,580 INFO SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 21:02:34,801 WARNING MainThread:574948 [internal.py:wandb_internal():147] Internal process interrupt: 2
2021-07-14 21:02:34,801 ERROR MainThread:574948 [internal.py:wandb_internal():150] Internal process interrupted.
2021-07-14 21:02:34,872 INFO SenderThread:574948 [sender.py:transition_state():308] send defer: 8
2021-07-14 21:02:34,872 INFO SenderThread:574948 [sender.py:finish():945] shutting down sender
2021-07-14 21:02:34,872 INFO SenderThread:574948 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 21:02:34,872 INFO SenderThread:574948 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 21:02:34,873 DEBUG HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:34,873 INFO HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 21:02:34,875 INFO HandlerThread:574948 [handler.py:finish():638] shutting down handler
2021-07-14 21:02:35,521 INFO WriterThread:574948 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/run-dn4954s5.wandb
2021-07-14 21:02:35,521 INFO MainThread:574948 [internal.py:handle_exit():78] Internal process exited
|