File size: 16,000 Bytes
f291f93
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
2021-07-14 21:00:58,647 INFO    MainThread:574948 [internal.py:wandb_internal():88] W&B internal server running at pid: 574948, started at: 2021-07-14 21:00:58.647052
2021-07-14 21:00:58,649 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 21:00:58,649 INFO    WriterThread:574948 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/run-dn4954s5.wandb
2021-07-14 21:00:58,650 DEBUG   SenderThread:574948 [sender.py:send():179] send: header
2021-07-14 21:00:58,650 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: check_version
2021-07-14 21:00:58,686 DEBUG   SenderThread:574948 [sender.py:send():179] send: run
2021-07-14 21:00:58,856 INFO    SenderThread:574948 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files
2021-07-14 21:00:58,856 INFO    SenderThread:574948 [sender.py:_start_run_threads():716] run started: dn4954s5 with start time 1626296458
2021-07-14 21:00:58,856 DEBUG   SenderThread:574948 [sender.py:send():179] send: summary
2021-07-14 21:00:58,857 INFO    SenderThread:574948 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:00:58,857 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 21:00:59,858 INFO    Thread-8  :574948 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/wandb-summary.json
2021-07-14 21:00:59,959 DEBUG   HandlerThread:574948 [meta.py:__init__():39] meta init
2021-07-14 21:00:59,959 DEBUG   HandlerThread:574948 [meta.py:__init__():53] meta init done
2021-07-14 21:00:59,959 DEBUG   HandlerThread:574948 [meta.py:probe():210] probe
2021-07-14 21:00:59,960 DEBUG   HandlerThread:574948 [meta.py:_setup_git():200] setup git
2021-07-14 21:00:59,989 DEBUG   HandlerThread:574948 [meta.py:_setup_git():207] setup git done
2021-07-14 21:00:59,989 DEBUG   HandlerThread:574948 [meta.py:_save_pip():57] save pip
2021-07-14 21:00:59,989 DEBUG   HandlerThread:574948 [meta.py:_save_pip():71] save pip done
2021-07-14 21:00:59,990 DEBUG   HandlerThread:574948 [meta.py:probe():252] probe done
2021-07-14 21:00:59,992 DEBUG   SenderThread:574948 [sender.py:send():179] send: files
2021-07-14 21:00:59,993 INFO    SenderThread:574948 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 21:00:59,998 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:00:59,999 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:01:00,132 DEBUG   SenderThread:574948 [sender.py:send():179] send: config
2021-07-14 21:01:00,133 DEBUG   SenderThread:574948 [sender.py:send():179] send: config
2021-07-14 21:01:00,133 DEBUG   SenderThread:574948 [sender.py:send():179] send: config
2021-07-14 21:01:00,445 INFO    Thread-11 :574948 [upload_job.py:push():137] Uploaded file /tmp/tmpqq6ketpgwandb/y90ikahm-wandb-metadata.json
2021-07-14 21:01:00,857 INFO    Thread-8  :574948 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log
2021-07-14 21:01:00,857 INFO    Thread-8  :574948 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/requirements.txt
2021-07-14 21:01:00,857 INFO    Thread-8  :574948 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/wandb-metadata.json
2021-07-14 21:01:14,863 INFO    Thread-8  :574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log
2021-07-14 21:01:15,134 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:01:15,135 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:01:20,865 INFO    Thread-8  :574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log
2021-07-14 21:01:28,042 DEBUG   SenderThread:574948 [sender.py:send():179] send: stats
2021-07-14 21:01:29,869 INFO    Thread-8  :574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/config.yaml
2021-07-14 21:01:30,286 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:01:30,287 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:01:45,439 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:01:45,440 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:01:58,125 DEBUG   SenderThread:574948 [sender.py:send():179] send: stats
2021-07-14 21:02:00,570 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:02:00,571 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:02:15,704 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:02:15,704 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:02:28,204 DEBUG   SenderThread:574948 [sender.py:send():179] send: stats
2021-07-14 21:02:30,841 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:02:30,841 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:02:32,894 INFO    Thread-8  :574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log
2021-07-14 21:02:33,493 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:02:33,493 DEBUG   SenderThread:574948 [sender.py:send():179] send: telemetry
2021-07-14 21:02:33,493 DEBUG   SenderThread:574948 [sender.py:send():179] send: exit
2021-07-14 21:02:33,493 INFO    SenderThread:574948 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 21:02:33,494 INFO    SenderThread:574948 [sender.py:send_exit():295] send defer
2021-07-14 21:02:33,494 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:02:33,494 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,494 INFO    HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 21:02:33,495 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,495 INFO    SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 21:02:33,495 INFO    SenderThread:574948 [sender.py:transition_state():308] send defer: 1
2021-07-14 21:02:33,495 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,495 INFO    HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 21:02:33,518 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,518 INFO    SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 21:02:33,518 INFO    SenderThread:574948 [sender.py:transition_state():308] send defer: 2
2021-07-14 21:02:33,518 DEBUG   SenderThread:574948 [sender.py:send():179] send: stats
2021-07-14 21:02:33,519 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,519 INFO    HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 21:02:33,519 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,519 INFO    SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 21:02:33,519 INFO    SenderThread:574948 [sender.py:transition_state():308] send defer: 3
2021-07-14 21:02:33,520 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,520 INFO    HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 21:02:33,520 DEBUG   SenderThread:574948 [sender.py:send():179] send: summary
2021-07-14 21:02:33,520 INFO    SenderThread:574948 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:02:33,520 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,520 INFO    SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 21:02:33,520 INFO    SenderThread:574948 [sender.py:transition_state():308] send defer: 4
2021-07-14 21:02:33,521 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,521 INFO    HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 21:02:33,521 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,521 INFO    SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 21:02:33,596 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:02:33,707 INFO    SenderThread:574948 [sender.py:transition_state():308] send defer: 5
2021-07-14 21:02:33,708 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:02:33,708 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,708 INFO    HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 21:02:33,709 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,709 INFO    SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 21:02:33,709 INFO    SenderThread:574948 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 21:02:33,810 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:02:33,895 INFO    Thread-8  :574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/config.yaml
2021-07-14 21:02:33,895 INFO    SenderThread:574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log
2021-07-14 21:02:33,895 INFO    SenderThread:574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/wandb-summary.json
2021-07-14 21:02:33,895 INFO    SenderThread:574948 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files
2021-07-14 21:02:33,896 INFO    SenderThread:574948 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/requirements.txt requirements.txt
2021-07-14 21:02:33,896 INFO    SenderThread:574948 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log output.log
2021-07-14 21:02:33,896 INFO    SenderThread:574948 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/wandb-metadata.json wandb-metadata.json
2021-07-14 21:02:33,896 INFO    SenderThread:574948 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/config.yaml config.yaml
2021-07-14 21:02:33,896 INFO    SenderThread:574948 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/wandb-summary.json wandb-summary.json
2021-07-14 21:02:33,896 INFO    SenderThread:574948 [sender.py:transition_state():308] send defer: 6
2021-07-14 21:02:33,897 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:02:33,908 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,908 INFO    HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 21:02:33,909 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,909 INFO    SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 21:02:33,909 INFO    SenderThread:574948 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 21:02:34,002 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:02:34,002 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:02:34,104 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:02:34,104 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:02:34,203 WARNING MainThread:574948 [internal.py:wandb_internal():147] Internal process interrupt: 1
2021-07-14 21:02:34,354 INFO    Thread-14 :574948 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/config.yaml
2021-07-14 21:02:34,366 INFO    Thread-13 :574948 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log
2021-07-14 21:02:34,377 INFO    Thread-12 :574948 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/requirements.txt
2021-07-14 21:02:34,379 INFO    Thread-15 :574948 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/wandb-summary.json
2021-07-14 21:02:34,579 INFO    Thread-7  :574948 [sender.py:transition_state():308] send defer: 7
2021-07-14 21:02:34,580 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:34,580 INFO    HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 21:02:34,580 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:34,580 INFO    SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 21:02:34,801 WARNING MainThread:574948 [internal.py:wandb_internal():147] Internal process interrupt: 2
2021-07-14 21:02:34,801 ERROR   MainThread:574948 [internal.py:wandb_internal():150] Internal process interrupted.
2021-07-14 21:02:34,872 INFO    SenderThread:574948 [sender.py:transition_state():308] send defer: 8
2021-07-14 21:02:34,872 INFO    SenderThread:574948 [sender.py:finish():945] shutting down sender
2021-07-14 21:02:34,872 INFO    SenderThread:574948 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 21:02:34,872 INFO    SenderThread:574948 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 21:02:34,873 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:34,873 INFO    HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 21:02:34,875 INFO    HandlerThread:574948 [handler.py:finish():638] shutting down handler
2021-07-14 21:02:35,521 INFO    WriterThread:574948 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/run-dn4954s5.wandb
2021-07-14 21:02:35,521 INFO    MainThread:574948 [internal.py:handle_exit():78] Internal process exited