File size: 25,025 Bytes
f291f93
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
2021-07-13 10:47:45,828 INFO    MainThread:342403 [internal.py:wandb_internal():88] W&B internal server running at pid: 342403, started at: 2021-07-13 10:47:45.828158
2021-07-13 10:47:45,830 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: check_version
2021-07-13 10:47:45,830 INFO    WriterThread:342403 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/run-1rl2j7or.wandb
2021-07-13 10:47:45,831 DEBUG   SenderThread:342403 [sender.py:send():179] send: header
2021-07-13 10:47:45,831 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: check_version
2021-07-13 10:47:45,871 DEBUG   SenderThread:342403 [sender.py:send():179] send: run
2021-07-13 10:47:46,041 INFO    SenderThread:342403 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files
2021-07-13 10:47:46,041 INFO    SenderThread:342403 [sender.py:_start_run_threads():716] run started: 1rl2j7or with start time 1626173265
2021-07-13 10:47:46,041 DEBUG   SenderThread:342403 [sender.py:send():179] send: summary
2021-07-13 10:47:46,041 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: run_start
2021-07-13 10:47:46,042 INFO    SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 10:47:47,043 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json
2021-07-13 10:47:47,215 DEBUG   HandlerThread:342403 [meta.py:__init__():39] meta init
2021-07-13 10:47:47,215 DEBUG   HandlerThread:342403 [meta.py:__init__():53] meta init done
2021-07-13 10:47:47,215 DEBUG   HandlerThread:342403 [meta.py:probe():210] probe
2021-07-13 10:47:47,217 DEBUG   HandlerThread:342403 [meta.py:_setup_git():200] setup git
2021-07-13 10:47:47,250 DEBUG   HandlerThread:342403 [meta.py:_setup_git():207] setup git done
2021-07-13 10:47:47,250 DEBUG   HandlerThread:342403 [meta.py:_save_pip():57] save pip
2021-07-13 10:47:47,251 DEBUG   HandlerThread:342403 [meta.py:_save_pip():71] save pip done
2021-07-13 10:47:47,251 DEBUG   HandlerThread:342403 [meta.py:probe():252] probe done
2021-07-13 10:47:47,255 DEBUG   SenderThread:342403 [sender.py:send():179] send: files
2021-07-13 10:47:47,255 INFO    SenderThread:342403 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-13 10:47:47,262 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:47:47,262 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:47:47,394 DEBUG   SenderThread:342403 [sender.py:send():179] send: config
2021-07-13 10:47:47,394 DEBUG   SenderThread:342403 [sender.py:send():179] send: config
2021-07-13 10:47:47,394 DEBUG   SenderThread:342403 [sender.py:send():179] send: config
2021-07-13 10:47:47,719 INFO    Thread-11 :342403 [upload_job.py:push():137] Uploaded file /tmp/tmpta17r5ywwandb/1f1555en-wandb-metadata.json
2021-07-13 10:47:48,042 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-metadata.json
2021-07-13 10:47:48,042 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/requirements.txt
2021-07-13 10:47:48,042 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:48:02,047 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:48:02,398 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:48:02,398 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:48:04,048 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:48:15,296 DEBUG   SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:48:17,054 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/config.yaml
2021-07-13 10:48:17,555 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:48:17,556 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:48:32,709 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:48:32,710 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:48:45,371 DEBUG   SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:48:47,840 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:48:47,840 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:49:02,980 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:49:02,980 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:49:15,445 DEBUG   SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:49:18,113 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:49:18,113 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:49:24,080 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:49:26,080 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:49:28,081 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:49:30,082 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:49:32,083 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:49:33,242 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:49:33,243 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:49:34,084 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:49:36,084 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:49:45,514 DEBUG   SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:49:48,375 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:49:48,375 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:49:58,179 DEBUG   SenderThread:342403 [sender.py:send():179] send: history
2021-07-13 10:49:58,180 DEBUG   SenderThread:342403 [sender.py:send():179] send: summary
2021-07-13 10:49:58,180 INFO    SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 10:49:59,093 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json
2021-07-13 10:50:00,093 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:02,094 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:03,510 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:50:03,510 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:50:04,095 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:15,583 DEBUG   SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:50:18,643 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:50:18,643 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:50:24,102 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:28,758 DEBUG   SenderThread:342403 [sender.py:send():179] send: history
2021-07-13 10:50:28,759 DEBUG   SenderThread:342403 [sender.py:send():179] send: summary
2021-07-13 10:50:28,763 INFO    SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 10:50:29,104 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json
2021-07-13 10:50:30,105 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:32,106 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:33,775 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:50:33,776 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:50:34,107 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:36,107 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:38,108 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:40,109 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:42,110 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:50:45,653 DEBUG   SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:50:48,905 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:50:48,906 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:51:04,035 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:51:04,035 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:51:04,964 DEBUG   SenderThread:342403 [sender.py:send():179] send: history
2021-07-13 10:51:04,964 DEBUG   SenderThread:342403 [sender.py:send():179] send: summary
2021-07-13 10:51:04,964 INFO    SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 10:51:05,119 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json
2021-07-13 10:51:06,119 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:08,120 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:15,726 DEBUG   SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:51:19,168 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:51:19,168 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:51:24,126 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:26,127 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:34,303 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:51:34,303 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:51:35,557 DEBUG   SenderThread:342403 [sender.py:send():179] send: history
2021-07-13 10:51:35,558 DEBUG   SenderThread:342403 [sender.py:send():179] send: summary
2021-07-13 10:51:35,558 INFO    SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 10:51:36,131 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json
2021-07-13 10:51:36,132 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:38,132 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:40,133 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:42,134 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:44,135 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:45,797 DEBUG   SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:51:46,136 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:48,137 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:51:49,438 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:51:49,438 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:51:50,137 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:04,579 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:52:04,580 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:52:11,761 DEBUG   SenderThread:342403 [sender.py:send():179] send: history
2021-07-13 10:52:11,762 DEBUG   SenderThread:342403 [sender.py:send():179] send: summary
2021-07-13 10:52:11,763 INFO    SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 10:52:12,146 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json
2021-07-13 10:52:14,147 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:15,867 DEBUG   SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:52:19,709 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:52:19,710 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:52:24,150 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:26,151 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:34,838 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:52:34,839 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:52:42,378 DEBUG   SenderThread:342403 [sender.py:send():179] send: history
2021-07-13 10:52:42,378 DEBUG   SenderThread:342403 [sender.py:send():179] send: summary
2021-07-13 10:52:42,379 INFO    SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 10:52:43,158 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json
2021-07-13 10:52:45,159 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:45,939 DEBUG   SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:52:47,160 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:49,161 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:49,969 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:52:49,970 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:52:51,161 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:53,162 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:55,163 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:52:57,164 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:53:05,101 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:53:05,101 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:53:16,014 DEBUG   SenderThread:342403 [sender.py:send():179] send: stats
2021-07-13 10:53:18,580 DEBUG   SenderThread:342403 [sender.py:send():179] send: history
2021-07-13 10:53:18,580 DEBUG   SenderThread:342403 [sender.py:send():179] send: summary
2021-07-13 10:53:18,580 INFO    SenderThread:342403 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 10:53:19,173 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json
2021-07-13 10:53:20,233 DEBUG   HandlerThread:342403 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 10:53:20,234 DEBUG   SenderThread:342403 [sender.py:send_request():193] send_request: stop_status
2021-07-13 10:53:21,173 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:53:25,175 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:53:27,176 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:53:29,177 INFO    Thread-8  :342403 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:53:34,237 WARNING MainThread:342403 [internal.py:wandb_internal():147] Internal process interrupt: 1
2021-07-13 10:53:34,484 WARNING MainThread:342403 [internal.py:wandb_internal():147] Internal process interrupt: 2
2021-07-13 10:53:34,484 ERROR   MainThread:342403 [internal.py:wandb_internal():150] Internal process interrupted.
2021-07-13 10:53:35,385 INFO    WriterThread:342403 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/run-1rl2j7or.wandb
2021-07-13 10:53:35,409 INFO    SenderThread:342403 [sender.py:finish():945] shutting down sender
2021-07-13 10:53:35,409 INFO    SenderThread:342403 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-13 10:53:35,414 INFO    HandlerThread:342403 [handler.py:finish():638] shutting down handler
2021-07-13 10:53:36,180 INFO    SenderThread:342403 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files
2021-07-13 10:53:36,180 INFO    SenderThread:342403 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/requirements.txt requirements.txt
2021-07-13 10:53:36,180 INFO    SenderThread:342403 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log output.log
2021-07-13 10:53:36,180 INFO    SenderThread:342403 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-metadata.json wandb-metadata.json
2021-07-13 10:53:36,180 INFO    SenderThread:342403 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/config.yaml config.yaml
2021-07-13 10:53:36,181 INFO    SenderThread:342403 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json wandb-summary.json
2021-07-13 10:53:36,181 INFO    SenderThread:342403 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 10:53:36,181 INFO    SenderThread:342403 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 10:53:36,622 INFO    Thread-14 :342403 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/config.yaml
2021-07-13 10:53:36,624 INFO    Thread-15 :342403 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/wandb-summary.json
2021-07-13 10:53:36,634 INFO    Thread-13 :342403 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/output.log
2021-07-13 10:53:36,654 INFO    Thread-12 :342403 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_104745-1rl2j7or/files/requirements.txt
2021-07-13 10:53:37,518 INFO    MainThread:342403 [internal.py:handle_exit():78] Internal process exited