2024-02-27 10:01:25,418 INFO StreamThr :1200 [internal.py:wandb_internal():86] W&B internal server running at pid: 1200, started at: 2024-02-27 10:01:25.417816 2024-02-27 10:01:25,425 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status 2024-02-27 10:01:26,746 INFO WriterThread:1200 [datastore.py:open_for_write():87] open: /content/wandb/run-20240227_100125-48zzkb8h/run-48zzkb8h.wandb 2024-02-27 10:01:26,747 DEBUG SenderThread:1200 [sender.py:send():382] send: header 2024-02-27 10:01:26,749 DEBUG SenderThread:1200 [sender.py:send():382] send: run 2024-02-27 10:01:27,005 INFO SenderThread:1200 [dir_watcher.py:__init__():211] watching files in: /content/wandb/run-20240227_100125-48zzkb8h/files 2024-02-27 10:01:27,005 INFO SenderThread:1200 [sender.py:_start_run_threads():1136] run started: 48zzkb8h with start time 1709028085.417921 2024-02-27 10:01:27,011 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: check_version 2024-02-27 10:01:27,012 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: check_version 2024-02-27 10:01:27,126 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: run_start 2024-02-27 10:01:27,136 DEBUG HandlerThread:1200 [system_info.py:__init__():27] System info init 2024-02-27 10:01:27,136 DEBUG HandlerThread:1200 [system_info.py:__init__():42] System info init done 2024-02-27 10:01:27,136 INFO HandlerThread:1200 [system_monitor.py:start():194] Starting system monitor 2024-02-27 10:01:27,136 INFO SystemMonitor:1200 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-02-27 10:01:27,137 INFO HandlerThread:1200 [system_monitor.py:probe():214] Collecting system info 2024-02-27 10:01:27,137 INFO SystemMonitor:1200 [interfaces.py:start():190] Started cpu monitoring 2024-02-27 10:01:27,138 INFO SystemMonitor:1200 [interfaces.py:start():190] Started disk monitoring 2024-02-27 10:01:27,139 INFO SystemMonitor:1200 [interfaces.py:start():190] Started gpu monitoring 2024-02-27 10:01:27,141 INFO SystemMonitor:1200 [interfaces.py:start():190] Started memory monitoring 2024-02-27 10:01:27,141 INFO SystemMonitor:1200 [interfaces.py:start():190] Started network monitoring 2024-02-27 10:01:27,152 DEBUG HandlerThread:1200 [system_info.py:probe():151] Probing system 2024-02-27 10:01:27,153 DEBUG HandlerThread:1200 [gitlib.py:_init_repo():56] git repository is invalid 2024-02-27 10:01:27,153 DEBUG HandlerThread:1200 [system_info.py:probe():199] Probing system done 2024-02-27 10:01:27,153 DEBUG HandlerThread:1200 [system_monitor.py:probe():223] {'os': 'Linux-6.1.58+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-02-27T10:01:27.152142', 'startedAt': '2024-02-27T10:01:25.411075', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'llm_trainer.ipynb', 'codePathLocal': None, 'colab': 'https://colab.research.google.com/notebook#fileId=1dgMRUGIo6hCY2R_ORmq2m7HIY7_B6rBN', 'host': 'e0994cadb423', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 1, 'cpu_count_logical': 2, 'cpu_freq': {'current': 2000.188, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.188, 'min': 0.0, 'max': 0.0}, {'current': 2000.188, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 78.1898422241211, 'used': 26.769989013671875}}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 12.674789428710938}} 2024-02-27 10:01:27,153 INFO HandlerThread:1200 [system_monitor.py:probe():224] Finished collecting system info 2024-02-27 10:01:27,153 INFO HandlerThread:1200 [system_monitor.py:probe():227] Publishing system info 2024-02-27 10:01:27,154 INFO HandlerThread:1200 [system_monitor.py:probe():229] Finished publishing system info 2024-02-27 10:01:27,158 DEBUG SenderThread:1200 [sender.py:send():382] send: files 2024-02-27 10:01:27,158 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now 2024-02-27 10:01:27,181 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: python_packages 2024-02-27 10:01:27,181 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: python_packages 2024-02-27 10:01:27,221 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:01:27,221 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:01:27,221 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: pause 2024-02-27 10:01:27,221 INFO HandlerThread:1200 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-02-27 10:01:27,221 INFO HandlerThread:1200 [system_monitor.py:finish():203] Stopping system monitor 2024-02-27 10:01:27,223 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:01:27,228 DEBUG SystemMonitor:1200 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-02-27 10:01:27,228 DEBUG SystemMonitor:1200 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-02-27 10:01:27,228 DEBUG SystemMonitor:1200 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-02-27 10:01:27,229 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined cpu monitor 2024-02-27 10:01:27,229 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined disk monitor 2024-02-27 10:01:27,231 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined gpu monitor 2024-02-27 10:01:27,232 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined memory monitor 2024-02-27 10:01:27,232 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined network monitor 2024-02-27 10:01:27,232 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: resume 2024-02-27 10:01:27,232 INFO HandlerThread:1200 [handler.py:handle_request_resume():699] starting system metrics thread 2024-02-27 10:01:27,232 INFO HandlerThread:1200 [system_monitor.py:start():194] Starting system monitor 2024-02-27 10:01:27,232 INFO SystemMonitor:1200 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-02-27 10:01:27,232 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: pause 2024-02-27 10:01:27,233 INFO HandlerThread:1200 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-02-27 10:01:27,233 INFO HandlerThread:1200 [system_monitor.py:finish():203] Stopping system monitor 2024-02-27 10:01:27,233 INFO SystemMonitor:1200 [interfaces.py:start():190] Started cpu monitoring 2024-02-27 10:01:27,233 DEBUG SystemMonitor:1200 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-02-27 10:01:27,233 DEBUG SystemMonitor:1200 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-02-27 10:01:27,233 DEBUG SystemMonitor:1200 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-02-27 10:01:27,234 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined cpu monitor 2024-02-27 10:01:27,234 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: resume 2024-02-27 10:01:27,234 INFO HandlerThread:1200 [handler.py:handle_request_resume():699] starting system metrics thread 2024-02-27 10:01:27,234 INFO HandlerThread:1200 [system_monitor.py:start():194] Starting system monitor 2024-02-27 10:01:27,234 INFO SystemMonitor:1200 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-02-27 10:01:27,235 INFO SystemMonitor:1200 [interfaces.py:start():190] Started cpu monitoring 2024-02-27 10:01:27,235 INFO SystemMonitor:1200 [interfaces.py:start():190] Started disk monitoring 2024-02-27 10:01:27,236 INFO SystemMonitor:1200 [interfaces.py:start():190] Started gpu monitoring 2024-02-27 10:01:27,238 INFO SystemMonitor:1200 [interfaces.py:start():190] Started memory monitoring 2024-02-27 10:01:27,239 INFO SystemMonitor:1200 [interfaces.py:start():190] Started network monitoring 2024-02-27 10:01:27,507 DEBUG SenderThread:1200 [sender.py:send():382] send: telemetry 2024-02-27 10:01:27,508 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:01:27,508 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:01:27,620 INFO wandb-upload_0:1200 [upload_job.py:push():131] Uploaded file /tmp/tmp3h4j7i0dwandb/vib42cr8-wandb-metadata.json 2024-02-27 10:01:28,007 INFO Thread-12 :1200 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240227_100125-48zzkb8h/files/requirements.txt 2024-02-27 10:01:28,007 INFO Thread-12 :1200 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-metadata.json 2024-02-27 10:01:28,008 INFO Thread-12 :1200 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240227_100125-48zzkb8h/files/output.log 2024-02-27 10:01:30,006 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/output.log 2024-02-27 10:01:30,510 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:01:35,513 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:01:38,015 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/output.log 2024-02-27 10:01:40,657 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:01:42,182 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:01:42,183 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:01:42,185 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:01:45,366 DEBUG SenderThread:1200 [sender.py:send():382] send: config 2024-02-27 10:01:45,406 DEBUG SenderThread:1200 [sender.py:send():382] send: metric 2024-02-27 10:01:45,407 DEBUG SenderThread:1200 [sender.py:send():382] send: telemetry 2024-02-27 10:01:45,407 DEBUG SenderThread:1200 [sender.py:send():382] send: metric 2024-02-27 10:01:45,407 WARNING SenderThread:1200 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) 2024-02-27 10:01:45,407 DEBUG SenderThread:1200 [sender.py:send():382] send: telemetry 2024-02-27 10:01:46,407 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:01:51,408 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:01:56,416 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:01:57,028 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/config.yaml 2024-02-27 10:01:57,182 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:01:57,182 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:01:57,185 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:02:02,279 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:02:07,280 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:02:11,503 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:02:11,506 DEBUG SenderThread:1200 [sender.py:send():382] send: metric 2024-02-27 10:02:11,506 DEBUG SenderThread:1200 [sender.py:send():382] send: metric 2024-02-27 10:02:11,506 DEBUG SenderThread:1200 [sender.py:send():382] send: metric 2024-02-27 10:02:11,506 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:02:11,507 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:02:11,507 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:02:12,034 INFO Thread-12 :1200 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:02:12,182 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:02:12,182 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:02:12,185 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:02:12,317 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:02:17,318 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:02:22,318 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:02:27,182 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:02:27,182 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:02:27,223 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:02:27,239 DEBUG SystemMonitor:1200 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-02-27 10:02:27,286 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:02:28,293 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:02:29,040 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/config.yaml 2024-02-27 10:02:33,426 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:02:38,427 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:02:42,182 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:02:42,182 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:02:42,224 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:02:44,333 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:02:49,334 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:02:54,334 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:02:57,182 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:02:57,183 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:02:57,223 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:02:57,312 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:03:00,315 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:03:05,316 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:03:06,462 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:03:06,464 DEBUG SenderThread:1200 [sender.py:send():382] send: metric 2024-02-27 10:03:06,464 DEBUG SenderThread:1200 [sender.py:send():382] send: metric 2024-02-27 10:03:06,464 DEBUG SenderThread:1200 [sender.py:send():382] send: metric 2024-02-27 10:03:06,464 DEBUG SenderThread:1200 [sender.py:send():382] send: metric 2024-02-27 10:03:06,465 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:03:06,465 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:03:06,469 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:03:07,054 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:03:10,470 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:03:12,183 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:03:12,183 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:03:12,184 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:03:16,284 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:03:21,285 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:03:26,288 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:03:27,182 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:03:27,183 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:03:27,223 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:03:27,274 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:03:30,732 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:03:30,733 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:03:30,733 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:03:30,734 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:03:31,062 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:03:31,742 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:03:32,062 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/config.yaml 2024-02-27 10:03:36,895 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:03:41,903 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:03:42,183 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:03:42,183 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:03:42,186 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:03:47,340 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:03:52,341 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:03:57,183 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:03:57,183 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:03:57,224 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:03:57,345 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:03:57,347 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:04:02,346 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:04:07,347 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:04:12,183 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:04:12,184 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:04:12,224 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:04:13,324 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:04:18,324 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:04:23,325 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:04:25,110 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:04:25,112 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:04:25,112 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:04:25,114 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:04:26,100 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:04:27,183 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:04:27,184 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:04:27,186 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:04:27,343 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:04:28,344 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:04:33,345 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:04:38,346 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:04:42,183 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:04:42,183 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:04:42,224 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:04:44,334 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:04:49,335 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:04:49,920 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:04:49,921 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:04:49,922 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:04:49,924 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:04:50,109 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:04:54,925 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:04:57,183 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:04:57,183 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:04:57,186 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:04:57,336 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:05:00,337 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:05:05,337 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:05:10,339 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:05:12,183 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:05:12,184 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:05:12,224 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:05:16,304 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:05:21,305 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:05:26,306 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:05:27,183 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:05:27,184 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:05:27,224 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:05:27,344 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:05:31,345 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:05:36,346 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:05:41,347 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:05:42,183 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:05:42,184 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:05:42,224 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:05:43,513 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:05:43,514 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:05:43,515 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:05:43,518 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:05:44,128 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:05:46,518 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:05:51,519 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:05:56,520 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:05:57,183 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:05:57,184 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:05:57,187 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:05:57,352 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:06:02,354 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:06:07,355 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:06:08,348 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:06:08,349 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:06:08,349 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:06:08,350 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:06:09,139 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:06:12,184 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:06:12,184 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:06:12,188 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:06:13,291 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:06:18,292 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:06:23,292 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:06:27,184 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:06:27,184 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:06:27,225 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:06:27,323 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:06:28,324 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:06:33,325 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:06:38,326 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:06:42,184 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:06:42,184 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:06:42,225 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:06:44,312 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:06:47,254 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: pause 2024-02-27 10:06:47,254 INFO HandlerThread:1200 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-02-27 10:06:47,254 INFO HandlerThread:1200 [system_monitor.py:finish():203] Stopping system monitor 2024-02-27 10:06:47,254 DEBUG SystemMonitor:1200 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-02-27 10:06:47,254 DEBUG SystemMonitor:1200 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-02-27 10:06:47,256 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined cpu monitor 2024-02-27 10:06:47,256 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined disk monitor 2024-02-27 10:06:47,267 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined gpu monitor 2024-02-27 10:06:47,267 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined memory monitor 2024-02-27 10:06:47,267 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined network monitor 2024-02-27 10:06:47,268 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:06:50,269 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:06:51,622 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: resume 2024-02-27 10:06:51,622 INFO HandlerThread:1200 [handler.py:handle_request_resume():699] starting system metrics thread 2024-02-27 10:06:51,622 INFO HandlerThread:1200 [system_monitor.py:start():194] Starting system monitor 2024-02-27 10:06:51,623 INFO SystemMonitor:1200 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-02-27 10:06:51,623 INFO SystemMonitor:1200 [interfaces.py:start():190] Started cpu monitoring 2024-02-27 10:06:51,624 INFO SystemMonitor:1200 [interfaces.py:start():190] Started disk monitoring 2024-02-27 10:06:51,626 INFO SystemMonitor:1200 [interfaces.py:start():190] Started gpu monitoring 2024-02-27 10:06:51,632 INFO SystemMonitor:1200 [interfaces.py:start():190] Started memory monitoring 2024-02-27 10:06:51,633 INFO SystemMonitor:1200 [interfaces.py:start():190] Started network monitoring 2024-02-27 10:06:54,795 DEBUG SenderThread:1200 [sender.py:send():382] send: config 2024-02-27 10:06:54,796 DEBUG SenderThread:1200 [sender.py:send():382] send: metric 2024-02-27 10:06:54,797 DEBUG SenderThread:1200 [sender.py:send():382] send: metric 2024-02-27 10:06:54,797 WARNING SenderThread:1200 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) 2024-02-27 10:06:55,797 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:06:57,184 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:06:57,184 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:06:57,190 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:07:01,345 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:07:06,344 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:07:11,352 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:07:12,167 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/config.yaml 2024-02-27 10:07:12,200 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:07:12,200 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:07:12,237 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:07:17,361 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:07:22,361 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:07:27,197 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:07:27,197 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:07:27,237 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:07:28,339 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:07:33,340 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:07:38,341 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:07:42,196 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:07:42,197 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:07:42,237 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:07:44,290 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:07:49,291 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:07:51,634 DEBUG SystemMonitor:1200 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-02-27 10:07:51,635 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:07:54,636 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:07:57,196 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:07:57,197 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:07:57,237 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:08:00,311 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:08:05,312 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:08:10,312 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:08:12,196 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:08:12,197 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:08:12,237 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:08:14,743 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:08:14,744 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:08:14,747 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:08:14,748 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:08:15,193 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:08:15,748 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:08:20,749 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:08:21,636 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:08:26,637 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:08:27,196 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:08:27,197 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:08:27,200 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:08:32,294 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:08:37,295 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:08:42,197 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:08:42,197 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:08:42,237 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:08:42,326 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:08:47,326 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:08:51,636 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:08:52,637 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:08:57,197 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:08:57,197 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:08:57,238 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:08:58,286 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:09:03,287 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:09:08,287 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:09:12,197 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:09:12,197 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:09:12,238 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:09:13,360 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:09:18,361 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:09:21,637 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:09:23,638 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:09:27,197 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:09:27,198 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:09:27,238 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:09:29,335 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:09:33,975 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:09:33,977 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:09:33,977 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:09:33,980 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:09:34,228 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:09:34,980 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:09:39,981 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:09:42,197 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:09:42,198 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:09:42,202 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:09:45,363 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:09:50,363 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:09:51,638 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:09:55,639 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:09:57,197 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:09:57,198 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:09:57,238 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:10:01,289 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:10:06,290 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:10:11,290 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:10:12,197 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:10:12,198 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:10:12,241 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:10:16,338 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:10:21,339 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:10:21,639 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:10:26,648 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:10:27,197 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:10:27,198 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:10:27,238 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:10:32,370 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:10:37,371 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:10:42,204 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:10:42,204 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:10:42,245 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:10:43,294 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:10:48,295 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:10:51,640 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:10:52,451 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:10:52,452 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:10:52,453 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:10:52,454 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:10:53,255 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:10:53,455 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:10:57,198 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:10:57,199 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:10:57,209 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:10:59,317 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:11:04,318 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:11:09,319 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:11:12,198 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:11:12,198 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:11:12,239 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:11:15,278 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:11:20,279 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:11:21,640 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:11:25,641 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:11:27,198 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:11:27,198 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:11:27,240 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:11:27,717 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: pause 2024-02-27 10:11:27,717 INFO HandlerThread:1200 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-02-27 10:11:27,717 INFO HandlerThread:1200 [system_monitor.py:finish():203] Stopping system monitor 2024-02-27 10:11:27,717 DEBUG SystemMonitor:1200 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-02-27 10:11:27,717 DEBUG SystemMonitor:1200 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-02-27 10:11:27,728 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined cpu monitor 2024-02-27 10:11:27,729 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined disk monitor 2024-02-27 10:11:27,751 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined gpu monitor 2024-02-27 10:11:27,751 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined memory monitor 2024-02-27 10:11:27,751 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined network monitor 2024-02-27 10:11:27,752 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:11:30,754 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:11:35,754 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:11:36,983 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: resume 2024-02-27 10:11:36,983 INFO HandlerThread:1200 [handler.py:handle_request_resume():699] starting system metrics thread 2024-02-27 10:11:36,983 INFO HandlerThread:1200 [system_monitor.py:start():194] Starting system monitor 2024-02-27 10:11:36,997 INFO SystemMonitor:1200 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-02-27 10:11:37,003 INFO SystemMonitor:1200 [interfaces.py:start():190] Started cpu monitoring 2024-02-27 10:11:37,009 INFO SystemMonitor:1200 [interfaces.py:start():190] Started disk monitoring 2024-02-27 10:11:37,029 INFO SystemMonitor:1200 [interfaces.py:start():190] Started gpu monitoring 2024-02-27 10:11:37,044 INFO SystemMonitor:1200 [interfaces.py:start():190] Started memory monitoring 2024-02-27 10:11:37,050 INFO SystemMonitor:1200 [interfaces.py:start():190] Started network monitoring 2024-02-27 10:11:40,755 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:11:41,227 DEBUG SenderThread:1200 [sender.py:send():382] send: config 2024-02-27 10:11:41,229 DEBUG SenderThread:1200 [sender.py:send():382] send: metric 2024-02-27 10:11:41,229 DEBUG SenderThread:1200 [sender.py:send():382] send: metric 2024-02-27 10:11:41,229 WARNING SenderThread:1200 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) 2024-02-27 10:11:42,198 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:11:42,198 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:11:42,209 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:11:46,365 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:11:51,366 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:11:56,372 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:11:57,198 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:11:57,199 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:11:57,239 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:11:57,277 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/config.yaml 2024-02-27 10:12:02,340 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:12:07,341 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:12:12,198 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:12:12,199 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:12:12,239 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:12:13,328 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:12:18,329 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:12:23,329 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:12:27,198 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:12:27,199 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:12:27,239 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:12:28,356 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:12:33,357 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:12:37,051 DEBUG SystemMonitor:1200 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-02-27 10:12:37,052 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:12:39,053 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:12:42,198 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:12:42,199 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:12:42,239 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:12:44,273 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:12:49,274 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:12:54,274 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:12:57,199 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:12:57,199 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:12:57,239 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:12:59,370 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:13:04,370 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:13:07,053 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:13:10,054 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:13:12,199 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:13:12,199 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:13:12,240 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:13:15,306 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:13:20,307 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:13:25,307 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:13:27,199 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:13:27,199 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:13:27,240 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:13:30,352 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:13:35,353 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:13:37,053 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:13:41,054 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:13:42,199 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:13:42,199 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:13:42,240 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:13:45,495 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:13:45,496 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:13:45,497 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:13:45,497 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:13:46,320 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:13:46,498 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:13:51,499 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:13:56,500 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:13:57,199 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:13:57,199 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:13:57,211 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:14:02,268 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:14:07,054 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:14:08,055 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:14:12,199 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:14:12,200 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:14:12,240 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:14:13,364 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:14:18,364 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:14:23,367 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:14:27,199 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:14:27,200 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:14:27,240 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:14:29,334 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:14:34,335 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:14:37,055 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:14:39,149 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:14:39,150 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:14:39,150 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:14:39,154 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:14:39,344 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:14:40,155 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:14:42,199 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:14:42,200 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:14:42,212 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:14:45,305 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:14:50,306 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:14:55,306 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:14:57,199 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:14:57,200 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:14:57,240 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:15:00,364 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:15:05,365 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:15:07,056 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:15:11,057 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:15:12,200 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:15:12,200 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:15:12,241 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:15:16,306 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:15:21,307 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:15:26,308 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:15:27,200 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:15:27,200 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:15:27,241 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:15:31,332 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:15:36,333 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:15:37,056 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:15:42,057 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:15:42,200 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:15:42,200 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:15:42,241 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:15:47,338 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:15:52,338 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:15:57,200 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:15:57,200 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:15:57,241 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:15:58,297 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:16:03,297 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:16:07,057 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:16:09,058 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:16:12,200 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:16:12,201 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:16:12,241 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:16:14,370 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:16:19,370 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:16:24,371 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:16:27,200 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:16:27,202 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:16:27,241 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:16:30,279 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:16:35,280 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:16:37,058 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:16:41,059 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:16:42,200 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:16:42,201 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:16:42,241 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:16:42,632 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:16:42,633 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:16:42,634 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:16:42,634 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:16:43,401 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:16:46,635 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:16:51,636 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:16:56,637 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:16:57,200 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:16:57,201 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:16:57,212 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:17:02,303 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:17:07,059 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:17:08,060 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:17:12,200 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:17:12,201 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:17:12,241 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:17:13,316 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:17:18,317 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:17:23,317 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:17:27,201 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:17:27,201 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:17:27,242 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:17:29,304 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:17:34,305 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:17:36,315 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:17:36,316 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:17:36,317 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:17:36,317 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:17:36,419 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:17:37,060 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:17:40,061 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:17:42,201 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:17:42,201 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:17:42,213 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:17:45,355 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:17:50,356 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:17:55,357 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:17:57,201 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:17:57,201 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:17:57,242 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:18:01,285 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:18:06,286 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:18:07,061 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:18:12,062 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:18:12,201 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:18:12,201 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:18:12,243 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:18:17,330 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:18:22,331 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:18:27,201 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:18:27,201 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:18:27,242 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:18:28,287 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:18:33,290 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:18:37,062 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:18:39,062 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:18:42,201 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:18:42,201 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:18:42,242 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:18:44,319 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:18:49,323 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:18:54,323 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:18:57,201 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:18:57,202 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:18:57,242 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:18:59,348 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:19:04,349 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:19:07,063 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:19:10,067 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:19:12,201 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:19:12,202 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:19:12,242 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:19:15,368 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:19:20,369 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:19:25,369 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:19:27,201 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:19:27,202 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:19:27,242 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:19:31,353 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:19:36,354 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:19:37,063 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:19:40,952 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:19:40,953 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:19:40,953 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:19:40,954 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:19:41,471 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:19:41,954 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:19:42,201 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:19:42,202 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:19:42,215 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:19:47,806 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:19:52,807 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:19:57,202 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:19:57,202 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:19:57,242 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:19:58,364 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:20:03,365 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:20:07,064 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:20:09,065 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:20:12,202 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:20:12,202 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:20:12,243 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:20:14,309 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:20:19,310 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:20:24,312 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:20:27,202 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:20:27,202 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:20:27,243 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:20:29,376 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:20:34,376 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:20:34,552 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:20:34,553 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:20:34,553 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:20:34,557 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:20:35,490 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:20:37,065 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:20:40,066 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:20:42,202 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:20:42,202 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:20:42,215 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:20:45,301 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:20:50,301 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:20:55,310 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:20:57,202 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:20:57,202 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:20:57,243 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:21:00,310 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:21:05,311 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:21:07,068 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:21:11,069 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:21:12,202 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:21:12,202 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:21:12,243 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:21:16,361 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:21:21,361 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:21:26,362 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:21:27,202 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:21:27,203 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:21:27,243 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:21:32,283 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:21:37,066 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:21:38,067 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:21:42,202 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:21:42,203 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:21:42,243 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:21:43,288 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:21:48,288 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:21:53,289 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:21:57,202 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:21:57,203 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:21:57,243 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:21:58,376 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:22:03,376 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:22:07,067 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:22:09,068 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:22:12,202 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:22:12,203 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:22:12,243 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:22:14,373 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:22:19,377 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:22:24,378 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:22:27,202 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:22:27,203 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:22:27,243 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:22:30,361 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:22:35,361 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:22:37,068 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:22:38,377 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:22:38,379 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:22:38,379 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:22:38,379 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:22:38,537 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:22:40,380 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:22:42,203 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:22:42,203 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:22:42,216 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:22:46,314 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:22:51,315 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:22:56,316 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:22:57,203 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:22:57,203 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:22:57,244 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:23:02,306 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:23:07,069 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:23:08,069 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:23:12,203 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:23:12,203 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:23:12,244 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:23:13,272 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:23:18,273 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:23:23,273 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:23:27,203 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:23:27,203 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:23:27,244 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:23:28,290 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:23:31,978 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:23:31,979 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:23:31,979 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:23:31,981 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:23:32,560 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:23:33,982 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:23:37,069 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:23:39,070 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:23:42,203 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:23:42,204 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:23:42,217 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:23:44,333 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:23:49,334 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:23:54,335 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:23:57,203 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:23:57,204 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:23:57,244 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:23:59,340 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:24:04,341 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:24:07,070 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:24:10,071 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:24:12,203 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:24:12,204 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:24:12,244 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:24:15,276 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:24:20,277 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:24:25,278 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:24:27,203 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:24:27,204 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:24:27,244 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:24:30,308 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:24:35,308 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:24:37,071 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:24:41,072 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:24:42,203 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:24:42,204 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:24:42,244 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:24:46,319 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:24:51,319 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:24:56,320 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:24:57,204 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:24:57,204 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:24:57,244 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:25:01,365 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:25:06,366 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:25:07,072 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:25:12,073 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:25:12,204 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:25:12,204 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:25:12,245 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:25:17,334 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:25:22,335 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:25:27,204 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:25:27,204 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:25:27,245 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:25:27,340 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:25:32,341 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:25:35,176 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:25:35,177 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:25:35,177 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:25:35,178 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:25:35,607 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:25:37,073 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:25:38,073 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:25:42,204 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:25:42,204 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:25:42,218 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:25:43,350 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:25:48,351 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:25:53,351 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:25:57,204 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:25:57,205 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:25:57,245 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:25:59,323 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:26:04,324 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:26:07,073 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:26:10,074 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:26:12,204 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:26:12,205 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:26:12,245 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:26:15,339 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:26:20,340 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:26:25,341 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:26:27,204 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:26:27,205 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:26:27,245 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:26:28,763 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:26:28,764 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:26:28,765 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:26:28,765 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:26:29,629 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:26:30,766 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:26:35,767 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:26:37,074 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:26:41,075 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:26:42,204 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:26:42,205 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:26:42,218 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:26:46,915 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:26:51,916 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:26:56,917 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:26:57,204 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:26:57,205 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:26:57,246 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:27:02,370 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:27:07,075 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:27:08,076 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:27:12,205 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:27:12,205 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:27:12,246 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:27:13,355 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:27:18,355 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:27:23,356 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:27:27,205 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:27:27,205 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:27:27,246 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:27:29,305 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:27:34,306 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:27:37,076 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:27:40,076 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:27:42,205 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:27:42,205 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:27:42,246 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:27:45,357 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:27:50,358 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:27:55,358 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:27:57,205 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:27:57,205 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:27:57,246 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:28:00,370 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:28:05,371 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:28:07,076 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:28:11,077 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:28:12,205 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:28:12,206 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:28:12,246 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:28:16,377 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:28:21,378 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:28:26,379 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:28:27,205 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:28:27,206 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:28:27,246 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:28:32,369 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:28:32,503 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:28:32,505 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:28:32,505 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:28:32,505 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:28:32,676 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:28:37,077 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:28:38,078 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:28:42,205 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:28:42,206 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:28:42,219 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:28:46,094 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:28:51,095 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:28:56,096 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:28:57,205 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:28:57,206 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:28:57,246 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:29:01,379 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:29:06,380 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:29:07,078 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:29:12,079 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:29:12,206 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:29:12,206 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:29:12,246 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:29:17,335 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:29:22,335 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:29:26,060 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:29:26,062 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:29:26,062 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:29:26,063 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:29:26,695 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:29:27,206 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:29:27,206 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:29:27,219 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:29:27,380 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:29:32,381 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:29:37,079 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:29:38,079 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:29:42,206 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:29:42,206 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:29:42,247 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:29:43,341 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:29:48,342 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:29:53,343 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:29:57,206 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:29:57,206 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:29:57,248 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:29:59,274 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:30:04,279 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:30:07,079 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:30:10,080 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:30:12,206 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:30:12,206 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:30:12,247 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:30:15,356 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:30:20,357 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:30:25,358 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:30:27,206 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:30:27,206 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:30:27,247 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:30:31,340 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:30:36,340 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:30:37,080 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:30:42,081 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:30:42,206 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:30:42,207 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:30:42,247 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:30:47,297 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:30:52,297 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:30:57,206 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:30:57,207 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:30:57,247 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:30:57,302 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:31:02,303 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:31:07,081 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:31:08,082 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:31:12,206 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:31:12,207 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:31:12,247 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:31:16,162 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:31:21,164 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:31:26,164 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:31:27,211 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:31:27,212 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:31:27,252 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:31:29,791 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:31:29,792 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:31:29,793 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:31:29,794 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:31:30,743 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:31:31,795 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:31:36,798 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:31:37,082 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:31:42,083 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:31:42,208 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:31:42,209 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:31:42,222 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:31:47,388 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:31:52,389 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:31:57,209 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:31:57,209 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:31:57,249 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:31:58,372 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:32:03,372 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:32:07,082 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:32:09,083 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:32:12,209 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:32:12,209 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:32:12,249 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:32:14,336 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:32:19,337 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:32:23,445 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:32:23,446 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:32:23,447 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:32:23,447 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:32:23,762 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:32:24,448 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:32:27,209 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:32:27,209 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:32:27,223 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:32:30,388 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:32:35,389 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:32:37,083 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:32:41,084 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:32:42,209 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:32:42,209 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:32:42,250 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:32:46,386 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:32:51,387 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:32:56,388 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:32:57,209 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:32:57,209 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:32:57,250 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:33:02,310 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:33:07,084 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:33:08,085 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:33:12,209 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:33:12,210 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:33:12,250 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:33:13,357 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:33:18,358 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:33:23,359 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:33:27,209 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:33:27,210 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:33:27,250 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:33:29,343 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:33:34,343 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:33:37,085 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:33:40,086 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:33:42,209 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:33:42,210 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:33:42,250 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:33:46,051 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:33:51,052 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:33:56,053 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:33:57,209 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:33:57,210 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:33:57,250 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:34:01,330 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:34:06,330 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:34:07,085 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:34:12,087 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:34:12,209 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:34:12,210 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:34:12,250 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:34:17,372 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:34:22,373 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:34:27,210 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:34:27,210 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:34:27,250 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:34:27,687 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:34:27,688 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:34:27,688 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:34:27,689 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:34:27,689 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:34:27,806 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:34:32,690 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:34:37,086 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:34:38,087 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:34:42,210 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:34:42,210 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:34:42,224 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:34:43,371 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:34:48,372 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:34:53,372 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:34:57,210 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:34:57,210 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:34:57,251 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:34:58,377 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:35:03,377 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:35:07,087 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:35:09,088 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:35:12,210 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:35:12,210 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:35:12,251 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:35:14,317 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:35:19,317 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:35:21,359 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:35:21,360 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:35:21,360 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:35:21,361 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:35:21,826 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:35:24,362 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:35:27,210 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:35:27,210 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:35:27,224 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:35:30,357 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:35:35,358 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:35:37,088 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:35:41,089 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:35:42,210 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:35:42,211 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:35:42,251 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:35:46,288 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:35:51,288 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:35:56,289 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:35:57,210 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:35:57,211 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:35:57,251 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:36:01,329 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:36:06,330 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:36:07,093 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:36:12,094 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:36:12,210 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:36:12,211 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:36:12,252 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:36:17,284 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:36:22,285 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:36:27,210 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:36:27,211 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:36:27,251 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:36:28,284 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:36:33,285 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:36:37,089 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:36:39,090 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:36:42,211 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:36:42,211 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:36:42,251 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:36:44,383 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:36:49,383 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:36:54,385 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:36:57,211 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:36:57,211 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:36:57,251 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:37:00,354 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:37:05,354 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:37:07,090 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:37:11,091 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:37:12,211 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:37:12,211 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:37:12,252 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:37:16,384 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:37:21,384 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:37:24,653 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:37:24,654 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:37:24,655 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:37:24,655 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:37:24,873 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:37:26,656 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:37:27,211 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:37:27,211 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:37:27,229 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:37:32,383 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:37:37,091 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:37:38,092 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:37:42,211 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:37:42,211 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:37:42,252 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:37:43,326 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:37:48,326 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:37:53,327 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:37:57,211 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:37:57,211 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:37:57,252 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:37:58,359 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:38:03,361 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:38:07,092 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:38:09,092 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:38:12,211 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:38:12,211 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:38:12,252 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:38:14,318 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:38:18,355 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:38:18,356 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:38:18,357 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:38:18,359 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:38:18,891 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:38:19,360 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:38:24,361 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:38:27,211 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:38:27,212 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:38:27,230 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:38:29,383 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:38:34,384 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:38:37,093 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:38:40,093 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:38:42,211 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:38:42,212 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:38:42,252 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:38:45,351 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:38:50,351 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:38:55,352 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:38:57,211 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:38:57,212 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:38:57,252 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:39:01,305 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:39:06,305 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:39:07,093 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:39:12,094 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:39:12,212 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:39:12,212 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:39:12,252 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:39:17,323 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:39:22,324 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:39:27,212 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:39:27,212 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:39:27,253 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:39:27,389 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:39:32,390 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:39:37,094 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:39:38,095 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:39:42,212 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:39:42,212 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:39:42,253 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:39:43,303 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:39:48,303 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:39:53,304 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:39:57,212 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:39:57,212 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:39:57,253 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:39:58,350 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:40:03,350 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:40:07,095 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:40:09,096 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:40:12,212 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:40:12,212 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:40:12,253 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:40:14,309 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:40:19,309 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:40:21,374 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:40:21,375 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:40:21,376 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:40:21,376 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:40:21,942 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:40:24,377 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:40:27,212 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:40:27,213 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:40:27,231 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:40:29,386 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:40:34,387 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:40:37,095 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:40:40,096 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:40:42,212 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:40:42,212 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:40:42,253 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:40:45,377 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:40:50,378 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:40:55,379 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:40:57,212 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:40:57,213 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:40:57,253 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:41:01,339 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:41:06,339 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:41:07,096 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:41:12,097 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:41:12,212 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: stop_status 2024-02-27 10:41:12,213 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: stop_status 2024-02-27 10:41:12,253 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:41:15,085 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:41:16,231 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:41:16,231 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:41:16,232 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:41:16,961 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:41:17,233 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:41:22,234 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:41:27,234 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:41:32,235 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:41:33,188 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:41:33,189 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:41:33,189 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:41:37,097 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:41:38,098 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:41:38,427 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:41:43,099 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:41:44,397 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:41:48,100 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:41:49,818 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:41:49,818 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:41:53,101 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:41:57,370 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:41:58,101 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:42:02,410 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:42:03,102 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:42:04,818 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:42:07,098 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:42:09,099 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:42:10,412 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:42:14,100 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:42:15,413 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:42:19,100 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:42:19,818 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:42:24,101 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:42:25,419 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:42:29,102 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:42:30,420 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:42:34,103 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:42:34,818 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:42:37,102 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:42:39,103 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:42:40,421 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:42:44,104 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:42:45,422 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:42:49,105 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:42:49,818 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:42:54,106 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:42:55,424 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:42:59,107 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:43:00,425 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:43:04,108 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:43:04,818 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:43:07,099 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:43:10,100 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:43:10,426 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:43:15,101 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:43:15,427 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:43:19,819 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:43:20,102 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:43:25,102 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:43:25,429 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:43:30,103 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:43:30,430 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:43:34,819 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:43:35,104 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:43:37,100 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:43:40,433 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:43:41,101 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:43:45,433 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:43:46,102 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:43:49,819 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:43:51,103 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:43:55,435 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:43:56,104 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:44:00,436 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:44:01,104 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:44:03,451 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:44:03,453 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:44:03,453 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:44:03,454 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:44:04,032 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:44:04,819 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:44:06,455 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:44:07,101 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:44:10,437 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:44:12,109 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:44:15,438 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:44:17,110 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:44:19,819 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:44:22,110 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:44:25,440 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:44:27,111 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:44:30,441 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:44:32,112 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:44:34,819 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:44:37,101 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:44:38,102 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:44:40,442 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:44:43,103 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:44:45,443 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:44:48,104 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:44:49,819 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:44:53,105 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:44:55,445 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:44:57,251 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:44:57,252 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:44:57,253 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:44:57,253 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:44:58,050 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:44:58,254 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:45:00,445 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:45:03,255 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:45:04,820 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:45:07,102 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:45:09,103 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:45:10,447 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:45:14,104 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:45:15,448 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:45:19,105 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:45:19,820 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:45:24,105 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:45:25,449 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:45:29,106 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:45:30,465 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:45:34,107 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:45:34,820 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:45:37,103 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:45:40,104 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:45:40,464 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:45:45,105 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:45:45,464 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:45:49,820 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:45:50,105 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:45:55,106 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:45:55,466 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:46:00,107 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:46:00,467 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:46:04,820 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:46:05,108 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:46:07,104 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:46:10,468 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:46:11,105 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:46:15,469 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:46:16,105 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:46:19,820 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:46:21,106 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:46:25,471 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:46:26,107 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:46:30,472 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:46:31,108 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:46:34,821 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:46:36,109 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:46:37,104 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:46:40,473 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:46:42,105 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:46:45,474 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:46:47,106 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:46:49,821 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:46:52,107 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:46:55,475 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:46:57,108 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:47:00,476 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:47:00,888 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:47:00,889 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:47:00,890 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:47:00,890 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:47:01,093 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:47:02,891 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:47:04,821 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:47:07,105 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:47:08,106 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:47:10,478 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:47:13,110 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:47:15,478 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:47:18,111 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:47:19,821 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:47:23,112 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:47:25,480 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:47:28,113 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:47:30,481 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:47:33,114 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:47:34,821 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:47:37,106 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:47:39,107 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:47:40,482 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:47:44,109 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:47:45,483 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:47:49,110 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:47:49,821 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:47:54,111 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:47:54,492 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:47:54,493 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:47:54,494 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:47:54,494 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:47:55,112 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:47:55,485 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:47:59,496 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:48:00,486 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:48:04,496 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:48:04,821 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:48:07,106 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:48:10,107 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:48:10,487 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:48:15,108 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:48:15,488 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:48:19,821 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:48:20,109 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:48:25,110 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:48:25,489 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:48:30,111 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:48:30,490 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:48:34,821 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:48:35,112 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:48:37,107 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:48:40,492 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:48:41,108 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:48:45,494 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:48:46,109 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:48:49,822 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:48:51,110 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:48:55,497 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:48:56,111 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:49:00,498 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:49:01,112 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:49:04,822 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:49:06,113 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:49:07,108 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:49:10,500 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:49:12,109 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:49:15,501 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:49:17,110 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:49:19,822 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:49:22,111 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:49:25,502 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:49:27,112 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:49:30,503 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:49:32,114 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:49:34,822 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:49:37,109 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:49:38,110 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:49:40,504 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:49:43,110 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:49:45,505 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:49:48,111 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:49:49,822 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:49:53,117 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:49:55,507 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:49:57,245 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:49:57,247 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:49:57,247 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:49:57,248 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:49:58,157 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:49:58,248 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:50:00,508 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:50:03,249 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:50:04,822 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:50:07,110 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:50:09,110 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:50:10,509 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:50:14,111 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:50:15,510 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:50:19,112 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:50:19,822 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:50:24,113 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:50:25,512 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:50:29,114 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:50:30,513 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:50:34,115 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:50:34,822 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:50:37,110 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:50:40,111 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:50:40,514 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:50:45,112 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:50:45,515 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:50:49,822 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:50:50,113 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:50:50,985 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:50:50,987 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:50:50,987 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:50:50,992 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:50:51,176 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:50:55,517 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:50:55,993 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:51:00,517 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:51:00,994 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:51:04,823 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:51:05,995 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:51:07,111 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:51:10,519 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:51:11,112 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:51:15,520 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:51:16,113 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:51:19,823 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:51:21,114 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:51:25,521 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:51:26,123 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:51:30,522 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:51:31,124 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:51:34,826 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:51:36,125 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:51:37,112 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:51:40,524 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:51:42,113 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:51:45,525 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:51:47,114 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:51:49,824 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:51:52,115 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:51:55,526 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:51:57,115 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:52:00,527 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:52:02,116 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:52:04,824 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:52:07,113 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:52:08,113 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:52:10,529 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:52:13,114 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:52:15,529 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:52:18,115 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:52:19,824 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:52:23,116 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:52:25,531 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:52:28,117 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:52:30,533 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:52:33,118 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:52:34,824 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:52:37,113 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:52:39,114 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:52:40,535 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:52:44,115 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:52:45,542 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:52:49,116 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:52:49,824 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:52:54,117 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:52:54,318 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:52:54,320 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:52:54,320 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:52:54,320 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:52:55,218 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:52:55,543 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:52:59,321 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:53:00,544 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:53:04,322 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:53:04,824 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:53:07,114 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:53:10,115 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:53:10,546 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:53:15,116 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:53:15,547 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:53:19,824 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:53:20,117 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:53:25,118 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:53:25,549 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:53:30,119 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:53:30,549 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:53:34,824 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:53:35,120 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:53:37,115 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:53:40,551 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:53:41,116 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:53:45,552 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:53:46,117 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:53:47,958 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:53:47,960 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:53:47,960 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:53:47,961 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:53:48,237 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:53:49,824 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:53:51,962 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:53:55,554 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:53:56,962 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:54:00,554 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:54:01,963 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:54:04,825 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:54:06,964 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:54:07,116 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:54:10,556 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:54:12,117 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:54:15,557 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:54:17,117 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:54:19,825 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:54:22,118 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:54:25,558 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:54:27,119 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:54:30,559 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:54:32,120 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:54:34,825 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:54:37,116 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:54:38,117 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:54:40,560 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:54:43,118 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:54:45,561 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:54:48,119 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:54:49,825 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:54:53,119 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:54:55,563 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:54:58,120 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:55:00,563 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:55:03,125 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:55:04,825 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:55:07,117 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:55:09,118 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:55:10,565 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:55:14,119 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:55:15,566 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:55:19,119 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:55:19,825 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:55:24,120 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:55:25,567 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:55:29,121 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:55:30,568 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:55:34,122 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:55:34,825 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:55:37,118 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:55:40,119 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:55:40,570 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:55:45,120 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:55:45,571 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:55:49,825 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:55:50,121 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:55:50,979 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:55:50,981 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:55:50,981 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:55:50,982 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:55:51,291 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:55:55,572 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:55:55,983 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:56:00,574 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:56:00,983 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:56:04,825 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:56:05,984 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:56:07,119 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:56:10,576 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:56:11,120 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:56:15,577 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:56:16,120 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:56:19,825 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:56:21,121 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:56:25,579 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:56:26,122 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:56:30,580 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:56:31,123 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:56:34,826 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:56:36,124 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:56:37,119 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:56:40,581 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:56:42,121 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:56:44,655 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:56:44,656 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:56:44,656 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:56:44,657 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:56:44,662 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: partial_history 2024-02-27 10:56:44,665 DEBUG SenderThread:1200 [sender.py:send():382] send: metric 2024-02-27 10:56:44,665 DEBUG SenderThread:1200 [sender.py:send():382] send: metric 2024-02-27 10:56:44,666 DEBUG SenderThread:1200 [sender.py:send():382] send: metric 2024-02-27 10:56:44,666 DEBUG SenderThread:1200 [sender.py:send():382] send: metric 2024-02-27 10:56:44,666 DEBUG SenderThread:1200 [sender.py:send():382] send: metric 2024-02-27 10:56:44,666 DEBUG SenderThread:1200 [sender.py:send():382] send: history 2024-02-27 10:56:44,666 DEBUG SenderThread:1200 [sender.py:send_request():409] send_request: summary_record 2024-02-27 10:56:44,668 INFO SenderThread:1200 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-27 10:56:44,676 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: pause 2024-02-27 10:56:44,676 INFO HandlerThread:1200 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-02-27 10:56:44,676 INFO HandlerThread:1200 [system_monitor.py:finish():203] Stopping system monitor 2024-02-27 10:56:44,678 DEBUG SystemMonitor:1200 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-02-27 10:56:44,678 DEBUG SystemMonitor:1200 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-02-27 10:56:44,679 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined cpu monitor 2024-02-27 10:56:44,679 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined disk monitor 2024-02-27 10:56:44,694 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined gpu monitor 2024-02-27 10:56:44,694 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined memory monitor 2024-02-27 10:56:44,694 INFO HandlerThread:1200 [interfaces.py:finish():202] Joined network monitor 2024-02-27 10:56:44,695 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 10:56:45,309 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/wandb-summary.json 2024-02-27 10:56:45,582 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:56:47,695 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:56:49,826 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:56:52,696 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:56:55,584 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:56:57,698 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:57:00,585 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:57:02,705 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:57:04,826 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:57:05,318 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/config.yaml 2024-02-27 10:57:09,323 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:57:10,586 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:57:14,325 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:57:15,587 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:57:19,326 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:57:19,826 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:57:24,327 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:57:25,589 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:57:29,328 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:57:30,590 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:57:34,329 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:57:34,826 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:57:39,330 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:57:40,591 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:57:44,331 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:57:45,592 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:57:49,331 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:57:49,826 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:57:54,332 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:57:55,594 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:57:59,333 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:58:00,595 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:58:04,334 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:58:04,826 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:58:09,334 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:58:10,596 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:58:14,335 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:58:15,597 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:58:19,336 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:58:19,826 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:58:24,336 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:58:25,599 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:58:29,337 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:58:30,599 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:58:34,338 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:58:34,826 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:58:39,338 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:58:40,601 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:58:44,339 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:58:45,602 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:58:49,340 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:58:49,827 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:58:54,341 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:58:55,604 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:58:59,341 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:59:00,604 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:59:04,342 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:59:04,827 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:59:09,343 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:59:10,606 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:59:14,344 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:59:15,607 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:59:19,345 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:59:19,827 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:59:24,345 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:59:25,609 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:59:29,346 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:59:30,609 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:59:34,347 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:59:34,827 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:59:39,348 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:59:40,611 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:59:44,349 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:59:45,612 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:59:49,349 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:59:49,827 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 10:59:54,350 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 10:59:55,614 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 10:59:59,351 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:00:00,614 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:00:04,352 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:00:04,827 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:00:09,353 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:00:10,616 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:00:14,353 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:00:15,617 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:00:19,355 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:00:19,827 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:00:24,356 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:00:25,619 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:00:29,357 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:00:30,619 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:00:34,358 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:00:34,827 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:00:39,359 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:00:40,621 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:00:44,360 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:00:45,622 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:00:49,361 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:00:49,828 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:00:54,362 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:00:55,624 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:00:59,363 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:01:00,625 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:01:04,364 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:01:04,828 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:01:09,365 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:01:10,626 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:01:14,365 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:01:15,627 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:01:19,366 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:01:19,828 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:01:24,367 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:01:25,628 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:01:29,368 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:01:30,629 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:01:34,368 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:01:34,828 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:01:39,370 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:01:40,631 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:01:44,371 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:01:45,632 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:01:49,372 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:01:49,828 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:01:54,373 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:01:55,633 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:01:59,374 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:02:00,634 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:02:04,374 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:02:04,828 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:02:09,375 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:02:10,636 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:02:14,376 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:02:15,637 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:02:19,377 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:02:19,828 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:02:24,378 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:02:25,638 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:02:29,379 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:02:30,639 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:02:34,380 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:02:34,829 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:02:39,381 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:02:40,641 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:02:44,382 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:02:45,642 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:02:49,383 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:02:49,829 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:02:54,383 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:02:55,643 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:02:59,384 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:03:00,644 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:03:04,385 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:03:04,829 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:03:09,386 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:03:10,645 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:03:14,387 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:03:15,646 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:03:19,388 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:03:19,829 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:03:24,388 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:03:25,648 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:03:29,389 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:03:30,649 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:03:34,390 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:03:34,829 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:03:39,391 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:03:40,650 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:03:44,391 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:03:45,651 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:03:49,392 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:03:49,829 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:03:54,393 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:03:55,652 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:03:59,394 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:04:00,653 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:04:04,394 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:04:04,829 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:04:09,395 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:04:10,655 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:04:14,396 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:04:15,656 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:04:19,396 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:04:19,830 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:04:24,397 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:04:25,657 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:04:29,398 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:04:30,658 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:04:34,399 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:04:34,830 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:04:39,400 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:04:40,660 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:04:44,400 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:04:45,661 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:04:49,401 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:04:49,830 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:04:54,402 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:04:55,663 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:04:59,403 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:05:00,664 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:05:04,403 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:05:04,830 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:05:09,404 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:05:10,665 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:05:14,405 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:05:15,666 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:05:19,406 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:05:19,830 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:05:24,406 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:05:25,668 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:05:29,408 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:05:30,694 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:05:32,827 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: resume 2024-02-27 11:05:32,827 INFO HandlerThread:1200 [handler.py:handle_request_resume():699] starting system metrics thread 2024-02-27 11:05:32,827 INFO HandlerThread:1200 [system_monitor.py:start():194] Starting system monitor 2024-02-27 11:05:32,839 INFO SystemMonitor:1200 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-02-27 11:05:32,839 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: pause 2024-02-27 11:05:32,843 INFO HandlerThread:1200 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-02-27 11:05:32,843 INFO HandlerThread:1200 [system_monitor.py:finish():203] Stopping system monitor 2024-02-27 11:05:32,843 WARNING HandlerThread:1200 [interfaces.py:finish():207] Failed to finish cpu monitoring: cannot join thread before it is started 2024-02-27 11:05:32,850 INFO SystemMonitor:1200 [interfaces.py:start():190] Started cpu monitoring 2024-02-27 11:05:32,850 DEBUG SystemMonitor:1200 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-02-27 11:05:32,851 DEBUG SystemMonitor:1200 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-02-27 11:05:32,851 DEBUG SystemMonitor:1200 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-02-27 11:05:32,851 DEBUG SenderThread:1200 [sender.py:send():382] send: stats 2024-02-27 11:05:33,490 INFO Thread-12 :1200 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240227_100125-48zzkb8h/files/output.log 2024-02-27 11:05:34,830 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:05:34,852 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:05:39,853 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:05:40,701 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:05:44,854 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:05:45,702 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:05:49,830 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:05:49,855 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:05:54,855 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:05:55,703 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:05:59,856 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:06:00,704 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:06:04,831 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:06:04,857 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:06:09,858 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:06:10,706 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:06:14,859 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:06:15,706 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:06:19,831 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:06:19,860 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:06:24,860 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:06:25,708 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:06:29,861 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:06:30,709 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:06:34,831 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:06:34,862 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:06:39,863 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:06:40,710 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:06:44,864 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:06:45,711 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:06:49,831 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:06:49,864 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:06:54,865 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:06:55,713 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:06:59,866 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:07:00,714 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:07:04,831 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:07:04,867 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:07:09,868 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:07:10,715 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:07:14,868 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:07:15,716 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:07:19,831 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:07:19,869 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:07:24,869 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:07:25,718 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:07:29,870 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:07:30,719 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:07:34,831 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:07:34,871 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:07:39,872 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:07:40,720 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:07:44,873 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:07:45,721 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:07:49,831 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:07:49,874 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:07:54,875 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:07:55,723 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:07:59,875 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:08:00,724 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:08:04,832 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:08:04,876 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:08:09,877 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:08:10,726 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:08:14,878 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:08:15,727 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:08:19,832 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:08:19,879 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:08:24,880 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:08:25,728 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:08:29,881 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:08:30,729 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:08:34,832 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:08:34,881 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:08:39,883 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:08:40,731 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:08:44,883 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:08:45,732 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:08:49,832 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:08:49,884 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:08:54,885 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:08:55,733 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:08:59,886 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:09:00,734 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:09:04,832 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:09:04,886 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:09:09,887 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:09:10,736 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:09:14,888 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:09:15,737 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:09:19,832 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:09:19,889 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:09:24,890 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:09:25,738 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:09:29,890 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:09:30,739 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:09:34,832 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:09:34,891 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:09:39,892 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:09:40,741 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:09:44,893 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:09:45,742 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:09:49,833 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:09:49,893 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:09:54,895 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:09:55,743 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:09:59,895 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:10:00,744 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:10:04,833 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:10:04,896 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:10:09,897 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:10:10,746 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:10:14,898 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:10:15,747 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:10:19,833 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:10:19,899 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:10:24,900 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:10:25,748 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:10:29,900 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:10:30,749 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:10:34,833 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:10:34,901 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:10:39,902 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:10:40,751 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:10:44,903 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:10:45,752 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:10:49,833 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:10:49,904 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:10:54,905 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:10:55,753 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:10:59,906 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:11:00,754 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:11:04,833 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:11:04,906 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:11:09,907 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:11:10,756 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:11:14,908 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:11:15,757 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:11:19,833 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:11:19,909 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:11:24,910 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:11:25,759 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:11:29,910 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:11:30,760 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:11:34,833 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:11:34,911 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:11:39,912 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:11:40,762 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:11:44,913 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:11:45,763 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:11:49,833 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:11:49,914 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:11:54,914 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:11:55,764 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:11:59,915 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:12:00,765 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:12:04,833 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:12:04,916 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:12:09,917 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:12:10,767 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:12:14,917 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:12:15,768 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:12:19,834 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:12:19,918 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:12:24,919 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:12:25,769 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:12:29,920 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:12:30,770 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:12:34,834 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:12:34,920 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:12:39,921 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:12:40,772 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:12:44,922 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:12:45,773 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:12:49,834 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:12:49,923 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:12:54,924 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:12:55,774 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:12:59,925 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:13:00,775 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:13:04,834 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:13:04,925 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:13:09,926 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:13:10,777 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:13:14,927 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:13:15,778 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:13:19,834 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:13:19,928 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:13:24,929 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:13:25,780 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:13:29,929 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:13:30,781 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:13:34,834 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:13:34,930 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:13:39,931 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:13:40,783 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:13:44,932 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:13:45,784 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:13:49,834 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:13:49,933 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:13:54,934 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:13:55,785 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:13:59,934 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:14:00,787 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:14:04,835 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:14:04,935 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:14:09,936 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:14:10,788 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:14:14,937 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:14:15,789 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:14:19,835 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:14:19,937 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:14:24,938 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:14:25,791 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:14:29,940 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:14:30,792 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:14:34,835 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:14:34,941 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:14:39,942 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:14:40,793 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:14:44,942 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:14:45,794 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:14:49,835 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:14:49,943 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:14:54,944 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:14:55,796 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:14:59,945 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:15:00,798 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:15:02,717 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: resume 2024-02-27 11:15:02,717 INFO HandlerThread:1200 [handler.py:handle_request_resume():699] starting system metrics thread 2024-02-27 11:15:02,717 INFO HandlerThread:1200 [system_monitor.py:start():194] Starting system monitor 2024-02-27 11:15:02,720 INFO SystemMonitor:1200 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-02-27 11:15:02,733 INFO SystemMonitor:1200 [interfaces.py:start():190] Started cpu monitoring 2024-02-27 11:15:02,744 INFO SystemMonitor:1200 [interfaces.py:start():190] Started disk monitoring 2024-02-27 11:15:02,749 INFO SystemMonitor:1200 [interfaces.py:start():190] Started gpu monitoring 2024-02-27 11:15:02,756 INFO SystemMonitor:1200 [interfaces.py:start():190] Started memory monitoring 2024-02-27 11:15:02,763 INFO SystemMonitor:1200 [interfaces.py:start():190] Started network monitoring 2024-02-27 11:15:04,835 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:15:04,946 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:15:09,953 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:15:15,336 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:15:19,061 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:15:19,835 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:15:20,337 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:15:25,061 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:15:25,339 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:15:30,062 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:15:30,342 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:15:34,835 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:15:35,343 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:15:40,064 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:15:40,344 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:15:45,065 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:15:45,344 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:15:49,835 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-27 11:15:50,345 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:15:55,066 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:15:55,346 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:16:00,067 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: keepalive 2024-02-27 11:16:00,347 DEBUG HandlerThread:1200 [handler.py:handle_request():146] handle_request: status_report 2024-02-27 11:16:02,764 DEBUG SystemMonitor:1200 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-02-27 11:16:02,765 DEBUG SenderThread:1200 [sender.py:send():382] send: stats