2024-04-12 07:40:44,508 INFO    StreamThr :485 [internal.py:wandb_internal():86] W&B internal server running at pid: 485, started at: 2024-04-12 07:40:44.508065
2024-04-12 07:40:44,510 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status
2024-04-12 07:40:44,782 INFO    WriterThread:485 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240412_074044-widovcn3/run-widovcn3.wandb
2024-04-12 07:40:44,782 DEBUG   SenderThread:485 [sender.py:send():379] send: header
2024-04-12 07:40:44,785 DEBUG   SenderThread:485 [sender.py:send():379] send: run
2024-04-12 07:40:44,894 INFO    SenderThread:485 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240412_074044-widovcn3/files
2024-04-12 07:40:44,895 INFO    SenderThread:485 [sender.py:_start_run_threads():1124] run started: widovcn3 with start time 1712907644.507718
2024-04-12 07:40:44,903 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: check_version
2024-04-12 07:40:44,904 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: check_version
2024-04-12 07:40:44,996 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: run_start
2024-04-12 07:40:45,008 DEBUG   HandlerThread:485 [system_info.py:__init__():26] System info init
2024-04-12 07:40:45,008 DEBUG   HandlerThread:485 [system_info.py:__init__():41] System info init done
2024-04-12 07:40:45,008 INFO    HandlerThread:485 [system_monitor.py:start():194] Starting system monitor
2024-04-12 07:40:45,008 INFO    SystemMonitor:485 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-04-12 07:40:45,008 INFO    HandlerThread:485 [system_monitor.py:probe():214] Collecting system info
2024-04-12 07:40:45,009 INFO    SystemMonitor:485 [interfaces.py:start():190] Started cpu monitoring
2024-04-12 07:40:45,009 INFO    SystemMonitor:485 [interfaces.py:start():190] Started disk monitoring
2024-04-12 07:40:45,011 INFO    SystemMonitor:485 [interfaces.py:start():190] Started gpu monitoring
2024-04-12 07:40:45,012 INFO    SystemMonitor:485 [interfaces.py:start():190] Started memory monitoring
2024-04-12 07:40:45,013 INFO    SystemMonitor:485 [interfaces.py:start():190] Started network monitoring
2024-04-12 07:40:45,023 DEBUG   HandlerThread:485 [system_info.py:probe():150] Probing system
2024-04-12 07:40:45,026 DEBUG   HandlerThread:485 [gitlib.py:_init_repo():56] git repository is invalid
2024-04-12 07:40:45,026 DEBUG   HandlerThread:485 [system_info.py:probe():198] Probing system done
2024-04-12 07:40:45,026 DEBUG   HandlerThread:485 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-04-12T07:40:45.023659', 'startedAt': '2024-04-12T07:40:44.501264', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': 'e5a48bec8248', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.138, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.138, 'min': 0.0, 'max': 0.0}, {'current': 2000.138, 'min': 0.0, 'max': 0.0}, {'current': 2000.138, 'min': 0.0, 'max': 0.0}, {'current': 2000.138, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5565.782573699951}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}}
2024-04-12 07:40:45,026 INFO    HandlerThread:485 [system_monitor.py:probe():224] Finished collecting system info
2024-04-12 07:40:45,026 INFO    HandlerThread:485 [system_monitor.py:probe():227] Publishing system info
2024-04-12 07:40:45,026 DEBUG   HandlerThread:485 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-04-12 07:40:45,897 INFO    Thread-12 :485 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/conda-environment.yaml
2024-04-12 07:41:00,040 ERROR   HandlerThread:485 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda
    subprocess.call(
  File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call
    return p.wait(timeout=timeout)
  File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait
    return self._wait(timeout=timeout)
  File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait
    raise TimeoutExpired(self.args, timeout)
subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds
2024-04-12 07:41:00,041 DEBUG   HandlerThread:485 [system_info.py:_save_conda():222] Saving conda packages done
2024-04-12 07:41:00,042 INFO    HandlerThread:485 [system_monitor.py:probe():229] Finished publishing system info
2024-04-12 07:41:00,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:00,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: keepalive
2024-04-12 07:41:00,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:00,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: keepalive
2024-04-12 07:41:00,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:00,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: keepalive
2024-04-12 07:41:00,048 DEBUG   SenderThread:485 [sender.py:send():379] send: files
2024-04-12 07:41:00,049 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now
2024-04-12 07:41:00,245 INFO    wandb-upload_0:485 [upload_job.py:push():131] Uploaded file /tmp/tmpqpxjmay6wandb/iki4gmew-wandb-metadata.json
2024-04-12 07:41:00,900 INFO    Thread-12 :485 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-metadata.json
2024-04-12 07:41:01,034 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: python_packages
2024-04-12 07:41:01,034 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: python_packages
2024-04-12 07:41:01,038 DEBUG   SenderThread:485 [sender.py:send():379] send: telemetry
2024-04-12 07:41:01,043 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:41:01,043 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:41:01,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:41:01,102 DEBUG   SenderThread:485 [sender.py:send():379] send: config
2024-04-12 07:41:01,103 DEBUG   SenderThread:485 [sender.py:send():379] send: metric
2024-04-12 07:41:01,103 DEBUG   SenderThread:485 [sender.py:send():379] send: telemetry
2024-04-12 07:41:01,104 DEBUG   SenderThread:485 [sender.py:send():379] send: metric
2024-04-12 07:41:01,104 WARNING SenderThread:485 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen)
2024-04-12 07:41:01,104 DEBUG   SenderThread:485 [sender.py:send():379] send: telemetry
2024-04-12 07:41:01,900 INFO    Thread-12 :485 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:41:01,901 INFO    Thread-12 :485 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/requirements.txt
2024-04-12 07:41:03,901 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:41:05,568 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:10,569 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:15,575 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:15,907 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/config.yaml
2024-04-12 07:41:16,035 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:41:16,036 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:41:16,036 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:41:21,106 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:22,063 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:41:22,066 DEBUG   SenderThread:485 [sender.py:send():379] send: metric
2024-04-12 07:41:22,066 DEBUG   SenderThread:485 [sender.py:send():379] send: metric
2024-04-12 07:41:22,066 DEBUG   SenderThread:485 [sender.py:send():379] send: metric
2024-04-12 07:41:22,066 DEBUG   SenderThread:485 [sender.py:send():379] send: metric
2024-04-12 07:41:22,066 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:41:22,066 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:41:22,068 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:41:22,910 INFO    Thread-12 :485 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:41:23,910 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:41:26,797 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:31,036 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:41:31,036 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:41:31,037 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:41:32,086 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:37,087 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:42,088 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:45,014 DEBUG   SystemMonitor:485 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-04-12 07:41:45,015 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:41:46,035 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:41:46,035 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:41:46,076 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:41:46,285 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:41:46,286 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:41:46,287 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:41:46,287 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:41:46,921 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:41:47,922 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:41:48,004 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:48,922 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/config.yaml
2024-04-12 07:41:53,106 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:58,107 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:01,035 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:42:01,036 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:42:01,038 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:42:03,131 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:08,132 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:09,300 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:42:09,301 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:42:09,301 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:42:09,303 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:42:09,932 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:42:11,933 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:42:14,018 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:15,016 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:42:16,035 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:42:16,036 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:42:16,039 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:42:19,086 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:24,087 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:29,088 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:31,035 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:42:31,036 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:42:31,076 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:42:34,125 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:35,366 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:42:35,367 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:42:35,367 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:42:35,369 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:42:35,942 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:42:37,943 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:42:40,060 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:45,017 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:42:46,018 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:46,035 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:42:46,036 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:42:46,039 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:42:51,081 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:56,082 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:00,593 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:43:00,594 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:43:00,594 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:43:00,595 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:43:00,953 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:43:01,036 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:43:01,037 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:43:01,039 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:43:01,159 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:03,954 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:43:06,305 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:11,305 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:15,018 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:43:16,036 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:43:16,037 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:43:16,039 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:43:17,087 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:22,088 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:25,211 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:43:25,212 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:43:25,213 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:43:25,213 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:43:25,965 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:43:27,934 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:27,966 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:43:31,036 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:43:31,037 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:43:31,037 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:43:33,175 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:38,176 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:43,177 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:45,019 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:43:46,036 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:43:46,037 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:43:46,077 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:43:47,760 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:43:47,761 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:43:47,761 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:43:47,763 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:43:47,973 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:43:48,211 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:49,974 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:43:53,212 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:58,213 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:01,043 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:44:01,043 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:44:01,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:44:04,068 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:09,069 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:14,070 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:15,021 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:44:16,039 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:44:16,041 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:44:16,041 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:44:16,358 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:44:16,359 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:44:16,359 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:44:16,360 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:44:16,986 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:44:17,987 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:44:19,084 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:24,085 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:29,086 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:31,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:44:31,042 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:44:31,043 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:44:34,122 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:39,122 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:39,436 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:44:39,437 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:44:39,437 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:44:39,439 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:44:39,995 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:44:41,996 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:44:44,242 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:45,023 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:44:46,039 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:44:46,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:44:46,041 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:44:50,094 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:55,095 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:00,097 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:01,039 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:45:01,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:45:01,041 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:45:03,599 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:45:03,600 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:45:03,600 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:45:03,601 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:45:04,005 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:45:05,363 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:06,006 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:45:10,364 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:15,024 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:45:16,025 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:16,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:45:16,043 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:45:16,043 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:45:21,075 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:24,892 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:45:24,894 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:45:24,894 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:45:24,896 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:45:25,016 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:45:26,600 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:28,017 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:45:31,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:45:31,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:45:31,041 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:45:32,105 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:37,105 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:42,106 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:45,025 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:45:46,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:45:46,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:45:46,041 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:45:47,112 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:48,638 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:45:48,639 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:45:48,639 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:45:48,640 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:45:49,027 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:45:50,028 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:45:52,334 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:57,334 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:01,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:46:01,041 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:46:01,042 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:46:03,170 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:08,171 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:13,173 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:15,026 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:46:16,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:46:16,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:46:16,041 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:46:16,643 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:46:16,644 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:46:16,644 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:46:16,645 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:46:17,039 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:46:18,040 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:46:18,335 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:23,336 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:28,337 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:31,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:46:31,041 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:46:31,042 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:46:34,086 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:39,086 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:44,087 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:45,027 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:46:46,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:46:46,041 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:46:46,042 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:46:46,101 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:46:46,159 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:46:46,159 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:46:46,160 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:46:47,054 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:46:48,054 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:46:49,803 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:54,804 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:59,805 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:01,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:47:01,041 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:47:01,042 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:47:05,173 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:10,173 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:11,145 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:47:11,147 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:47:11,147 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:47:11,147 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:47:12,063 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:47:14,064 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:47:15,028 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:47:16,028 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:16,041 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:47:16,041 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:47:16,042 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:47:21,127 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:26,128 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:31,041 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:47:31,041 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:47:31,042 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:47:32,121 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:35,137 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:47:35,138 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:47:35,138 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:47:35,139 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:47:36,075 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:47:37,863 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:38,076 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:47:42,864 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:45,029 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:47:46,042 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:47:46,044 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:47:46,044 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:47:48,168 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:53,169 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:58,170 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:00,044 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:48:00,045 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:48:00,046 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:48:00,046 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:48:00,086 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:48:01,050 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:48:01,050 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:48:01,051 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:48:02,087 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:48:04,168 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:09,168 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:14,169 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:15,034 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:48:16,045 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:48:16,045 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:48:16,085 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:48:20,145 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:25,146 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:25,939 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:48:25,941 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:48:25,941 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:48:25,941 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:48:26,098 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:48:28,098 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:48:30,668 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:31,045 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:48:31,046 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:48:31,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:48:36,131 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:41,132 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:45,031 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:48:46,045 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:48:46,046 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:48:46,046 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:48:46,174 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:49,770 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:48:49,772 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:48:49,772 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:48:49,774 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:48:50,107 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:48:51,487 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:53,109 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:48:56,488 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:01,046 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:49:01,046 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:49:01,047 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:49:02,083 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:07,084 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:12,085 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:15,032 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:49:15,266 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:49:15,267 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:49:15,267 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:49:15,268 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:49:16,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:49:16,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:49:16,048 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:49:16,119 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:49:17,088 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:19,120 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:49:22,090 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:27,091 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:31,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:49:31,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:49:31,048 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:49:32,150 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:37,151 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:38,408 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:49:38,409 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:49:38,409 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:49:38,412 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:49:39,128 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:49:41,129 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:49:42,654 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:45,033 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:49:46,049 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:49:46,049 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:49:46,050 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:49:48,164 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:53,165 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:58,166 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:01,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:50:01,047 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:50:01,076 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:50:02,019 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:50:02,021 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:50:02,021 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:50:02,021 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:50:02,136 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:50:03,788 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:05,137 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:50:08,789 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:13,790 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:15,034 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:50:16,046 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:50:16,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:50:16,047 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:50:19,090 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:23,388 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:50:23,389 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:50:23,390 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:50:23,390 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:50:24,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:24,184 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:50:27,186 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:50:29,159 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:31,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:50:31,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:50:31,048 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:50:35,072 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:40,073 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:45,035 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:50:46,036 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:46,046 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:50:46,046 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:50:46,087 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:50:50,249 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:50:50,250 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:50:50,251 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:50:50,251 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:50:51,194 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:50:51,950 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:53,195 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:50:56,951 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:01,049 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:51:01,049 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:51:01,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:51:02,168 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:07,170 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:12,170 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:15,036 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:51:15,993 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:51:15,994 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:51:15,995 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:51:15,995 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:51:16,046 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:51:16,047 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:51:16,049 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:51:16,205 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:51:17,738 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:19,207 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:51:22,739 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:27,740 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:31,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:51:31,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:51:31,048 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:51:33,136 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:38,137 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:43,138 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:43,648 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:51:43,649 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:51:43,649 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:51:43,649 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:51:44,219 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:51:45,038 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:51:46,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:51:46,047 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:51:46,050 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:51:47,220 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:51:49,131 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:54,132 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:59,133 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:01,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:52:01,048 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:52:01,088 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:52:05,109 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:09,162 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:52:09,163 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:52:09,163 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:52:09,163 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:52:09,228 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:52:10,896 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:11,229 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:52:15,039 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:52:16,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:16,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:52:16,048 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:52:16,050 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:52:21,157 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:26,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:31,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:52:31,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:52:31,049 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:52:32,074 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:34,130 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:52:34,132 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:52:34,132 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:52:34,132 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:52:34,237 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:52:37,238 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:52:37,867 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:42,868 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:45,040 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:52:46,049 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:52:46,050 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:52:46,050 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:52:48,079 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:53,080 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:57,000 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:52:57,001 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:52:57,002 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:52:57,002 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:52:57,246 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:52:58,985 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:01,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:53:01,048 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:53:01,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:53:01,247 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:53:04,136 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:09,137 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:14,137 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:15,041 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:53:16,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:53:16,048 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:53:16,089 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:53:20,128 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:20,631 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:53:20,633 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:53:20,633 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:53:20,633 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:53:21,254 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:53:23,255 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:53:25,336 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:30,336 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:31,050 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:53:31,050 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:53:31,091 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:53:36,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:41,159 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:43,275 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:53:43,277 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:53:43,277 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:53:43,277 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:53:44,262 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:53:45,042 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:53:46,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:53:46,049 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:53:46,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:53:47,104 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:47,264 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:53:52,105 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:57,106 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:01,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:54:01,048 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:54:01,089 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:54:02,130 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:07,131 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:07,631 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:54:07,632 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:54:07,633 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:54:07,633 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:54:08,272 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:54:11,274 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:54:12,325 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:15,044 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:54:16,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:54:16,049 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:54:16,049 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:54:18,135 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:23,136 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:28,137 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:28,915 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:54:28,916 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:54:28,917 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:54:28,917 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:54:29,280 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:54:31,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:54:31,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:54:31,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:54:31,281 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:54:34,099 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:39,100 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:44,101 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:45,045 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:54:46,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:54:46,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:54:46,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:54:49,133 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:54,134 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:55,471 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:54:55,473 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:54:55,473 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:54:55,474 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:54:56,290 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:54:59,163 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:59,292 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:55:01,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:55:01,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:55:01,086 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:55:05,086 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:10,087 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:15,047 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:55:16,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:16,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:55:16,051 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:55:16,091 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:55:21,039 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:55:21,041 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:55:21,041 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:55:21,041 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:55:21,301 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:55:21,770 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:23,302 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:55:26,771 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:31,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:55:31,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:55:31,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:55:32,156 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:37,157 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:42,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:44,185 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:55:44,186 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:55:44,187 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:55:44,187 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:55:44,311 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:55:45,047 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:55:46,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:55:46,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:55:46,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:55:47,185 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:47,312 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:55:52,186 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:57,187 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:01,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:56:01,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:56:01,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:56:03,130 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:08,131 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:11,587 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:56:11,588 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:56:11,589 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:56:11,589 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:56:12,321 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:56:13,299 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:15,049 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:56:15,322 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:56:16,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:56:16,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:56:16,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:56:19,157 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:24,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:29,159 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:31,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:56:31,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:56:31,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:56:33,453 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:56:33,454 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:56:33,455 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:56:33,455 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:56:34,183 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:34,330 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:56:35,330 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:56:39,184 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:44,185 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:45,050 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:56:46,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:56:46,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:56:46,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:56:50,157 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:54,460 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:56:54,461 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:56:54,461 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:56:54,462 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:56:55,183 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:55,339 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:56:57,340 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:57:00,184 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:01,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:57:01,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:57:01,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:57:06,080 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:11,081 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:15,054 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:57:16,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:57:16,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:57:16,053 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:57:17,080 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:19,400 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:57:19,401 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:57:19,401 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:57:19,402 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:57:20,350 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:57:21,350 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:57:22,131 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:27,132 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:31,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:57:31,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:57:31,053 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:57:33,082 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:38,083 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:43,084 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:43,651 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:57:43,652 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:57:43,652 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:57:43,653 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:57:44,359 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:57:45,052 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:57:46,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:57:46,054 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:57:46,054 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:57:47,361 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:57:48,095 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:53,096 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:58,097 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:01,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:58:01,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:58:01,053 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:58:03,104 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:06,143 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:58:06,144 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:58:06,144 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:58:06,145 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:58:06,368 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:58:08,860 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:09,370 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:58:13,861 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:15,053 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:58:16,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:58:16,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:58:16,053 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:58:19,086 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:24,087 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:29,088 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:31,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:58:31,054 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:58:31,054 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:58:31,420 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:58:31,421 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:58:31,422 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:58:31,424 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:58:32,378 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:58:33,379 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:58:34,218 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:39,219 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:44,220 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:45,054 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:58:46,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:58:46,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:58:46,054 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:58:50,168 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:55,013 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:58:55,014 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:58:55,014 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:58:55,015 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:58:55,388 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:58:55,747 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:57,388 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:59:00,748 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:01,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:59:01,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:59:01,054 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:59:06,180 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:11,181 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:15,056 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:59:16,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:59:16,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:59:16,054 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:59:17,156 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:19,267 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:59:19,268 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:59:19,268 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:59:19,269 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:59:19,398 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:59:21,398 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:59:22,993 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:27,995 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:31,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:59:31,054 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:59:31,054 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:59:33,153 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:38,154 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:43,155 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:45,057 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:59:45,600 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:59:45,602 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:59:45,603 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:59:45,603 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:59:46,291 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:59:46,292 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:59:46,292 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:59:46,409 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:59:47,409 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:59:48,396 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:53,397 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:58,398 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:01,288 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:00:01,289 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:00:01,290 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:00:04,365 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:09,366 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:11,984 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:00:11,985 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:00:11,985 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:00:11,986 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:00:12,418 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:00:14,698 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:15,058 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:00:15,419 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:00:16,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:00:16,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:00:16,288 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:00:20,422 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:25,423 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:30,424 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:31,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:00:31,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:00:31,288 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:00:36,160 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:00:36,161 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:00:36,161 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:00:36,162 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:36,162 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:00:36,427 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:00:39,428 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:00:42,012 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:45,059 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:00:46,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:00:46,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:00:46,288 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:00:47,379 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:52,380 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:57,381 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:01,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:01:01,288 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:01:01,288 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:01:02,667 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:01:02,668 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:01:02,668 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:01:02,668 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:02,669 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:01:03,437 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:01:05,439 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:01:08,417 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:13,418 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:15,060 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:01:16,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:01:16,287 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:01:16,288 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:01:19,406 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:24,407 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:25,810 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:01:25,811 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:01:25,812 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:01:25,812 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:01:26,449 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:01:27,449 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:01:29,572 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:31,289 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:01:31,290 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:01:31,292 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:01:35,396 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:40,397 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:45,061 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:01:46,062 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:46,289 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:01:46,289 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:01:46,290 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:01:51,408 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:52,071 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:01:52,072 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:01:52,073 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:01:52,073 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:01:52,460 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:01:55,461 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:01:56,785 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:01,289 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:02:01,290 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:02:01,290 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:02:02,404 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:07,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:12,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:15,062 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:02:16,240 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:02:16,241 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:02:16,242 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:02:16,242 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:02:16,288 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:02:16,288 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:02:16,290 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:02:16,469 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:02:17,979 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:19,471 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:02:22,980 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:27,981 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:31,288 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:02:31,288 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:02:31,329 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:02:33,409 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:38,410 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:39,529 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:02:39,531 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:02:39,531 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:02:39,531 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:02:40,479 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:02:41,479 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:02:44,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:45,064 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:02:46,288 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:02:46,289 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:02:46,329 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:02:49,362 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:54,363 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:59,364 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:01,288 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:03:01,289 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:03:01,329 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:03:05,330 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:07,184 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:03:07,185 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:03:07,186 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:03:07,186 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:03:07,490 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:03:09,491 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:03:10,947 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:15,065 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:03:16,066 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:16,288 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:03:16,288 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:03:16,291 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:03:21,356 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:26,357 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:30,601 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:03:30,602 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:03:30,602 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:03:30,603 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:03:31,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:03:31,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:03:31,313 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:03:31,385 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:31,501 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:03:33,502 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:03:36,386 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:41,387 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:45,066 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:03:46,307 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:03:46,308 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:03:46,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:03:47,354 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:51,370 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:03:51,372 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:03:51,372 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:03:51,372 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:03:51,510 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:03:53,129 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:53,510 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:03:58,130 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:01,308 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:04:01,308 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:04:01,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:04:03,360 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:08,361 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:13,362 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:15,067 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:04:16,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:04:16,309 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:04:16,350 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:04:16,758 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:04:16,760 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:04:16,760 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:04:16,761 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:04:17,520 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:04:18,510 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:19,521 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:04:23,511 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:28,512 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:31,308 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:04:31,309 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:04:31,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:04:34,356 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:39,357 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:42,654 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:04:42,656 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:04:42,656 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:04:42,656 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:04:43,531 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:04:44,365 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:45,069 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:04:45,532 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:04:46,308 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:04:46,308 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:04:46,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:04:50,358 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:55,358 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:00,359 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:01,308 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:05:01,309 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:05:01,349 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:05:05,420 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:10,421 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:10,582 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:05:10,583 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:05:10,583 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:05:10,584 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:05:11,543 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:05:13,544 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:05:15,070 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:05:16,071 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:16,308 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:05:16,309 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:05:16,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:05:21,394 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:26,395 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:31,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:05:31,309 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:05:31,335 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:05:32,335 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:32,484 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:05:32,485 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:05:32,486 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:05:32,486 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:05:32,553 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:05:35,554 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:05:37,389 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:42,390 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:45,071 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:05:46,308 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:05:46,309 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:05:46,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:05:47,420 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:52,421 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:55,258 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:05:55,260 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:05:55,260 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:05:55,261 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:05:55,562 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:05:57,563 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:05:58,003 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:01,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:06:01,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:06:01,310 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:06:03,389 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:08,390 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:13,391 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:15,077 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:06:16,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:06:16,310 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:06:16,350 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:06:18,427 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:20,569 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:06:20,570 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:06:20,571 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:06:20,571 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:06:20,574 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:06:23,576 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:06:24,347 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:29,348 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:31,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:06:31,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:06:31,310 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:06:34,356 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:39,357 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:44,358 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:45,073 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:06:46,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:06:46,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:06:46,311 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:06:47,067 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:06:47,069 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:06:47,069 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:06:47,071 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:06:47,586 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:06:49,586 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:06:49,816 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:54,817 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:59,818 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:01,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:07:01,310 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:07:01,350 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:07:05,429 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:10,429 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:12,553 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:07:12,555 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:07:12,556 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:07:12,556 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:07:12,596 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:07:15,075 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:07:15,597 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:07:16,075 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:16,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:07:16,310 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:07:16,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:07:21,416 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:26,417 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:31,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:07:31,310 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:07:31,351 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:07:32,393 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:37,394 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:37,530 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:07:37,531 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:07:37,531 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:07:37,534 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:07:37,605 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:07:39,606 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:07:43,266 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:45,076 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:07:46,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:07:46,310 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:07:46,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:07:48,431 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:53,432 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:58,433 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:01,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:08:01,311 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:08:01,344 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:08:03,838 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:08:03,839 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:08:03,840 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:08:03,840 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:03,840 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:08:04,615 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:08:05,616 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:08:09,549 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:14,550 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:15,077 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:08:16,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:08:16,311 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:08:16,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:08:20,406 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:25,407 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:26,840 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:08:26,841 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:08:26,842 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:08:26,842 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:08:27,624 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:08:29,625 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:08:30,589 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:31,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:08:31,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:08:31,312 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:08:36,367 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:41,368 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:45,078 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:08:46,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:08:46,311 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:08:46,343 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:08:47,343 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:52,339 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:08:52,340 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:08:52,340 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:08:52,341 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:08:52,633 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:08:53,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:53,634 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:08:58,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:01,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:09:01,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:09:01,312 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:09:03,441 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:08,442 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:13,443 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:15,080 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:09:16,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:09:16,312 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:09:16,352 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:09:16,855 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:09:16,856 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:09:16,856 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:09:16,857 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:09:17,643 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:09:18,845 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:19,644 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:09:23,846 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:28,847 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:31,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:09:31,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:09:31,313 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:09:34,363 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:37,604 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:09:37,606 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:09:37,606 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:09:37,606 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:09:37,652 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:09:39,429 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:39,653 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:09:44,429 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:45,081 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:09:46,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:09:46,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:09:46,313 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:09:49,432 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:54,433 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:58,710 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:09:58,711 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:09:58,712 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:09:58,712 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:09:59,661 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:10:00,431 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:01,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:10:01,312 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:10:01,315 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:10:01,662 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:10:06,370 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:11,371 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:15,081 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:10:16,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:10:16,312 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:10:16,352 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:10:16,440 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:21,441 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:22,705 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:10:22,706 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:10:22,707 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:10:22,707 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:10:23,670 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:10:25,671 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:10:27,429 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:31,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:10:31,314 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:10:31,354 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:10:33,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:38,406 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:43,407 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:45,083 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:10:46,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:10:46,313 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:10:46,353 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:10:48,434 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:49,687 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:10:49,688 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:10:49,688 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:10:49,689 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:10:50,682 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:10:51,682 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:10:54,428 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:59,428 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:01,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:11:01,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:11:01,313 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:11:04,440 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:09,441 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:14,442 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:15,083 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:11:16,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:11:16,313 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:11:16,340 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:11:16,773 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:11:16,774 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:11:16,775 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:11:16,775 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:11:17,694 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:11:19,507 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:19,695 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:11:24,509 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:29,509 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:31,315 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:11:31,316 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:11:31,316 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:11:35,441 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:40,210 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:11:40,211 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:11:40,211 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:11:40,212 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:11:40,704 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:11:40,933 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:41,705 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:11:45,085 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:11:46,086 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:46,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:11:46,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:11:46,314 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:11:51,409 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:56,410 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:01,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:12:01,313 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:12:01,354 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:12:02,400 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:04,914 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:12:04,915 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:12:04,915 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:12:04,916 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:12:05,715 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:12:07,654 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:07,716 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:12:12,655 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:15,086 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:12:16,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:12:16,313 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:12:16,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:12:18,435 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:23,436 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:28,437 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:31,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:12:31,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:12:31,314 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:12:31,374 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:12:31,375 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:12:31,376 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:12:31,376 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:12:31,726 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:12:33,729 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:12:34,257 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:39,258 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:44,259 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:45,087 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:12:46,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:12:46,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:12:46,314 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:12:49,392 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:54,393 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:55,097 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:12:55,099 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:12:55,099 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:12:55,099 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:12:55,738 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:12:58,739 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:12:59,832 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:01,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:13:01,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:13:01,314 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:13:05,357 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:10,358 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:15,088 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:13:16,089 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:16,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:13:16,315 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:13:16,355 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:13:19,507 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:13:19,508 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:13:19,509 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:13:19,509 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:13:19,748 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:13:21,253 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:22,749 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:13:26,254 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:31,255 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:31,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:13:31,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:13:31,314 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:13:36,457 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:41,458 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:45,089 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:13:45,227 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:13:45,228 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:13:45,229 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:13:45,229 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:13:45,759 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:13:46,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:13:46,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:13:46,315 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:13:47,341 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:48,760 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:13:52,342 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:57,343 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:01,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:14:01,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:14:01,315 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:14:03,334 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:08,334 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:09,458 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:14:09,459 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:14:09,459 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:14:09,459 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:14:09,768 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:14:12,770 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:14:14,192 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:15,090 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:14:16,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:14:16,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:14:16,315 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:14:19,369 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:24,370 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:29,371 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:31,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:14:31,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:14:31,315 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:14:33,527 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:14:33,528 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:14:33,528 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:14:33,528 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:14:33,778 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:14:35,275 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:36,780 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:14:40,276 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:45,091 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:14:46,092 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:46,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:14:46,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:14:46,315 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:14:51,388 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:54,679 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:14:54,680 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:14:54,680 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:14:54,680 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:14:54,786 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:14:56,399 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:56,787 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:15:01,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:15:01,315 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:15:01,316 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:15:01,435 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:06,436 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:11,437 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:15,097 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:15:15,807 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:15:15,808 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:15:15,808 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:15:15,808 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:15:16,503 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:16,504 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:15:16,504 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:15:16,544 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:15:16,795 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:15:18,796 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:15:21,631 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:26,632 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:31,497 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:15:31,498 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:15:31,538 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:15:32,576 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:37,577 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:39,618 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:15:39,619 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:15:39,620 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:15:39,620 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:15:39,803 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:15:42,805 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:15:43,315 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:45,094 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:15:46,497 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:15:46,498 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:15:46,500 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:15:48,638 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:53,639 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:58,640 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:00,473 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:16:00,475 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:16:00,475 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:16:00,477 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:16:00,812 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:16:01,498 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:16:01,498 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:16:01,502 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:16:02,812 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:16:04,621 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:09,622 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:14,622 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:15,095 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:16:16,498 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:16:16,499 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:16:16,539 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:16:20,573 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:22,223 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:16:22,224 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:16:22,224 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:16:22,225 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:16:22,820 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:16:24,821 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:16:25,924 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:30,925 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:31,498 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:16:31,498 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:16:31,501 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:16:36,584 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:41,585 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:45,095 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:16:46,502 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:16:46,503 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:16:46,543 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:16:46,618 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:47,880 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:16:47,881 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:16:47,881 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:16:47,882 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:16:48,830 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:16:50,831 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:16:51,653 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:56,654 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:01,499 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:17:01,499 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:17:01,503 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:17:02,546 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:07,546 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:11,638 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:17:11,640 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:17:11,640 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:17:11,640 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:17:11,839 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:17:13,340 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:14,840 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:17:15,097 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:17:16,499 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:17:16,499 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:17:16,503 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:17:18,544 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:23,545 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:28,546 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:31,500 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:17:31,501 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:17:31,533 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:17:34,533 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:39,259 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:17:39,261 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:17:39,261 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:17:39,262 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:17:39,851 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:17:39,996 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:42,852 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:17:44,997 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:45,098 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:17:46,499 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:17:46,500 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:17:46,503 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:17:50,631 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:55,632 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:00,632 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:01,499 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:18:01,499 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:18:01,540 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:18:06,569 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:06,620 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:18:06,621 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:18:06,621 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:18:06,622 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:18:06,861 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:18:08,862 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:18:12,348 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:15,099 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:18:16,499 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:18:16,500 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:18:16,503 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:18:17,552 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:22,553 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:27,554 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:31,500 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:18:31,500 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:18:31,541 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:18:31,955 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:18:31,956 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:18:31,956 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:18:31,957 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:18:32,719 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:32,872 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:18:34,873 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:18:37,720 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:42,721 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:45,100 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:18:46,499 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:18:46,500 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:18:46,504 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:18:48,589 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:53,590 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:53,851 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:18:53,852 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:18:53,853 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:18:53,853 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:18:53,881 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:18:56,883 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:18:59,583 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:01,500 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:19:01,500 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:19:01,504 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:19:05,552 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:10,553 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:15,101 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:19:16,102 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:16,500 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:19:16,500 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:19:16,541 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:19:18,398 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:19:18,399 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:19:18,400 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:19:18,400 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:19:18,893 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:19:20,894 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:19:21,266 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:26,267 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:31,268 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:31,500 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:19:31,500 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:19:31,504 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:19:36,553 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:41,006 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:19:41,008 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:19:41,008 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:19:41,009 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:19:41,791 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:41,902 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:19:42,902 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:19:45,102 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:19:46,500 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:19:46,501 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:19:46,504 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:19:47,605 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:52,606 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:57,608 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:01,500 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:20:01,501 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:20:01,541 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:20:02,608 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:07,563 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:20:07,564 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:20:07,564 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:20:07,564 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:20:07,911 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:20:08,308 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:10,913 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:20:13,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:15,104 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:20:16,501 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:20:16,502 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:20:16,505 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:20:18,536 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:23,537 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:28,538 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:31,501 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:20:31,501 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:20:31,536 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:20:31,759 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:20:31,760 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:20:31,760 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:20:31,761 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:20:31,920 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:20:34,479 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:34,921 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:20:39,480 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:44,481 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:45,105 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:20:46,501 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:20:46,502 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:20:46,505 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:20:49,567 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:54,560 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:20:54,561 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:20:54,561 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:20:54,562 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:20:54,929 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:20:55,295 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:56,929 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:21:00,296 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:01,501 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:21:01,502 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:21:01,505 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:21:05,621 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:10,622 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:15,106 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:21:16,107 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:16,501 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:21:16,502 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:21:16,542 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:21:17,495 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:21:17,497 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:21:17,497 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:21:17,497 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:21:17,938 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:21:20,939 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:21:21,217 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:26,218 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:31,219 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:31,501 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:21:31,502 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:21:31,506 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:21:36,551 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:41,552 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:45,107 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:21:45,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:21:45,159 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:21:45,160 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:21:45,160 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:21:45,950 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:21:46,501 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:21:46,502 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:21:46,506 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:21:46,950 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:21:47,536 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:52,537 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:57,538 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:01,502 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:22:01,502 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:22:01,542 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:22:02,546 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:07,423 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:22:07,425 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:22:07,425 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:22:07,426 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:22:07,958 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:22:08,147 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:08,959 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:22:13,148 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:15,109 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:22:16,502 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:22:16,503 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:22:16,506 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:22:18,578 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:23,579 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:28,580 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:31,337 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:22:31,338 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:22:31,338 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:22:31,339 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:22:31,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:22:31,514 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:22:31,517 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:22:31,968 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:22:32,969 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:22:34,066 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:39,068 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:44,069 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:45,109 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:22:46,512 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:22:46,512 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:22:46,516 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:22:49,640 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:54,640 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:56,698 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:22:56,699 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:22:56,700 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:22:56,700 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:22:56,978 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:22:58,979 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:23:00,440 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:01,512 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:23:01,512 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:23:01,516 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:23:05,620 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:10,620 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:15,111 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:23:16,112 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:16,512 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:23:16,513 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:23:16,553 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:23:21,600 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:22,901 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:23:22,902 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:23:22,902 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:23:22,903 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:23:22,989 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:23:24,989 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:23:26,647 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:31,512 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:23:31,513 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:23:31,516 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:23:32,639 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:37,640 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:42,641 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:45,112 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:23:46,512 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:23:46,513 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:23:46,553 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:23:46,704 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:23:46,705 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:23:46,705 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:23:46,706 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:23:47,000 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:23:48,434 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:49,000 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:23:53,435 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:58,436 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:01,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:24:01,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:24:01,513 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:24:03,586 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:08,587 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:09,545 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:24:09,546 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:24:09,547 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:24:09,547 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:24:10,010 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:24:13,011 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:24:14,397 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:15,113 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:24:16,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:24:16,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:24:16,514 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:24:19,545 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:24,546 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:29,546 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:31,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:24:31,514 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:24:31,551 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:24:34,465 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:24:34,466 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:24:34,466 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:24:34,467 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:24:35,021 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:24:35,195 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:37,022 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:24:40,196 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:45,114 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:24:46,115 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:46,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:24:46,514 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:24:46,518 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:24:51,649 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:56,650 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:01,392 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:25:01,393 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:25:01,393 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:25:01,394 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:25:01,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:25:01,514 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:25:01,516 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:25:02,032 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:25:02,134 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:03,032 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:25:07,136 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:12,137 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:15,116 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:25:16,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:25:16,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:25:16,514 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:25:17,560 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:22,561 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:26,744 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:25:26,745 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:25:26,746 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:25:26,746 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:25:27,043 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:25:28,462 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:29,044 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:25:31,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:25:31,514 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:25:31,517 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:25:33,635 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:38,636 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:43,637 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:45,117 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:25:46,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:25:46,514 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:25:46,554 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:25:47,103 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:25:47,105 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:25:47,105 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:25:47,106 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:25:48,052 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:25:48,865 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:49,053 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:25:53,866 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:58,867 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:01,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:26:01,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:26:01,515 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:26:04,644 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:09,645 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:12,782 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:26:12,783 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:26:12,783 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:26:12,784 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:26:13,062 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:26:15,062 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:26:15,117 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:26:15,119 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:16,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:26:16,515 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:26:16,515 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:26:20,618 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:25,619 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:30,621 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:31,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:26:31,515 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:26:31,549 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:26:34,255 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:26:34,257 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:26:34,257 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:26:34,257 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:26:35,071 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:26:36,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:37,072 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:26:41,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:45,119 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:26:46,119 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:46,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:26:46,515 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:26:46,517 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:26:51,618 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:56,620 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:58,963 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:26:58,965 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:26:58,965 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:26:58,965 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:26:59,081 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:27:01,082 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:27:01,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:27:01,515 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:27:01,517 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:27:01,633 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:06,634 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:11,635 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:15,119 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:27:16,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:27:16,515 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:27:16,556 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:27:17,597 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:22,598 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:26,555 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:27:26,557 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:27:26,557 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:27:26,558 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:27:27,093 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:27:28,303 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:29,094 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:27:31,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:27:31,515 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:27:31,518 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:27:33,618 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:38,619 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:43,620 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:45,121 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:27:46,515 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:27:46,515 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:27:46,556 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:27:49,565 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:51,496 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:27:51,497 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:27:51,498 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:27:51,498 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:27:52,103 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:27:53,104 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:27:55,234 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:00,235 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:01,515 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:28:01,516 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:28:01,519 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:28:05,549 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:10,551 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:15,082 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:28:15,083 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:28:15,084 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:28:15,084 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:28:15,113 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:28:15,122 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:28:15,787 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:16,515 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:28:16,516 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:28:16,517 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:28:17,114 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:28:21,642 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:26,643 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:31,525 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:28:31,526 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:28:31,566 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:28:32,571 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:37,572 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:41,267 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:28:41,269 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:28:41,269 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:28:41,269 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:28:42,124 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:28:42,998 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:43,124 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:28:45,123 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:28:46,520 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:28:46,521 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:28:46,521 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:28:48,566 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:53,567 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:58,568 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:01,520 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:29:01,520 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:29:01,521 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:29:03,589 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:05,159 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:29:05,160 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:29:05,161 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:29:05,161 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:29:06,134 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:29:07,135 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:29:08,989 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:13,991 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:15,124 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:29:16,521 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:29:16,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:29:16,522 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:29:19,600 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:24,601 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:29,602 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:31,520 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:29:31,521 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:29:31,521 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:29:32,263 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:29:32,264 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:29:32,264 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:29:32,265 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:29:33,145 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:29:35,049 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:35,145 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:29:40,050 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:45,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:45,125 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:29:46,520 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:29:46,521 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:29:46,521 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:29:50,564 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:55,565 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:55,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:29:55,762 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:29:55,762 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:29:55,763 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:29:56,154 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:29:57,154 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:30:01,520 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:30:01,521 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:01,521 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:30:01,522 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:30:06,578 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:11,579 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:15,127 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:30:16,521 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:30:16,521 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:30:16,521 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:30:16,640 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:21,641 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:25,139 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:30:25,140 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:30:25,140 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:30:25,141 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:30:25,166 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:30:26,878 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:27,167 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:30:31,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:30:31,523 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:30:31,525 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:30:32,557 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:37,559 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:42,560 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:45,128 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:30:46,521 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:30:46,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:30:46,522 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:30:47,628 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:48,429 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:30:48,430 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:30:48,430 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:30:48,431 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:30:49,176 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:30:51,177 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:30:53,277 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:58,278 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:01,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:31:01,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:31:01,524 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:31:03,571 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:08,572 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:11,181 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:31:11,183 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:31:11,183 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:31:11,183 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:31:11,185 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:31:13,185 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:31:13,902 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:15,129 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:31:16,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:31:16,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:31:16,523 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:31:19,605 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:24,606 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:29,607 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:31,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:31:31,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:31:31,524 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:31:34,407 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:31:34,408 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:31:34,408 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:31:34,409 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:31:35,121 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:35,194 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:31:37,195 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:31:40,122 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:45,123 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:45,130 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:31:46,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:31:46,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:31:46,523 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:31:50,653 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:55,653 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:58,282 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:31:58,284 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:31:58,284 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:31:58,285 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:31:59,203 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:32:01,003 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:01,204 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:32:01,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:32:01,523 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:32:01,564 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:32:06,646 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:11,647 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:15,132 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:32:16,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:32:16,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:32:16,523 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:32:17,614 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:22,615 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:22,957 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:32:22,959 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:32:22,959 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:32:22,960 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:32:23,213 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:32:25,214 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:32:27,687 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:31,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:32:31,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:32:31,523 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:32:33,614 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:38,615 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:43,616 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:45,133 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:32:46,435 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:32:46,437 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:32:46,437 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:32:46,437 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:32:46,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:32:46,523 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:32:46,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:32:47,224 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:32:49,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:49,225 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:32:54,159 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:59,160 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:01,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:33:01,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:33:01,524 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:33:04,594 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:09,595 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:09,888 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:33:09,894 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:33:09,894 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:33:09,894 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:33:10,234 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:33:11,234 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:33:14,744 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:15,134 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:33:16,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:33:16,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:33:16,524 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:33:20,569 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:25,570 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:30,571 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:31,336 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:33:31,338 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:33:31,338 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:33:31,338 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:33:31,797 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:33:31,797 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:33:31,798 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:33:32,242 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:33:33,243 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:33:36,079 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:41,080 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:45,135 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:33:46,136 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:46,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:33:46,761 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:33:46,802 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:33:51,831 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:56,832 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:58,388 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:33:58,389 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:33:58,389 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:33:58,390 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:33:59,253 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:34:01,254 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:34:01,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:34:01,761 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:34:01,802 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:34:02,806 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:07,807 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:12,808 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:15,136 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:34:16,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:34:16,761 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:34:16,802 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:34:17,897 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:22,665 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:34:22,666 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:34:22,666 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:34:22,666 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:34:23,263 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:34:23,439 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:25,264 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:34:28,440 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:31,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:34:31,761 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:34:31,802 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:34:33,824 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:38,825 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:43,827 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:45,137 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:34:46,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:34:46,762 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:34:46,802 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:34:48,192 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:34:48,193 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:34:48,193 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:34:48,194 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:34:48,272 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:34:48,995 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:49,273 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:34:53,996 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:58,997 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:01,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:35:01,762 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:35:01,802 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:35:04,825 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:09,826 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:10,348 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:35:10,349 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:35:10,350 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:35:10,350 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:35:11,283 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:35:11,283 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:35:15,068 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:15,138 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:35:16,762 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:35:16,762 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:35:16,802 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:35:20,874 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:25,875 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:30,876 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:31,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:35:31,762 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:35:31,790 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:35:34,354 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:35:34,355 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:35:34,356 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:35:34,356 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:35:35,294 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:35:35,294 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:35:36,101 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:41,102 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:45,139 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:35:46,140 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:46,762 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:35:46,762 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:35:46,804 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:35:51,842 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:56,843 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:36:01,762 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:36:01,762 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:36:01,803 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:36:02,832 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:36:02,977 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:36:02,978 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:36:02,978 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:36:02,979 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:36:03,305 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:36:05,306 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:36:08,153 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:36:13,154 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:36:15,140 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:36:16,762 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:36:16,762 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:36:16,803 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:36:18,829 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:36:23,830 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:36:25,212 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:36:25,213 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:36:25,213 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:36:25,214 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:36:25,315 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json