| 2023-11-18 13:10:05,570 INFO StreamThr :56156 [internal.py:wandb_internal():86] W&B internal server running at pid: 56156, started at: 2023-11-18 13:10:05.569504 | |
| 2023-11-18 13:10:05,572 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status | |
| 2023-11-18 13:10:05,575 INFO WriterThread:56156 [datastore.py:open_for_write():85] open: /mnt/e/wandb/run-20231118_131005-esux4gxo/run-esux4gxo.wandb | |
| 2023-11-18 13:10:05,577 DEBUG SenderThread:56156 [sender.py:send():380] send: header | |
| 2023-11-18 13:10:05,577 DEBUG SenderThread:56156 [sender.py:send():380] send: run | |
| 2023-11-18 13:10:05,992 INFO SenderThread:56156 [dir_watcher.py:__init__():211] watching files in: /mnt/e/wandb/run-20231118_131005-esux4gxo/files | |
| 2023-11-18 13:10:05,992 INFO SenderThread:56156 [sender.py:_start_run_threads():1124] run started: esux4gxo with start time 1700305805.57009 | |
| 2023-11-18 13:10:06,003 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: check_version | |
| 2023-11-18 13:10:06,004 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: check_version | |
| 2023-11-18 13:10:06,058 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: run_start | |
| 2023-11-18 13:10:06,175 DEBUG HandlerThread:56156 [system_info.py:__init__():32] System info init | |
| 2023-11-18 13:10:06,176 DEBUG HandlerThread:56156 [system_info.py:__init__():47] System info init done | |
| 2023-11-18 13:10:06,176 INFO HandlerThread:56156 [system_monitor.py:start():194] Starting system monitor | |
| 2023-11-18 13:10:06,176 INFO SystemMonitor:56156 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2023-11-18 13:10:06,177 INFO HandlerThread:56156 [system_monitor.py:probe():214] Collecting system info | |
| 2023-11-18 13:10:06,178 INFO SystemMonitor:56156 [interfaces.py:start():190] Started cpu monitoring | |
| 2023-11-18 13:10:06,179 INFO SystemMonitor:56156 [interfaces.py:start():190] Started disk monitoring | |
| 2023-11-18 13:10:06,180 INFO SystemMonitor:56156 [interfaces.py:start():190] Started gpu monitoring | |
| 2023-11-18 13:10:06,181 INFO SystemMonitor:56156 [interfaces.py:start():190] Started memory monitoring | |
| 2023-11-18 13:10:06,183 INFO SystemMonitor:56156 [interfaces.py:start():190] Started network monitoring | |
| 2023-11-18 13:10:06,189 DEBUG HandlerThread:56156 [system_info.py:probe():196] Probing system | |
| 2023-11-18 13:10:06,294 DEBUG HandlerThread:56156 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2023-11-18 13:10:06,295 DEBUG HandlerThread:56156 [system_info.py:probe():244] Probing system done | |
| 2023-11-18 13:10:06,295 DEBUG HandlerThread:56156 [system_monitor.py:probe():223] {'os': 'Linux-5.15.123.1-microsoft-standard-WSL2-x86_64-with-glibc2.35', 'python': '3.10.13', 'heartbeatAt': '2023-11-18T11:10:06.189833', 'startedAt': '2023-11-18T11:10:05.529442', 'docker': None, 'cuda': None, 'args': ('--local_rank=0', '--deepspeed=ds_config.json', '--model_name_or_path=openai/whisper-large-v3', '--train_dataset_names', 'mozilla-foundation/common_voice_11_0', 'mozilla-foundation/common_voice_11_0', 'google/fleurs', 'Finnish-NLP/css10_audio_processed', 'Finnish-NLP/youtube_fi_110k_preprocessed', 'Finnish-NLP/voxpopuli_fi_audio_processed', 'Finnish-NLP/parliament_audio_processed', 'Finnish-NLP/aalto_eduskunta_asr_audio_processed', 'Finnish-NLP/fbc_dialog_audio_processed', 'Finnish-NLP/fbc_monolog_audio_processed', '--train_dataset_config_names', 'fi', 'fi', 'fi_fi', '', '', '', '', '', '', '', '--train_dataset_split_name', 'train', 'validation', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', '--train_dataset_text_column_names', 'sentence', 'sentence', 'raw_transcription', 'sentence', 'texts', 'sentence', 'sentence', 'sentence', 'sentence', 'sentence', '--eval_dataset_name=mozilla-foundation/common_voice_11_0', '--eval_dataset_config_name=fi', '--eval_split_name=test', '--language=finnish', '--model_index_name=WhisperLargeV3Finnish', '--max_steps=60000', '--output_dir=./', '--per_device_train_batch_size=8', '--per_device_eval_batch_size=4', '--gradient_accumulation_steps=2', '--logging_steps=20', '--learning_rate=3e-6', '--warmup_steps=1000', '--evaluation_strategy=steps', '--eval_steps=1000', '--save_strategy=steps', '--save_steps=1000', '--generation_max_length=225', '--length_column_name=input_length', '--max_duration_in_seconds=30', '--text_column_name=sentence', '--freeze_feature_encoder=False', '--report_to=wandb', '--metric_for_best_model=wer', '--greater_is_better=False', '--load_best_model_at_end', '--gradient_checkpointing', '--bf16', '--torch_compile', '--overwrite_output_dir', '--do_train', '--do_eval', '--predict_with_generate', '--do_normalize_eval', '--use_auth_token', '--push_to_hub', '--push_to_hub_model_id=WhisperLargeFinnishV3'), 'state': 'running', 'program': '/mnt/e/run_speech_recognition_seq2seq_streaming.py', 'codePathLocal': 'run_speech_recognition_seq2seq_streaming.py', 'codePath': 'run_speech_recognition_seq2seq_streaming.py', 'host': 'DESKTOP-59O9VN1', 'username': 'rasmus', 'executable': '/home/rasmus/miniconda3/envs/WhisperFinetuneEnv/bin/python', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3911.9949999999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 1006.853931427002, 'used': 37.435340881347656}}, 'gpu': 'NVIDIA GeForce RTX 4080', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4080', 'memory_total': 17171480576}], 'memory': {'total': 54.92573928833008}} | |
| 2023-11-18 13:10:06,295 INFO HandlerThread:56156 [system_monitor.py:probe():224] Finished collecting system info | |
| 2023-11-18 13:10:06,295 INFO HandlerThread:56156 [system_monitor.py:probe():227] Publishing system info | |
| 2023-11-18 13:10:06,296 DEBUG HandlerThread:56156 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment | |
| 2023-11-18 13:10:06,297 DEBUG HandlerThread:56156 [system_info.py:_save_pip():68] Saving pip packages done | |
| 2023-11-18 13:10:06,298 DEBUG HandlerThread:56156 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment | |
| 2023-11-18 13:10:06,996 INFO Thread-12 :56156 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/conda-environment.yaml | |
| 2023-11-18 13:10:06,996 INFO Thread-12 :56156 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/requirements.txt | |
| 2023-11-18 13:10:08,527 DEBUG HandlerThread:56156 [system_info.py:_save_conda():87] Saving conda packages done | |
| 2023-11-18 13:10:08,539 INFO HandlerThread:56156 [system_monitor.py:probe():229] Finished publishing system info | |
| 2023-11-18 13:10:08,542 DEBUG SenderThread:56156 [sender.py:send():380] send: files | |
| 2023-11-18 13:10:08,543 INFO SenderThread:56156 [sender.py:_save_file():1380] saving file wandb-metadata.json with policy now | |
| 2023-11-18 13:10:08,550 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 13:10:08,551 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 13:10:08,554 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 13:10:08,730 DEBUG SenderThread:56156 [sender.py:send():380] send: telemetry | |
| 2023-11-18 13:10:08,731 DEBUG SenderThread:56156 [sender.py:send():380] send: config | |
| 2023-11-18 13:10:08,732 DEBUG SenderThread:56156 [sender.py:send():380] send: metric | |
| 2023-11-18 13:10:08,732 DEBUG SenderThread:56156 [sender.py:send():380] send: telemetry | |
| 2023-11-18 13:10:08,732 DEBUG SenderThread:56156 [sender.py:send():380] send: metric | |
| 2023-11-18 13:10:08,733 WARNING SenderThread:56156 [sender.py:send_metric():1331] Seen metric with glob (shouldn't happen) | |
| 2023-11-18 13:10:08,733 DEBUG SenderThread:56156 [sender.py:send():380] send: telemetry | |
| 2023-11-18 13:10:09,003 INFO Thread-12 :56156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/conda-environment.yaml | |
| 2023-11-18 13:10:09,005 INFO Thread-12 :56156 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/output.log | |
| 2023-11-18 13:10:09,005 INFO Thread-12 :56156 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/wandb-metadata.json | |
| 2023-11-18 13:10:10,150 INFO wandb-upload_0:56156 [upload_job.py:push():131] Uploaded file /tmp/tmpum_ty6bjwandb/lr973oc6-wandb-metadata.json | |
| 2023-11-18 13:10:11,011 INFO Thread-12 :56156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/output.log | |
| 2023-11-18 13:10:11,303 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:10:13,019 INFO Thread-12 :56156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/output.log | |
| 2023-11-18 13:10:16,303 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:10:21,304 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:10:23,550 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 13:10:23,551 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 13:10:23,551 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 13:10:26,717 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:10:31,718 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:10:36,721 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:10:37,123 INFO Thread-12 :56156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/config.yaml | |
| 2023-11-18 13:10:38,551 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 13:10:38,551 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 13:10:38,605 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 13:10:42,797 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:10:47,798 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:10:52,799 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:10:53,551 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 13:10:53,552 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 13:10:53,595 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 13:10:57,815 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:11:02,816 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:11:06,183 DEBUG SystemMonitor:56156 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2023-11-18 13:11:06,185 DEBUG SenderThread:56156 [sender.py:send():380] send: stats | |
| 2023-11-18 13:11:08,186 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:11:08,551 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 13:11:08,552 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 13:11:08,595 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 13:11:13,806 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:11:18,808 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:11:23,551 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 13:11:23,552 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 13:11:23,595 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 13:11:23,815 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:11:28,816 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:11:33,817 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:11:36,186 DEBUG SenderThread:56156 [sender.py:send():380] send: stats | |
| 2023-11-18 13:11:38,551 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 13:11:38,552 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 13:11:38,605 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 13:11:39,781 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:11:44,782 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:11:49,783 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:11:53,552 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 13:11:53,552 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 13:11:53,605 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 13:11:54,799 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:11:59,801 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:12:04,801 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:12:06,186 DEBUG SenderThread:56156 [sender.py:send():380] send: stats | |
| 2023-11-18 13:12:06,293 DEBUG SenderThread:56156 [sender.py:send():380] send: exit | |
| 2023-11-18 13:12:06,293 INFO SenderThread:56156 [sender.py:send_exit():585] handling exit code: 1 | |
| 2023-11-18 13:12:06,294 INFO SenderThread:56156 [sender.py:send_exit():587] handling runtime: 120 | |
| 2023-11-18 13:12:06,306 INFO SenderThread:56156 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end | |
| 2023-11-18 13:12:06,308 INFO SenderThread:56156 [sender.py:send_exit():593] send defer | |
| 2023-11-18 13:12:06,309 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 13:12:06,309 INFO HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 0 | |
| 2023-11-18 13:12:06,310 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 13:12:06,310 INFO SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 0 | |
| 2023-11-18 13:12:06,310 INFO SenderThread:56156 [sender.py:transition_state():613] send defer: 1 | |
| 2023-11-18 13:12:06,310 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 13:12:06,311 INFO HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 1 | |
| 2023-11-18 13:12:06,311 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 13:12:06,311 INFO SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 1 | |
| 2023-11-18 13:12:06,312 INFO SenderThread:56156 [sender.py:transition_state():613] send defer: 2 | |
| 2023-11-18 13:12:06,312 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 13:12:06,312 INFO HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 2 | |
| 2023-11-18 13:12:06,312 INFO HandlerThread:56156 [system_monitor.py:finish():203] Stopping system monitor | |
| 2023-11-18 13:12:06,313 INFO HandlerThread:56156 [interfaces.py:finish():202] Joined cpu monitor | |
| 2023-11-18 13:12:06,313 DEBUG SystemMonitor:56156 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2023-11-18 13:12:06,313 INFO HandlerThread:56156 [interfaces.py:finish():202] Joined disk monitor | |
| 2023-11-18 13:12:06,314 DEBUG SystemMonitor:56156 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2023-11-18 13:12:06,325 INFO HandlerThread:56156 [interfaces.py:finish():202] Joined gpu monitor | |
| 2023-11-18 13:12:06,325 INFO HandlerThread:56156 [interfaces.py:finish():202] Joined memory monitor | |
| 2023-11-18 13:12:06,325 INFO HandlerThread:56156 [interfaces.py:finish():202] Joined network monitor | |
| 2023-11-18 13:12:06,326 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 13:12:06,326 INFO SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 2 | |
| 2023-11-18 13:12:06,326 INFO SenderThread:56156 [sender.py:transition_state():613] send defer: 3 | |
| 2023-11-18 13:12:06,327 DEBUG SenderThread:56156 [sender.py:send():380] send: stats | |
| 2023-11-18 13:12:06,327 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 13:12:06,327 INFO HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 3 | |
| 2023-11-18 13:12:06,328 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 13:12:06,328 INFO SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 3 | |
| 2023-11-18 13:12:06,328 INFO SenderThread:56156 [sender.py:transition_state():613] send defer: 4 | |
| 2023-11-18 13:12:06,329 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 13:12:06,329 INFO HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 4 | |
| 2023-11-18 13:12:06,330 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 13:12:06,330 INFO SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 4 | |
| 2023-11-18 13:12:06,330 INFO SenderThread:56156 [sender.py:transition_state():613] send defer: 5 | |
| 2023-11-18 13:12:06,331 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 13:12:06,331 INFO HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 5 | |
| 2023-11-18 13:12:06,331 DEBUG SenderThread:56156 [sender.py:send():380] send: summary | |
| 2023-11-18 13:12:06,333 INFO SenderThread:56156 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end | |
| 2023-11-18 13:12:06,335 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 13:12:06,335 INFO SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 5 | |
| 2023-11-18 13:12:06,335 INFO SenderThread:56156 [sender.py:transition_state():613] send defer: 6 | |
| 2023-11-18 13:12:06,336 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 13:12:06,336 INFO HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 6 | |
| 2023-11-18 13:12:06,337 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 13:12:06,337 INFO SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 6 | |
| 2023-11-18 13:12:06,337 INFO SenderThread:56156 [sender.py:transition_state():613] send defer: 7 | |
| 2023-11-18 13:12:06,337 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 13:12:06,338 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 13:12:06,338 INFO HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 7 | |
| 2023-11-18 13:12:06,338 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 13:12:06,339 INFO SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 7 | |
| 2023-11-18 13:12:06,523 INFO Thread-12 :56156 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/wandb-summary.json | |
| 2023-11-18 13:12:06,872 INFO SenderThread:56156 [sender.py:transition_state():613] send defer: 8 | |
| 2023-11-18 13:12:06,872 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 13:12:06,872 INFO HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 8 | |
| 2023-11-18 13:12:06,873 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 13:12:06,873 INFO SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 8 | |
| 2023-11-18 13:12:06,874 INFO SenderThread:56156 [job_builder.py:build():298] Attempting to build job artifact | |
| 2023-11-18 13:12:06,880 INFO SenderThread:56156 [job_builder.py:_get_source_type():439] no source found | |
| 2023-11-18 13:12:06,880 INFO SenderThread:56156 [sender.py:transition_state():613] send defer: 9 | |
| 2023-11-18 13:12:06,881 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 13:12:06,881 INFO HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 9 | |
| 2023-11-18 13:12:06,882 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 13:12:06,882 INFO SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 9 | |
| 2023-11-18 13:12:06,882 INFO SenderThread:56156 [dir_watcher.py:finish():358] shutting down directory watcher | |
| 2023-11-18 13:12:07,292 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2023-11-18 13:12:07,540 INFO SenderThread:56156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/output.log | |
| 2023-11-18 13:12:07,541 INFO SenderThread:56156 [dir_watcher.py:finish():388] scan: /mnt/e/wandb/run-20231118_131005-esux4gxo/files | |
| 2023-11-18 13:12:07,543 INFO SenderThread:56156 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/conda-environment.yaml conda-environment.yaml | |
| 2023-11-18 13:12:07,548 INFO SenderThread:56156 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/config.yaml config.yaml | |
| 2023-11-18 13:12:07,553 INFO SenderThread:56156 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/output.log output.log | |
| 2023-11-18 13:12:07,558 INFO SenderThread:56156 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/requirements.txt requirements.txt | |
| 2023-11-18 13:12:07,563 INFO SenderThread:56156 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/wandb-metadata.json wandb-metadata.json | |
| 2023-11-18 13:12:07,564 INFO SenderThread:56156 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/wandb-summary.json wandb-summary.json | |
| 2023-11-18 13:12:07,570 INFO SenderThread:56156 [sender.py:transition_state():613] send defer: 10 | |
| 2023-11-18 13:12:07,570 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: poll_exit | |
| 2023-11-18 13:12:07,571 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 13:12:07,572 INFO HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 10 | |
| 2023-11-18 13:12:07,574 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 13:12:07,574 INFO SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 10 | |
| 2023-11-18 13:12:07,575 INFO SenderThread:56156 [file_pusher.py:finish():175] shutting down file pusher | |
| 2023-11-18 13:12:07,936 INFO wandb-upload_1:56156 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_131005-esux4gxo/files/config.yaml | |
| 2023-11-18 13:12:07,982 INFO wandb-upload_0:56156 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_131005-esux4gxo/files/conda-environment.yaml | |
| 2023-11-18 13:12:08,041 INFO wandb-upload_2:56156 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_131005-esux4gxo/files/output.log | |
| 2023-11-18 13:12:08,059 INFO wandb-upload_3:56156 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_131005-esux4gxo/files/requirements.txt | |
| 2023-11-18 13:12:08,059 INFO wandb-upload_4:56156 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_131005-esux4gxo/files/wandb-summary.json | |
| 2023-11-18 13:12:08,260 INFO Thread-11 (_thread_body):56156 [sender.py:transition_state():613] send defer: 11 | |
| 2023-11-18 13:12:08,260 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 13:12:08,260 INFO HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 11 | |
| 2023-11-18 13:12:08,261 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 13:12:08,261 INFO SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 11 | |
| 2023-11-18 13:12:08,261 INFO SenderThread:56156 [file_pusher.py:join():181] waiting for file pusher | |
| 2023-11-18 13:12:08,261 INFO SenderThread:56156 [sender.py:transition_state():613] send defer: 12 | |
| 2023-11-18 13:12:08,262 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 13:12:08,262 INFO HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 12 | |
| 2023-11-18 13:12:08,262 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 13:12:08,262 INFO SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 12 | |
| 2023-11-18 13:12:08,263 INFO SenderThread:56156 [file_stream.py:finish():595] file stream finish called | |
| 2023-11-18 13:12:08,292 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2023-11-18 13:12:08,775 INFO SenderThread:56156 [file_stream.py:finish():599] file stream finish is done | |
| 2023-11-18 13:12:08,776 INFO SenderThread:56156 [sender.py:transition_state():613] send defer: 13 | |
| 2023-11-18 13:12:08,776 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: poll_exit | |
| 2023-11-18 13:12:08,776 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 13:12:08,777 INFO HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 13 | |
| 2023-11-18 13:12:08,777 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 13:12:08,777 INFO SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 13 | |
| 2023-11-18 13:12:08,778 INFO SenderThread:56156 [sender.py:transition_state():613] send defer: 14 | |
| 2023-11-18 13:12:08,778 DEBUG SenderThread:56156 [sender.py:send():380] send: final | |
| 2023-11-18 13:12:08,778 DEBUG SenderThread:56156 [sender.py:send():380] send: footer | |
| 2023-11-18 13:12:08,778 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 13:12:08,779 INFO HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 14 | |
| 2023-11-18 13:12:08,780 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 13:12:08,780 INFO SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 14 | |
| 2023-11-18 13:12:08,781 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2023-11-18 13:12:08,781 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2023-11-18 13:12:08,781 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: poll_exit | |
| 2023-11-18 13:12:08,781 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: server_info | |
| 2023-11-18 13:12:08,782 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: poll_exit | |
| 2023-11-18 13:12:08,782 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: get_summary | |
| 2023-11-18 13:12:08,782 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: server_info | |
| 2023-11-18 13:12:08,783 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: sampled_history | |
| 2023-11-18 13:12:08,784 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 13:12:08,785 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: job_info | |
| 2023-11-18 13:12:08,929 DEBUG SenderThread:56156 [sender.py:send_request():407] send_request: job_info | |
| 2023-11-18 13:12:08,930 INFO MainThread:56156 [wandb_run.py:_footer_history_summary_info():3760] rendering history | |
| 2023-11-18 13:12:08,930 INFO MainThread:56156 [wandb_run.py:_footer_history_summary_info():3792] rendering summary | |
| 2023-11-18 13:12:08,931 INFO MainThread:56156 [wandb_run.py:_footer_sync_info():3719] logging synced files | |
| 2023-11-18 13:12:08,931 DEBUG HandlerThread:56156 [handler.py:handle_request():146] handle_request: shutdown | |
| 2023-11-18 13:12:08,932 INFO HandlerThread:56156 [handler.py:finish():866] shutting down handler | |
| 2023-11-18 13:12:09,785 INFO WriterThread:56156 [datastore.py:close():294] close: /mnt/e/wandb/run-20231118_131005-esux4gxo/run-esux4gxo.wandb | |
| 2023-11-18 13:12:09,930 INFO SenderThread:56156 [sender.py:finish():1536] shutting down sender | |
| 2023-11-18 13:12:09,930 INFO SenderThread:56156 [file_pusher.py:finish():175] shutting down file pusher | |
| 2023-11-18 13:12:09,931 INFO SenderThread:56156 [file_pusher.py:join():181] waiting for file pusher | |