| 2025-08-26 16:23:28,730 - __main__ - INFO - 📊 Configuration: | |
| 2025-08-26 16:23:28,731 - __main__ - INFO - Model: VLSP2025-LegalSML/qwen3-4b-legal-pretrain | |
| 2025-08-26 16:23:28,732 - __main__ - INFO - Dataset: thangvip/combined-vietnamese-legal-qa-pretrain-tokenized-8k | |
| 2025-08-26 16:23:28,733 - __main__ - INFO - Training mode: Full parameter training | |
| 2025-08-26 16:23:28,733 - __main__ - INFO - Distributed strategy: DDP (DistributedDataParallel) | |
| 2025-08-26 16:23:28,734 - __main__ - INFO - Accelerator state: Distributed environment: DistributedType.MULTI_GPU Backend: nccl | |
| Num processes: 8 | |
| Process index: 0 | |
| Local process index: 0 | |
| Device: cuda:0 | |
| Mixed precision type: bf16 | |
| 2025-08-26 16:23:28,734 - __main__ - INFO - Number of processes: 8 | |
| 2025-08-26 16:23:28,735 - __main__ - INFO - Device: cuda:0 | |
| 2025-08-26 16:23:28,736 - __main__ - INFO - Mixed precision: bf16 | |
| 2025-08-26 16:23:28,736 - __main__ - INFO - 🚀 DDP Optimizations: | |
| 2025-08-26 16:23:28,737 - __main__ - INFO - DDP bucket size: 25MB | |
| 2025-08-26 16:23:28,737 - __main__ - INFO - DDP broadcast buffers: True | |
| 2025-08-26 16:23:28,737 - __main__ - INFO - DDP find unused parameters: False | |
| 2025-08-26 16:23:28,738 - __main__ - INFO - Strategy: Data Parallelism - each GPU has full model copy | |
| 2025-08-26 16:23:28,744 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-08-26 16:23:29,025 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-08-26 16:23:29,025 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-08-26 16:23:29,061 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-08-26 16:23:29,063 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-08-26 16:23:29,069 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-08-26 16:23:29,069 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-08-26 16:23:29,069 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-08-26 16:23:33,163 - __main__ - INFO - 🔧 Loading model... | |
| 2025-08-26 16:23:33,184 - __main__ - INFO - 🔧 Loading model... | |
| 2025-08-26 16:23:33,194 - __main__ - INFO - 🔧 Loading model... | |
| 2025-08-26 16:23:33,195 - __main__ - INFO - 🔧 Loading model... | |
| 2025-08-26 16:23:33,196 - __main__ - INFO - 🔧 Loading model... | |
| 2025-08-26 16:23:33,237 - __main__ - INFO - 🔧 Loading model... | |
| 2025-08-26 16:23:33,241 - __main__ - INFO - 🔧 Loading model... | |
| 2025-08-26 16:23:33,255 - __main__ - INFO - 🔧 Loading model... | |
| 2025-08-26 16:24:35,063 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-08-26 16:24:38,797 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-08-26 16:24:38,808 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-08-26 16:24:38,814 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-08-26 16:24:38,830 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-08-26 16:24:38,830 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-08-26 16:24:38,875 - __main__ - INFO - 🔥 Full Parameter Training Enabled | |
| 2025-08-26 16:24:38,875 - __main__ - INFO - Total parameters: 4,022,468,096 | |
| 2025-08-26 16:24:38,876 - __main__ - INFO - Trainable parameters: 4,022,468,096 | |
| 2025-08-26 16:24:38,876 - __main__ - INFO - Trainable %: 100.00% | |
| 2025-08-26 16:24:38,882 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-08-26 16:24:39,006 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-08-26 16:28:30,132 - __main__ - INFO - 📊 Configuration: | |
| 2025-08-26 16:28:30,410 - __main__ - INFO - Model: VLSP2025-LegalSML/qwen3-4b-legal-pretrain | |
| 2025-08-26 16:28:30,411 - __main__ - INFO - Dataset: thangvip/combined-vietnamese-legal-qa-pretrain-tokenized-8k | |
| 2025-08-26 16:28:30,412 - __main__ - INFO - Training mode: Full parameter training | |
| 2025-08-26 16:28:30,412 - __main__ - INFO - Distributed strategy: DDP (DistributedDataParallel) | |
| 2025-08-26 16:28:30,413 - __main__ - INFO - Accelerator state: Distributed environment: DistributedType.MULTI_GPU Backend: nccl | |
| Num processes: 8 | |
| Process index: 0 | |
| Local process index: 0 | |
| Device: cuda:0 | |
| Mixed precision type: bf16 | |
| 2025-08-26 16:28:30,413 - __main__ - INFO - Number of processes: 8 | |
| 2025-08-26 16:28:30,414 - __main__ - INFO - Device: cuda:0 | |
| 2025-08-26 16:28:30,415 - __main__ - INFO - Mixed precision: bf16 | |
| 2025-08-26 16:28:30,415 - __main__ - INFO - 🚀 DDP Optimizations: | |
| 2025-08-26 16:28:30,416 - __main__ - INFO - DDP bucket size: 25MB | |
| 2025-08-26 16:28:30,416 - __main__ - INFO - DDP broadcast buffers: True | |
| 2025-08-26 16:28:30,417 - __main__ - INFO - DDP find unused parameters: False | |
| 2025-08-26 16:28:30,417 - __main__ - INFO - Strategy: Data Parallelism - each GPU has full model copy | |
| 2025-08-26 16:28:30,418 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-08-26 16:28:30,422 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-08-26 16:28:30,451 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-08-26 16:28:30,451 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-08-26 16:28:30,456 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-08-26 16:28:30,457 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-08-26 16:28:30,459 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-08-26 16:28:30,459 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-08-26 16:28:34,692 - __main__ - INFO - 🔧 Loading model... | |
| 2025-08-26 16:28:34,693 - __main__ - INFO - 🔧 Loading model... | |
| 2025-08-26 16:28:34,701 - __main__ - INFO - 🔧 Loading model... | |
| 2025-08-26 16:28:34,702 - __main__ - INFO - 🔧 Loading model... | |
| 2025-08-26 16:28:34,709 - __main__ - INFO - 🔧 Loading model... | |
| 2025-08-26 16:28:34,724 - __main__ - INFO - 🔧 Loading model... | |
| 2025-08-26 16:28:34,727 - __main__ - INFO - 🔧 Loading model... | |
| 2025-08-26 16:28:34,736 - __main__ - INFO - 🔧 Loading model... | |
| 2025-08-26 16:29:31,500 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-08-26 16:29:35,334 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-08-26 16:29:35,347 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-08-26 16:29:35,359 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-08-26 16:29:35,378 - __main__ - INFO - 🔥 Full Parameter Training Enabled | |
| 2025-08-26 16:29:35,379 - __main__ - INFO - Total parameters: 4,022,468,096 | |
| 2025-08-26 16:29:35,379 - __main__ - INFO - Trainable parameters: 4,022,468,096 | |
| 2025-08-26 16:29:35,380 - __main__ - INFO - Trainable %: 100.00% | |
| 2025-08-26 16:29:35,381 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-08-26 16:29:35,385 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-08-26 16:29:35,521 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-08-26 16:29:35,572 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-08-26 16:30:06,773 - __main__ - INFO - Dataset size: 60238 training examples | |
| 2025-08-26 16:30:06,781 - __main__ - INFO - Dataset size: 60238 training examples | |
| 2025-08-26 16:30:06,785 - __main__ - INFO - Dataset size: 60238 training examples | |
| 2025-08-26 16:30:06,789 - __main__ - INFO - Dataset size: 60238 training examples | |
| 2025-08-26 16:30:06,808 - __main__ - INFO - Dataset size: 60238 training examples | |
| 2025-08-26 16:30:06,823 - __main__ - INFO - Dataset size: 60238 training examples | |
| 2025-08-26 16:30:06,850 - __main__ - INFO - Dataset size: 60238 training examples | |
| 2025-08-26 16:30:06,897 - __main__ - INFO - Dataset size: 60238 training examples | |
| 2025-08-26 16:30:06,980 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-08-26 16:30:07,015 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-08-26 16:30:07,035 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-08-26 16:30:07,038 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-08-26 16:30:07,048 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-08-26 16:30:07,067 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-08-26 16:30:07,081 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-08-26 16:30:07,101 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-08-26 16:30:11,320 - accelerate.utils.other - WARNING - Detected kernel version 4.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. | |
| 2025-08-26 16:30:12,781 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-08-26 16:30:12,782 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-08-26 16:30:12,825 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-08-26 16:30:12,832 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-08-26 16:30:12,850 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-08-26 16:30:12,907 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-08-26 16:30:12,920 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-08-26 16:30:13,626 - __main__ - INFO - 🚂 Starting TRL training... | |