root@c35059eec1c7:~/alpaca-lora# WORLD_SIZE=4 CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=4 --master_port=3192 finetune.py --base_model '/root/llama-7b-hf/' --data_path './alpaca_data_cleaned.json' --output_dir './lora-alpaca' --batch_size 1024 --micro_batch_size 128 WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** ===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues ================================================================================ ===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues ================================================================================ ===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues ================================================================================ ===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues ================================================================================ /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/tmp/torchelastic__vgbqj6n/none_m7wuat9y/attempt_0/1/error.json')} warn(msg) CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64... CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so CUDA SETUP: Highest compute capability among GPUs detected: 8.0 CUDA SETUP: Detected CUDA version 116 CUDA SETUP: Loading binary /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so... /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/tmp/torchelastic__vgbqj6n/none_m7wuat9y/attempt_0/2/error.json')} warn(msg) CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64... CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so CUDA SETUP: Highest compute capability among GPUs detected: 8.0 CUDA SETUP: Detected CUDA version 116 CUDA SETUP: Loading binary /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so... /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/tmp/torchelastic__vgbqj6n/none_m7wuat9y/attempt_0/0/error.json')} warn(msg) CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64... CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so CUDA SETUP: Highest compute capability among GPUs detected: 8.0 CUDA SETUP: Detected CUDA version 116 CUDA SETUP: Loading binary /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so... /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/tmp/torchelastic__vgbqj6n/none_m7wuat9y/attempt_0/3/error.json')} warn(msg) CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64... CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so CUDA SETUP: Highest compute capability among GPUs detected: 8.0 CUDA SETUP: Detected CUDA version 116 CUDA SETUP: Loading binary /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so... Training Alpaca-LoRA model with params: base_model: /root/llama-7b-hf/ data_path: ./alpaca_data_cleaned.json output_dir: ./lora-alpaca batch_size: 1024 micro_batch_size: 128 num_epochs: 3 learning_rate: 0.0003 cutoff_len: 256 val_set_size: 2000 lora_r: 8 lora_alpha: 16 lora_dropout: 0.05 lora_target_modules: ['q_proj', 'v_proj'] train_on_inputs: True group_by_length: False wandb_project: wandb_run_name: wandb_watch: wandb_log_model: resume_from_checkpoint: None Training Alpaca-LoRA model with params: base_model: /root/llama-7b-hf/ data_path: ./alpaca_data_cleaned.json output_dir: ./lora-alpaca batch_size: 1024 micro_batch_size: 128 num_epochs: 3 learning_rate: 0.0003 cutoff_len: 256 val_set_size: 2000 lora_r: 8 lora_alpha: 16 lora_dropout: 0.05 lora_target_modules: ['q_proj', 'v_proj'] train_on_inputs: True group_by_length: False wandb_project: wandb_run_name: wandb_watch: wandb_log_model: resume_from_checkpoint: None Training Alpaca-LoRA model with params: base_model: /root/llama-7b-hf/ data_path: ./alpaca_data_cleaned.json output_dir: ./lora-alpaca batch_size: 1024 micro_batch_size: 128 num_epochs: 3 learning_rate: 0.0003 cutoff_len: 256 val_set_size: 2000 lora_r: 8 lora_alpha: 16 lora_dropout: 0.05 lora_target_modules: ['q_proj', 'v_proj'] train_on_inputs: True group_by_length: False wandb_project: wandb_run_name: wandb_watch: wandb_log_model: resume_from_checkpoint: None Training Alpaca-LoRA model with params: base_model: /root/llama-7b-hf/ data_path: ./alpaca_data_cleaned.json output_dir: ./lora-alpaca batch_size: 1024 micro_batch_size: 128 num_epochs: 3 learning_rate: 0.0003 cutoff_len: 256 val_set_size: 2000 lora_r: 8 lora_alpha: 16 lora_dropout: 0.05 lora_target_modules: ['q_proj', 'v_proj'] train_on_inputs: True group_by_length: False wandb_project: wandb_run_name: wandb_watch: wandb_log_model: resume_from_checkpoint: None Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:10<00:00, 5.12s/it] Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:10<00:00, 5.28s/it] Found cached dataset json (/root/.cache/huggingface/datasets/json/default-089f33be67988669/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51) 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 536.56it/s] trainable params: 4194304 || all params: 6742609920 || trainable%: 0.06220594176090199 Loading cached split indices for dataset at /root/.cache/huggingface/datasets/json/default-089f33be67988669/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51/cache-014b126db45d32ac.arrow and /root/.cache/huggingface/datasets/json/default-089f33be67988669/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51/cache-24bb91d28b1fd433.arrow Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:10<00:00, 5.35s/it] Map: 1%|█▌ | 389/49759 [00:00<00:38, 1294.21 examples/s]Found cached dataset json (/root/.cache/huggingface/datasets/json/default-089f33be67988669/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51) 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 554.80it/s] trainable params: 4194304 || all params: 6742609920 || trainable%: 0.06220594176090199 Loading cached split indices for dataset at /root/.cache/huggingface/datasets/json/default-089f33be67988669/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51/cache-014b126db45d32ac.arrow and /root/.cache/huggingface/datasets/json/default-089f33be67988669/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51/cache-24bb91d28b1fd433.arrow Map: 1%|██ | 523/49759 [00:00<00:37, 1309.32 examples/s]Found cached dataset json (/root/.cache/huggingface/datasets/json/default-089f33be67988669/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51) 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 374.96it/s] trainable params: 4194304 || all params: 6742609920 || trainable%: 0.06220594176090199 Loading cached split indices for dataset at /root/.cache/huggingface/datasets/json/default-089f33be67988669/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51/cache-014b126db45d32ac.arrow and /root/.cache/huggingface/datasets/json/default-089f33be67988669/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51/cache-24bb91d28b1fd433.arrow Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:12<00:00, 6.13s/it] Map: 5%|█████████▉ | 2525/49759 [00:01<00:36, 1279.49 examples/s]Found cached dataset json (/root/.cache/huggingface/datasets/json/default-089f33be67988669/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51) 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 463.66it/s] trainable params: 4194304 || all params: 6742609920 || trainable%: 0.06220594176090199 Loading cached split indices for dataset at /root/.cache/huggingface/datasets/json/default-089f33be67988669/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51/cache-014b126db45d32ac.arrow and /root/.cache/huggingface/datasets/json/default-089f33be67988669/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51/cache-24bb91d28b1fd433.arrow wandb: Currently logged in as: jeffwan. Use `wandb login --relogin` to force relogin wandb: Tracking run with wandb version 0.14.0 wandb: Run data is saved locally in /root/alpaca-lora/wandb/run-20230328_174559-mzmf7gh8 wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run curious-moon-1 wandb: ⭐️ View project at https://wandb.ai/jeffwan/alpaca wandb: 🚀 View run at https://wandb.ai/jeffwan/alpaca/runs/mzmf7gh8 {'loss': 2.2763, 'learning_rate': 2.9999999999999997e-05, 'epoch': 0.2} {'loss': 2.1957, 'learning_rate': 5.9999999999999995e-05, 'epoch': 0.41} {'loss': 1.9686, 'learning_rate': 8.999999999999999e-05, 'epoch': 0.61} {'loss': 1.5397, 'learning_rate': 0.00011999999999999999, 'epoch': 0.82} {'loss': 1.2312, 'learning_rate': 0.00015, 'epoch': 1.02} {'loss': 1.1251, 'learning_rate': 0.00017999999999999998, 'epoch': 1.22} {'loss': 0.9606, 'learning_rate': 0.00020999999999999998, 'epoch': 1.43} {'loss': 0.8537, 'learning_rate': 0.00023999999999999998, 'epoch': 1.63} {'loss': 0.8412, 'learning_rate': 0.00027, 'epoch': 1.84} {'loss': 0.838, 'learning_rate': 0.0003, 'epoch': 2.04} {'loss': 0.8305, 'learning_rate': 0.00023617021276595742, 'epoch': 2.24} {'loss': 0.8215, 'learning_rate': 0.0001723404255319149, 'epoch': 2.45} {'loss': 0.8147, 'learning_rate': 0.00010851063829787234, 'epoch': 2.65} {'loss': 0.8147, 'learning_rate': 4.468085106382978e-05, 'epoch': 2.86} {'train_runtime': 3187.753, 'train_samples_per_second': 46.828, 'train_steps_per_second': 0.046, 'train_loss': 1.2030427374807344, 'epoch': 3.0} 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 147/147 [53:05<00:00, 21.67s/it] If there's a warning about missing keys above, please disregard :) If there's a warning about missing keys above, please disregard :) If there's a warning about missing keys above, please disregard :) If there's a warning about missing keys above, please disregard :) wandb: Waiting for W&B process to finish... (success). wandb: / 0.015 MB of 0.015 MB uploaded (0.000 MB deduped) wandb: Run history: wandb: train/epoch ▁▂▂▃▃▄▄▅▅▆▆▇▇██ wandb: train/global_step ▁▂▂▃▃▄▄▅▅▆▆▇▇██ wandb: train/learning_rate ▁▂▃▃▄▅▆▆▇█▆▅▃▁ wandb: train/loss ██▇▄▃▂▂▁▁▁▁▁▁▁ wandb: train/total_flos ▁ wandb: train/train_loss ▁ wandb: train/train_runtime ▁ wandb: train/train_samples_per_second ▁ wandb: train/train_steps_per_second ▁ wandb: wandb: Run summary: wandb: train/epoch 3.0 wandb: train/global_step 147 wandb: train/learning_rate 4e-05 wandb: train/loss 0.8147 wandb: train/total_flos 1.5159002374476923e+18 wandb: train/train_loss 1.20304 wandb: train/train_runtime 3187.753 wandb: train/train_samples_per_second 46.828 wandb: train/train_steps_per_second 0.046 wandb: wandb: 🚀 View run curious-moon-1 at: https://wandb.ai/jeffwan/alpaca/runs/mzmf7gh8 wandb: Synced 5 W&B file(s), 0 media file(s), 2 artifact file(s) and 0 other file(s) wandb: Find logs at: ./wandb/run-20230328_174559-mzmf7gh8/logs