nohup: ignoring input
[2022-12-19 10:45:52,072] [WARNING] [runner.py:179:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2022-12-19 10:45:52,087] [INFO] [runner.py:508:main] cmd = /usr/bin/python3 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMV19 --master_addr=127.0.0.1 --master_port=29500 tune_gpt.py --deepspeed deepspeed.json --limit=10 --local_rank=-1
[2022-12-19 10:45:53,665] [INFO] [launch.py:142:main] WORLD INFO DICT: {'localhost': [0, 1]}
[2022-12-19 10:45:53,665] [INFO] [launch.py:148:main] nnodes=1, num_local_procs=2, node_rank=0
[2022-12-19 10:45:53,665] [INFO] [launch.py:161:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1]})
[2022-12-19 10:45:53,665] [INFO] [launch.py:162:main] dist_world_size=2
[2022-12-19 10:45:53,665] [INFO] [launch.py:164:main] Setting CUDA_VISIBLE_DEVICES=0,1
No config specified, defaulting to: apps/all
Found cached dataset apps (/home/user/.cache/huggingface/datasets/codeparrot___apps/all/0.0.0/04ac807715d07d6e5cc580f59cdc8213cd7dc4529d0bb819cca72c9f8e8c1aa5)
Max length: 2048
PyTorch: setting up devices
[2022-12-19 10:46:03,625] [INFO] [comm.py:654:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
No config specified, defaulting to: apps/all
Found cached dataset apps (/home/user/.cache/huggingface/datasets/codeparrot___apps/all/0.0.0/04ac807715d07d6e5cc580f59cdc8213cd7dc4529d0bb819cca72c9f8e8c1aa5)
Max length: 2048
PyTorch: setting up devices
The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
GPU memory occupied: 3108 MB.
The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
GPU memory occupied: 3108 MB.
Using /home/user/.cache/torch_extensions/py38_cu116 as PyTorch extensions root...
Using /home/user/.cache/torch_extensions/py38_cu116 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/user/.cache/torch_extensions/py38_cu116/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.6374411582946777 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.604843854904175 seconds
Using /home/user/.cache/torch_extensions/py38_cu116 as PyTorch extensions root...
Using /home/user/.cache/torch_extensions/py38_cu116 as PyTorch extensions root...
Emitting ninja build file /home/user/.cache/torch_extensions/py38_cu116/utils/build.ninja...
Building extension module utils...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module utils...
Time to load utils op: 0.2902805805206299 seconds
Loading extension module utils...
Time to load utils op: 0.2025001049041748 seconds
Rank: 0 partition count [2] and sizes[(62599296, False)] 
Rank: 1 partition count [2] and sizes[(62599296, False)] 
Using /home/user/.cache/torch_extensions/py38_cu116 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.0009868144989013672 seconds
Using /home/user/.cache/torch_extensions/py38_cu116 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.0003800392150878906 seconds

  0%|          | 0/55 [00:00<?, ?it/s]
  2%|▏         | 1/55 [00:01<01:13,  1.37s/it]
                                              
{'loss': 4.65, 'learning_rate': 0.0, 'epoch': 0.09}

  2%|▏         | 1/55 [00:01<01:13,  1.37s/it]
  4%|▎         | 2/55 [00:02<01:13,  1.39s/it]
  5%|▌         | 3/55 [00:04<01:13,  1.42s/it]
  7%|▋         | 4/55 [00:05<01:12,  1.42s/it]
  9%|▉         | 5/55 [00:07<01:10,  1.41s/it]
                                              
{'loss': 3.3673, 'learning_rate': 1.294882868674145e-05, 'epoch': 0.45}

  9%|▉         | 5/55 [00:07<01:10,  1.41s/it]
 11%|█         | 6/55 [00:08<01:09,  1.41s/it]
 13%|█▎        | 7/55 [00:09<01:07,  1.41s/it]
 15%|█▍        | 8/55 [00:11<01:06,  1.41s/it]
 16%|█▋        | 9/55 [00:12<01:04,  1.41s/it]
 18%|█▊        | 10/55 [00:14<01:03,  1.41s/it]
                                               
{'loss': 1.5333, 'learning_rate': 1.852558565662928e-05, 'epoch': 0.91}

 18%|█▊        | 10/55 [00:14<01:03,  1.41s/it]
 20%|██        | 11/55 [00:15<01:02,  1.41s/it]
 22%|██▏       | 12/55 [00:16<01:00,  1.41s/it]
 24%|██▎       | 13/55 [00:18<00:59,  1.41s/it]
 25%|██▌       | 14/55 [00:19<00:57,  1.41s/it]
 27%|██▋       | 15/55 [00:21<00:56,  1.41s/it]
                                               
{'loss': 0.7749, 'learning_rate': 2.1787779359648994e-05, 'epoch': 1.36}

 27%|██▋       | 15/55 [00:21<00:56,  1.41s/it]
 29%|██▉       | 16/55 [00:22<00:55,  1.41s/it]
 31%|███       | 17/55 [00:23<00:53,  1.41s/it]
 33%|███▎      | 18/55 [00:25<00:52,  1.41s/it]
 35%|███▍      | 19/55 [00:26<00:50,  1.41s/it]
 36%|███▋      | 20/55 [00:28<00:49,  1.41s/it]
                                               
{'loss': 0.7098, 'learning_rate': 2.41023426265171e-05, 'epoch': 1.82}

 36%|███▋      | 20/55 [00:28<00:49,  1.41s/it]
 38%|███▊      | 21/55 [00:29<00:48,  1.41s/it]
 40%|████      | 22/55 [00:31<00:46,  1.41s/it]
 42%|████▏     | 23/55 [00:32<00:45,  1.41s/it]
 44%|████▎     | 24/55 [00:33<00:43,  1.41s/it]
 45%|████▌     | 25/55 [00:35<00:42,  1.41s/it]
                                               
{'loss': 0.6322, 'learning_rate': 2.58976573734829e-05, 'epoch': 2.27}

 45%|████▌     | 25/55 [00:35<00:42,  1.41s/it]
 47%|████▋     | 26/55 [00:36<00:40,  1.40s/it]
 49%|████▉     | 27/55 [00:38<00:39,  1.40s/it]
 51%|█████     | 28/55 [00:39<00:37,  1.40s/it]
 53%|█████▎    | 29/55 [00:40<00:36,  1.40s/it]
 55%|█████▍    | 30/55 [00:42<00:35,  1.41s/it]
                                               
{'loss': 0.591, 'learning_rate': 2.7364536329536817e-05, 'epoch': 2.73}

 55%|█████▍    | 30/55 [00:42<00:35,  1.41s/it]
 56%|█████▋    | 31/55 [00:43<00:33,  1.41s/it]
 58%|█████▊    | 32/55 [00:45<00:32,  1.41s/it]
 60%|██████    | 33/55 [00:46<00:30,  1.41s/it]
 62%|██████▏   | 34/55 [00:47<00:29,  1.41s/it]
 64%|██████▎   | 35/55 [00:49<00:28,  1.41s/it]
                                               
{'loss': 0.5036, 'learning_rate': 2.8604764815275082e-05, 'epoch': 3.18}

 64%|██████▎   | 35/55 [00:49<00:28,  1.41s/it]
 65%|██████▌   | 36/55 [00:50<00:26,  1.41s/it]
 67%|██████▋   | 37/55 [00:52<00:25,  1.42s/it]
 69%|██████▉   | 38/55 [00:53<00:24,  1.42s/it]
 71%|███████   | 39/55 [00:55<00:22,  1.42s/it]
 73%|███████▎  | 40/55 [00:56<00:21,  1.42s/it]
                                               
{'loss': 0.5058, 'learning_rate': 2.9679099596404923e-05, 'epoch': 3.64}

 73%|███████▎  | 40/55 [00:56<00:21,  1.42s/it]
 75%|███████▍  | 41/55 [00:57<00:19,  1.42s/it]
 76%|███████▋  | 42/55 [00:59<00:18,  1.42s/it]
 78%|███████▊  | 43/55 [01:00<00:17,  1.42s/it]
 80%|████████  | 44/55 [01:02<00:15,  1.42s/it]
 82%|████████▏ | 45/55 [01:03<00:14,  1.42s/it]
                                               
{'loss': 0.404, 'learning_rate': 3.0626730032556536e-05, 'epoch': 4.09}

 82%|████████▏ | 45/55 [01:03<00:14,  1.42s/it]
 84%|████████▎ | 46/55 [01:04<00:12,  1.41s/it]
 85%|████████▌ | 47/55 [01:06<00:11,  1.41s/it]
 87%|████████▋ | 48/55 [01:07<00:09,  1.41s/it]
 89%|████████▉ | 49/55 [01:09<00:08,  1.41s/it]
 91%|█████████ | 50/55 [01:10<00:07,  1.41s/it]
                                               
{'loss': 0.3794, 'learning_rate': 3.147441434337073e-05, 'epoch': 4.55}

 91%|█████████ | 50/55 [01:10<00:07,  1.41s/it]
 93%|█████████▎| 51/55 [01:11<00:05,  1.42s/it]
 95%|█████████▍| 52/55 [01:13<00:04,  1.42s/it]
 96%|█████████▋| 53/55 [01:14<00:02,  1.42s/it]
 98%|█████████▊| 54/55 [01:16<00:01,  1.42s/it]
100%|██████████| 55/55 [01:17<00:00,  1.42s/it]
                                               
{'loss': 0.3672, 'learning_rate': 3.224123807782732e-05, 'epoch': 5.0}

100%|██████████| 55/55 [01:17<00:00,  1.42s/it]Time: 77.86
Samples/second: 8.86
                                               

{'train_runtime': 77.6739, 'train_samples_per_second': 8.883, 'train_steps_per_second': 0.708, 'train_loss': 0.9113625873218884, 'epoch': 5.0}

100%|██████████| 55/55 [01:17<00:00,  1.42s/it]GPU memory occupied: 44704 MB.

100%|██████████| 55/55 [01:17<00:00,  1.41s/it]
Time: 77.67
Samples/second: 8.88
GPU memory occupied: 44704 MB.
Traceback (most recent call last):
  File "tune_gpt.py", line 223, in <module>
    shutil.move(os.path.join(pwd_path, "output.log"), os.path.join(final_save_dir))
  File "/usr/lib/python3.8/shutil.py", line 789, in move
    raise Error("Destination path '%s' already exists" % real_dst)
shutil.Error: Destination path 'experiments/2022-12-19-ab8f3a39c84fea7f66bf71860384bbce5df5fb3523e7dabd22b35c3ecfefb154/output.log' already exists
[2022-12-19 10:47:33,785] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 144520
[2022-12-19 10:47:33,786] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 144521
[2022-12-19 10:47:33,793] [ERROR] [launch.py:324:sigkill_handler] ['/usr/bin/python3', '-u', 'tune_gpt.py', '--local_rank=1', '--deepspeed', 'deepspeed.json', '--limit=10', '--local_rank=-1'] exits with return code = 1