Upload experiments/2023-02-21-b0010c97cb1f06debca911602ea05b6ff85a8270fb9487d27b3d52eb4eb29e9e with huggingface_hub

experiments/2023-02-21-b0010c97cb1f06debca911602ea05b6ff85a8270fb9487d27b3d52eb4eb29e9e/output.log

@@ -0,0 +1,543 @@
+nohup: ignoring input
+[2023-02-20 17:05:49,355] [WARNING] [runner.py:186:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
+[2023-02-20 17:05:49,405] [INFO] [runner.py:548:main] cmd = /opt/conda/bin/python3 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMiwgMywgNCwgNSwgNl19 --master_addr=127.0.0.1 --master_port=29500 --enable_each_rank_log=None tune_gpt.py --deepspeed deepspeed.json --upload-experiment
+/opt/conda/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
+  from pandas import MultiIndex, Int64Index
+[2023-02-20 17:05:51,897] [INFO] [launch.py:135:main] 0 NCCL_VERSION=2.11.4
+[2023-02-20 17:05:51,897] [INFO] [launch.py:142:main] WORLD INFO DICT: {'localhost': [0, 1, 2, 3, 4, 5, 6]}
+[2023-02-20 17:05:51,897] [INFO] [launch.py:148:main] nnodes=1, num_local_procs=7, node_rank=0
+[2023-02-20 17:05:51,897] [INFO] [launch.py:161:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2, 3, 4, 5, 6]})
+[2023-02-20 17:05:51,897] [INFO] [launch.py:162:main] dist_world_size=7
+[2023-02-20 17:05:51,897] [INFO] [launch.py:164:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6
+/opt/conda/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
+  from pandas import MultiIndex, Int64Index
+/opt/conda/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
+  from pandas import MultiIndex, Int64Index
+/opt/conda/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
+  from pandas import MultiIndex, Int64Index
+/opt/conda/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
+  from pandas import MultiIndex, Int64Index
+/opt/conda/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
+  from pandas import MultiIndex, Int64Index
+/opt/conda/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
+  from pandas import MultiIndex, Int64Index
+/opt/conda/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
+  from pandas import MultiIndex, Int64Index
+No config specified, defaulting to: apps/all
+No config specified, defaulting to: apps/all
+Found cached dataset apps (/home/.cache/huggingface/datasets/codeparrot___apps/all/0.0.0/04ac807715d07d6e5cc580f59cdc8213cd7dc4529d0bb819cca72c9f8e8c1aa5)
+Found cached dataset apps (/home/.cache/huggingface/datasets/codeparrot___apps/all/0.0.0/04ac807715d07d6e5cc580f59cdc8213cd7dc4529d0bb819cca72c9f8e8c1aa5)
+No config specified, defaulting to: apps/all
+Found cached dataset apps (/home/.cache/huggingface/datasets/codeparrot___apps/all/0.0.0/04ac807715d07d6e5cc580f59cdc8213cd7dc4529d0bb819cca72c9f8e8c1aa5)
+No config specified, defaulting to: apps/all
+Found cached dataset apps (/home/.cache/huggingface/datasets/codeparrot___apps/all/0.0.0/04ac807715d07d6e5cc580f59cdc8213cd7dc4529d0bb819cca72c9f8e8c1aa5)
+No config specified, defaulting to: apps/all
+Found cached dataset apps (/home/.cache/huggingface/datasets/codeparrot___apps/all/0.0.0/04ac807715d07d6e5cc580f59cdc8213cd7dc4529d0bb819cca72c9f8e8c1aa5)
+No config specified, defaulting to: apps/all
+No config specified, defaulting to: apps/all
+Found cached dataset apps (/home/.cache/huggingface/datasets/codeparrot___apps/all/0.0.0/04ac807715d07d6e5cc580f59cdc8213cd7dc4529d0bb819cca72c9f8e8c1aa5)
+Found cached dataset apps (/home/.cache/huggingface/datasets/codeparrot___apps/all/0.0.0/04ac807715d07d6e5cc580f59cdc8213cd7dc4529d0bb819cca72c9f8e8c1aa5)
+Max length: 2048
+PyTorch: setting up devices
+Max length: 2048
+PyTorch: setting up devices
+Max length: 2048
+PyTorch: setting up devices
+Max length: 2048
+PyTorch: setting up devices
+[2023-02-20 17:06:11,414] [INFO] [comm.py:657:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
+Max length: 2048
+PyTorch: setting up devices
+Max length: 2048
+PyTorch: setting up devices
+Max length: 2048
+PyTorch: setting up devices
+The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
+The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
+The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
+The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
+The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
+The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
+The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
+GPU memory occupied: 6883 MB.
+GPU memory occupied: 6883 MB.
+GPU memory occupied: 6883 MB.
+GPU memory occupied: 6883 MB.
+GPU memory occupied: 6883 MB.
+GPU memory occupied: 6883 MB.
+GPU memory occupied: 6883 MB.
+[2023-02-20 17:06:12,424] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed info: version=0.8.1, git-hash=unknown, git-branch=unknown
+[2023-02-20 17:06:14,006] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
+Installed CUDA version 11.6 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
+Installed CUDA version 11.6 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
+Installed CUDA version 11.6 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
+Installed CUDA version 11.6 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
+Installed CUDA version 11.6 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
+Installed CUDA version 11.6 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
+Installed CUDA version 11.6 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+Detected CUDA files, patching ldflags
+Emitting ninja build file /home/.cache/torch_extensions/py38_cu117/cpu_adam/build.ninja...
+Building extension module cpu_adam...
+Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+ninja: no work to do.
+Loading extension module cpu_adam...
+Time to load cpu_adam op: 2.825831890106201 seconds
+Loading extension module cpu_adam...
+Time to load cpu_adam op: 2.6894984245300293 seconds
+Loading extension module cpu_adam...
+Time to load cpu_adam op: 2.815955877304077 seconds
+Loading extension module cpu_adam...
+Time to load cpu_adam op: 2.816244125366211 seconds
+Loading extension module cpu_adam...
+Time to load cpu_adam op: 2.7123100757598877 seconds
+Loading extension module cpu_adam...
+Time to load cpu_adam op: 2.8215184211730957 seconds
+Loading extension module cpu_adam...
+Time to load cpu_adam op: 2.789081573486328 seconds
+Adam Optimizer #0 is created with AVX2 arithmetic capability.
+Config: alpha=0.000050, betas=(0.900000, 0.999000), weight_decay=0.050000, adam_w=1
+[2023-02-20 17:06:19,789] [INFO] [logging.py:75:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer
+[2023-02-20 17:06:19,794] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
+[2023-02-20 17:06:19,795] [INFO] [utils.py:53:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
+[2023-02-20 17:06:19,795] [INFO] [logging.py:75:log_dist] [Rank 0] Creating torch.float32 ZeRO stage 2 optimizer
+[2023-02-20 17:06:19,795] [INFO] [stage_1_and_2.py:144:__init__] Reduce bucket size 500000000
+[2023-02-20 17:06:19,795] [INFO] [stage_1_and_2.py:145:__init__] Allgather bucket size 500000000
+[2023-02-20 17:06:19,795] [INFO] [stage_1_and_2.py:146:__init__] CPU Offload: True
+[2023-02-20 17:06:19,795] [INFO] [stage_1_and_2.py:147:__init__] Round robin gradient partitioning: False
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+Emitting ninja build file /home/.cache/torch_extensions/py38_cu117/utils/build.ninja...
+Building extension module utils...
+Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
+ninja: no work to do.
+Loading extension module utils...
+Time to load utils op: 0.34289026260375977 seconds
+Loading extension module utils...
+Time to load utils op: 0.2027883529663086 seconds
+Loading extension module utils...
+Loading extension module utils...
+Time to load utils op: 0.2021796703338623 seconds
+Loading extension module utils...
+Time to load utils op: 0.2025763988494873 seconds
+Loading extension module utils...
+Time to load utils op: 0.2033846378326416 seconds
+Time to load utils op: 0.2029557228088379 seconds
+Loading extension module utils...
+Time to load utils op: 0.30292582511901855 seconds
+Rank: 6 partition count [7] and sizes[(17885514, False)] 
+Rank: 5 partition count [7] and sizes[(17885514, False)] 
+Rank: 4 partition count [7] and sizes[(17885514, False)] 
+Rank: 2 partition count [7] and sizes[(17885514, False)] 
+Rank: 3 partition count [7] and sizes[(17885514, False)] 
+Rank: 1 partition count [7] and sizes[(17885514, False)] 
+Rank: 0 partition count [7] and sizes[(17885514, False)] 
+[2023-02-20 17:06:27,470] [INFO] [utils.py:825:see_memory_usage] Before initializing optimizer states
+[2023-02-20 17:06:27,471] [INFO] [utils.py:826:see_memory_usage] MA 0.66 GB         Max_MA 0.66 GB         CA 0.85 GB         Max_CA 1 GB 
+[2023-02-20 17:06:27,471] [INFO] [utils.py:834:see_memory_usage] CPU Virtual Memory:  used = 39.85 GB, percent = 7.9%
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+No modifications detected for re-loaded extension module utils, skipping build step...
+Loading extension module utils...
+Time to load utils op: 0.00165557861328125 seconds
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+No modifications detected for re-loaded extension module utils, skipping build step...
+Loading extension module utils...
+Time to load utils op: 0.008014678955078125 seconds
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+No modifications detected for re-loaded extension module utils, skipping build step...
+Loading extension module utils...
+No modifications detected for re-loaded extension module utils, skipping build step...
+Loading extension module utils...
+Time to load utils op: 0.03653693199157715 seconds
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+No modifications detected for re-loaded extension module utils, skipping build step...
+Loading extension module utils...
+Time to load utils op: 0.008858203887939453 seconds
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+No modifications detected for re-loaded extension module utils, skipping build step...
+Loading extension module utils...
+Time to load utils op: 0.0007452964782714844 seconds
+Time to load utils op: 0.046510934829711914 seconds
+You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
+You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
+You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
+You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
+You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
+You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
+[2023-02-20 17:06:28,120] [INFO] [utils.py:825:see_memory_usage] After initializing optimizer states
+[2023-02-20 17:06:28,121] [INFO] [utils.py:826:see_memory_usage] MA 0.66 GB         Max_MA 0.66 GB         CA 0.85 GB         Max_CA 1 GB 
+[2023-02-20 17:06:28,121] [INFO] [utils.py:834:see_memory_usage] CPU Virtual Memory:  used = 40.4 GB, percent = 8.0%
+[2023-02-20 17:06:28,121] [INFO] [stage_1_and_2.py:527:__init__] optimizer state initialized
+[2023-02-20 17:06:28,222] [INFO] [utils.py:825:see_memory_usage] After initializing ZeRO optimizer
+[2023-02-20 17:06:28,222] [INFO] [utils.py:826:see_memory_usage] MA 0.66 GB         Max_MA 0.66 GB         CA 0.85 GB         Max_CA 1 GB 
+[2023-02-20 17:06:28,223] [INFO] [utils.py:834:see_memory_usage] CPU Virtual Memory:  used = 40.4 GB, percent = 8.0%
+[2023-02-20 17:06:28,223] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw
+[2023-02-20 17:06:28,223] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed using configured LR scheduler = WarmupLR
+[2023-02-20 17:06:28,223] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed LR Scheduler = <deepspeed.runtime.lr_schedules.WarmupLR object at 0x7efce1a33f40>
+[2023-02-20 17:06:28,224] [INFO] [logging.py:75:log_dist] [Rank 0] step=0, skipped=0, lr=[5e-05], mom=[[0.9, 0.999]]
+[2023-02-20 17:06:28,226] [INFO] [config.py:1009:print] DeepSpeedEngine configuration:
+[2023-02-20 17:06:28,226] [INFO] [config.py:1013:print]   activation_checkpointing_config  {
+    "partition_activations": false, 
+    "contiguous_memory_optimization": false, 
+    "cpu_checkpointing": false, 
+    "number_checkpoints": null, 
+    "synchronize_checkpoint_boundary": false, 
+    "profile": false
+}
+[2023-02-20 17:06:28,226] [INFO] [config.py:1013:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}
+[2023-02-20 17:06:28,226] [INFO] [config.py:1013:print]   amp_enabled .................. False
+[2023-02-20 17:06:28,226] [INFO] [config.py:1013:print]   amp_params ................... False
+[2023-02-20 17:06:28,226] [INFO] [config.py:1013:print]   autotuning_config ............ {
+    "enabled": false, 
+    "start_step": null, 
+    "end_step": null, 
+    "metric_path": null, 
+    "arg_mappings": null, 
+    "metric": "throughput", 
+    "model_info": null, 
+    "results_dir": "autotuning_results", 
+    "exps_dir": "autotuning_exps", 
+    "overwrite": true, 
+    "fast": true, 
+    "start_profile_step": 3, 
+    "end_profile_step": 5, 
+    "tuner_type": "gridsearch", 
+    "tuner_early_stopping": 5, 
+    "tuner_num_trials": 50, 
+    "model_info_path": null, 
+    "mp_size": 1, 
+    "max_train_batch_size": null, 
+    "min_train_batch_size": 1, 
+    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
+    "min_train_micro_batch_size_per_gpu": 1, 
+    "num_tuning_micro_batch_sizes": 3
+}
+[2023-02-20 17:06:28,226] [INFO] [config.py:1013:print]   bfloat16_enabled ............. False
+[2023-02-20 17:06:28,226] [INFO] [config.py:1013:print]   checkpoint_parallel_write_pipeline  False
+[2023-02-20 17:06:28,226] [INFO] [config.py:1013:print]   checkpoint_tag_validation_enabled  True
+[2023-02-20 17:06:28,226] [INFO] [config.py:1013:print]   checkpoint_tag_validation_fail  False
+[2023-02-20 17:06:28,226] [INFO] [config.py:1013:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7efd0eb74250>
+[2023-02-20 17:06:28,226] [INFO] [config.py:1013:print]   communication_data_type ...... None
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   curriculum_enabled_legacy .... False
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   curriculum_params_legacy ..... False
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   data_efficiency_enabled ...... False
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   dataloader_drop_last ......... False
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   disable_allgather ............ False
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   dump_state ................... False
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   dynamic_loss_scale_args ...... None
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   eigenvalue_enabled ........... False
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   eigenvalue_gas_boundary_resolution  1
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   eigenvalue_layer_name ........ bert.encoder.layer
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   eigenvalue_layer_num ......... 0
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   eigenvalue_max_iter .......... 100
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   eigenvalue_stability ......... 1e-06
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   eigenvalue_tol ............... 0.01
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   eigenvalue_verbose ........... False
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   elasticity_enabled ........... False
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   flops_profiler_config ........ {
+    "enabled": false, 
+    "profile_step": 1, 
+    "module_depth": -1, 
+    "top_modules": 1, 
+    "detailed": true, 
+    "output_file": null
+}
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   fp16_auto_cast ............... None
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   fp16_enabled ................. False
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   fp16_master_weights_and_gradients  False
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   global_rank .................. 0
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   grad_accum_dtype ............. None
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   gradient_accumulation_steps .. 64
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   gradient_clipping ............ 1.0
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   gradient_predivide_factor .... 1.0
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   initial_dynamic_scale ........ 65536
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   load_universal_checkpoint .... False
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   loss_scale ................... 0
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   memory_breakdown ............. False
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=True, output_path='logs/', job_name='train_neo') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=True
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   nebula_config ................ {
+    "enabled": false, 
+    "persistent_storage_path": null, 
+    "persistent_time_interval": 100, 
+    "num_of_version_in_retention": 2, 
+    "enable_nebula_load": true, 
+    "load_path": null
+}
+[2023-02-20 17:06:28,227] [INFO] [config.py:1013:print]   optimizer_legacy_fusion ...... False
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   optimizer_name ............... adamw
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   optimizer_params ............. {'lr': 5e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.05}
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   pld_enabled .................. False
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   pld_params ................... False
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   prescale_gradients ........... False
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   scheduler_name ............... WarmupLR
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   scheduler_params ............. {'warmup_min_lr': 0, 'warmup_max_lr': 5e-05, 'warmup_num_steps': 500}
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   sparse_attention ............. None
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   sparse_gradients_enabled ..... False
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   steps_per_print .............. 2000
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   train_batch_size ............. 2688
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   train_micro_batch_size_per_gpu  6
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   use_node_local_storage ....... False
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   wall_clock_breakdown ......... False
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   world_size ................... 7
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   zero_allow_untested_optimizer  True
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   zero_config .................. stage=2 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500000000 allgather_partitions=True allgather_bucket_size=500000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='cpu', nvme_path=None, buffer_count=4, pin_memory=True, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False) sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   zero_enabled ................. True
+[2023-02-20 17:06:28,228] [INFO] [config.py:1013:print]   zero_optimization_stage ...... 2
+[2023-02-20 17:06:28,228] [INFO] [config.py:998:print_user_config]   json = {
+    "optimizer": {
+        "type": "AdamW", 
+        "params": {
+            "lr": 5e-05, 
+            "betas": [0.9, 0.999], 
+            "eps": 1e-08, 
+            "weight_decay": 0.05
+        }
+    }, 
+    "scheduler": {
+        "type": "WarmupLR", 
+        "params": {
+            "warmup_min_lr": 0, 
+            "warmup_max_lr": 5e-05, 
+            "warmup_num_steps": 500
+        }
+    }, 
+    "zero_optimization": {
+        "stage": 2, 
+        "offload_optimizer": {
+            "device": "cpu", 
+            "pin_memory": true
+        }, 
+        "allgather_partitions": true, 
+        "allgather_bucket_size": 5.000000e+08, 
+        "overlap_comm": true, 
+        "reduce_scatter": true, 
+        "reduce_bucket_size": 5.000000e+08, 
+        "contiguous_gradients": true
+    }, 
+    "tensorboard": {
+        "enabled": true, 
+        "output_path": "logs/", 
+        "job_name": "train_neo"
+    }, 
+    "zero_allow_untested_optimizer": true, 
+    "gradient_accumulation_steps": 64, 
+    "gradient_clipping": 1.0, 
+    "steps_per_print": 2.000000e+03, 
+    "train_batch_size": 2.688000e+03, 
+    "train_micro_batch_size_per_gpu": 6, 
+    "wall_clock_breakdown": false
+}
+Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
+No modifications detected for re-loaded extension module utils, skipping build step...
+Loading extension module utils...
+Time to load utils op: 0.00042748451232910156 seconds
+***** Running training *****
+  Num examples = 117232
+  Num Epochs = 10
+  Instantaneous batch size per device = 6
+  Total train batch size (w. parallel, distributed & accumulation) = 2688
+  Gradient Accumulation steps = 64
+  Total optimization steps = 430
+  Number of trainable parameters = 125198592
+
  0%|          | 0/430 [00:00<?, ?it/s]You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
+
  0%|          | 1/430 [01:21<9:41:43, 81.36s/it]
                                                 
+
  0%|          | 1/430 [01:21<9:41:43, 81.36s/it]
  0%|          | 2/430 [02:41<9:36:10, 80.77s/it]
  1%|          | 3/430 [04:02<9:33:33, 80.59s/it]
  1%|          | 4/430 [05:22<9:31:45, 80.53s/it]
  1%|          | 5/430 [06:42<9:30:07, 80.49s/it]
                                                 
+
  1%|          | 5/430 [06:42<9:30:07, 80.49s/it]
  1%|▏         | 6/430 [08:03<9:28:34, 80.46s/it]
  2%|▏         | 7/430 [09:24<9:28:07, 80.58s/it]
  2%|▏         | 8/430 [10:44<9:26:23, 80.53s/it]
  2%|▏         | 9/430 [12:05<9:24:58, 80.52s/it]
  2%|▏         | 10/430 [13:25<9:23:23, 80.49s/it]
                                                  
+
  2%|▏         | 10/430 [13:25<9:23:23, 80.49s/it]
  3%|▎         | 11/430 [14:45<9:22:02, 80.48s/it]
  3%|▎         | 12/430 [16:06<9:20:48, 80.50s/it]
  3%|▎         | 13/430 [17:26<9:19:19, 80.48s/it]
  3%|▎         | 14/430 [18:47<9:18:17, 80.52s/it]
  3%|▎         | 15/430 [20:08<9:16:48, 80.50s/it]
                                                  
+
  3%|▎         | 15/430 [20:08<9:16:48, 80.50s/it]
  4%|▎         | 16/430 [21:28<9:15:34, 80.52s/it]
  4%|▍         | 17/430 [22:49<9:14:08, 80.51s/it]
  4%|▍         | 18/430 [24:09<9:12:33, 80.47s/it]
  4%|▍         | 19/430 [25:30<9:11:22, 80.49s/it]
  5%|▍         | 20/430 [26:50<9:09:51, 80.47s/it]
                                                  
+
  5%|▍         | 20/430 [26:50<9:09:51, 80.47s/it]
  5%|▍         | 21/430 [28:10<9:08:25, 80.45s/it]
  5%|▌         | 22/430 [29:31<9:08:02, 80.59s/it]
  5%|▌         | 23/430 [30:52<9:06:16, 80.53s/it]
  6%|▌         | 24/430 [32:12<9:04:45, 80.51s/it]
  6%|▌         | 25/430 [33:32<9:03:06, 80.46s/it]
                                                  
+
  6%|▌         | 25/430 [33:32<9:03:06, 80.46s/it]
  6%|▌         | 26/430 [34:53<9:01:27, 80.42s/it]
  6%|▋         | 27/430 [36:13<8:59:58, 80.39s/it]
  7%|▋         | 28/430 [37:33<8:58:26, 80.36s/it]
  7%|▋         | 29/430 [38:54<8:57:43, 80.46s/it]
  7%|▋         | 30/430 [40:14<8:56:19, 80.45s/it]
                                                  
+
  7%|▋         | 30/430 [40:14<8:56:19, 80.45s/it]
  7%|▋         | 31/430 [41:35<8:55:05, 80.46s/it]
  7%|▋         | 32/430 [42:55<8:53:43, 80.46s/it]
  8%|▊         | 33/430 [44:16<8:52:17, 80.45s/it]
  8%|▊         | 34/430 [45:36<8:50:56, 80.45s/it]
  8%|▊         | 35/430 [46:57<8:49:37, 80.45s/it]
                                                  
+
  8%|▊         | 35/430 [46:57<8:49:37, 80.45s/it]
  8%|▊         | 36/430 [48:17<8:48:22, 80.46s/it]
  9%|▊         | 37/430 [49:38<8:46:59, 80.46s/it]
  9%|▉         | 38/430 [50:58<8:45:38, 80.46s/it]
  9%|▉         | 39/430 [52:19<8:44:24, 80.47s/it]
  9%|▉         | 40/430 [53:39<8:43:03, 80.47s/it]
                                                  
+
  9%|▉         | 40/430 [53:39<8:43:03, 80.47s/it]
 10%|▉         | 41/430 [55:00<8:42:48, 80.64s/it]
 10%|▉         | 42/430 [56:21<8:41:07, 80.59s/it]
 10%|█         | 43/430 [57:41<8:39:31, 80.55s/it]
 10%|█         | 44/430 [59:50<10:12:21, 95.19s/it]
 10%|█         | 45/430 [1:01:11<9:42:25, 90.77s/it]
                                                    
+
 10%|█         | 45/430 [1:01:11<9:42:25, 90.77s/it]
 11%|█         | 46/430 [1:02:31<9:21:05, 87.67s/it]
 11%|█         | 47/430 [1:03:52<9:06:42, 85.65s/it]
 11%|█         | 48/430 [1:05:13<8:55:45, 84.15s/it]
 11%|█▏        | 49/430 [1:06:33<8:47:15, 83.03s/it]
 12%|█▏        | 50/430 [1:07:54<8:41:00, 82.27s/it]
                                                    
+
 12%|█▏        | 50/430 [1:07:54<8:41:00, 82.27s/it]
 12%|█▏        | 51/430 [1:09:14<8:36:14, 81.73s/it]
 12%|█▏        | 52/430 [1:10:35<8:32:33, 81.36s/it]
 12%|█▏        | 53/430 [1:11:55<8:30:00, 81.17s/it]
 13%|█▎        | 54/430 [1:13:16<8:27:22, 80.96s/it]
 13%|█▎        | 55/430 [1:14:37<8:25:21, 80.86s/it]
                                                    
+
 13%|█▎        | 55/430 [1:14:37<8:25:21, 80.86s/it]
 13%|█▎        | 56/430 [1:15:57<8:23:29, 80.77s/it]
 13%|█▎        | 57/430 [1:17:18<8:21:34, 80.68s/it]
 13%|█▎        | 58/430 [1:18:38<8:19:56, 80.63s/it]
 14%|█▎        | 59/430 [1:19:59<8:18:24, 80.60s/it]
 14%|█▍        | 60/430 [1:21:19<8:16:44, 80.55s/it]
                                                    
+
 14%|█▍        | 60/430 [1:21:19<8:16:44, 80.55s/it]
 14%|█▍        | 61/430 [1:22:40<8:15:19, 80.54s/it]
 14%|█▍        | 62/430 [1:24:00<8:13:53, 80.53s/it]
 15%|█▍        | 63/430 [1:25:21<8:12:48, 80.57s/it]
 15%|█▍        | 64/430 [1:26:41<8:11:20, 80.55s/it]
 15%|█▌        | 65/430 [1:28:02<8:09:50, 80.52s/it]
                                                    
+
 15%|█▌        | 65/430 [1:28:02<8:09:50, 80.52s/it]
 15%|█▌        | 66/430 [1:29:22<8:08:31, 80.52s/it]
 16%|█▌        | 67/430 [1:30:43<8:07:03, 80.51s/it]
 16%|█▌        | 68/430 [1:32:03<8:05:39, 80.50s/it]
 16%|█▌        | 69/430 [1:33:24<8:04:17, 80.49s/it]
 16%|█▋        | 70/430 [1:34:44<8:02:50, 80.47s/it]
                                                    
+
 16%|█▋        | 70/430 [1:34:44<8:02:50, 80.47s/it]
 17%|█▋        | 71/430 [1:36:05<8:02:07, 80.58s/it]
 17%|█▋        | 72/430 [1:37:25<8:00:27, 80.52s/it]
 17%|█▋        | 73/430 [1:38:46<7:58:58, 80.50s/it]
 17%|█▋        | 74/430 [1:40:06<7:57:28, 80.47s/it]
 17%|█▋        | 75/430 [1:41:27<7:56:03, 80.46s/it]
                                                    
+
 17%|█▋        | 75/430 [1:41:27<7:56:03, 80.46s/it]
 18%|█▊        | 76/430 [1:42:47<7:54:59, 80.51s/it]
 18%|█▊        | 77/430 [1:44:08<7:53:37, 80.50s/it]
 18%|█▊        | 78/430 [1:45:28<7:52:21, 80.52s/it]
 18%|█▊        | 79/430 [1:46:49<7:50:59, 80.51s/it]
 19%|█▊        | 80/430 [1:48:09<7:49:36, 80.51s/it]
                                                    
+
 19%|█▊        | 80/430 [1:48:09<7:49:36, 80.51s/it]
 19%|█▉        | 81/430 [1:49:30<7:48:23, 80.53s/it]
 19%|█▉        | 82/430 [1:50:50<7:47:05, 80.53s/it]
 19%|█▉        | 83/430 [1:52:11<7:45:34, 80.50s/it]
 20%|█▉        | 84/430 [1:53:32<7:45:20, 80.69s/it]
 20%|█▉        | 85/430 [1:54:52<7:43:34, 80.62s/it]
                                                    
+
 20%|█▉        | 85/430 [1:54:52<7:43:34, 80.62s/it]
 20%|██        | 86/430 [1:56:13<7:42:01, 80.59s/it]
 20%|██        | 87/430 [1:58:23<9:05:42, 95.46s/it]
 20%|██        | 88/430 [1:59:44<8:38:35, 90.98s/it]
 21%|██        | 89/430 [2:01:04<8:19:08, 87.82s/it]
 21%|██        | 90/430 [2:02:25<8:05:18, 85.64s/it]
                                                    
+
 21%|██        | 90/430 [2:02:25<8:05:18, 85.64s/it]
 21%|██        | 91/430 [2:03:45<7:55:07, 84.09s/it]
 21%|██▏       | 92/430 [2:05:06<7:47:38, 83.01s/it]
 22%|██▏       | 93/430 [2:06:26<7:42:05, 82.27s/it]
 22%|██▏       | 94/430 [2:07:47<7:37:48, 81.75s/it]
 22%|██▏       | 95/430 [2:09:07<7:34:41, 81.44s/it]
                                                    
+
 22%|██▏       | 95/430 [2:09:07<7:34:41, 81.44s/it]
 22%|██▏       | 96/430 [2:10:28<7:31:43, 81.15s/it]
 23%|██▎       | 97/430 [2:11:48<7:29:16, 80.95s/it]
 23%|██▎       | 98/430 [2:13:09<7:27:04, 80.80s/it]
 23%|██▎       | 99/430 [2:14:29<7:25:22, 80.73s/it]
 23%|██▎       | 100/430 [2:15:50<7:23:40, 80.67s/it]
                                                     
+
 23%|██▎       | 100/430 [2:15:50<7:23:40, 80.67s/it]
 23%|██▎       | 101/430 [2:17:10<7:22:04, 80.62s/it]
 24%|██▎       | 102/430 [2:18:31<7:20:28, 80.58s/it]
 24%|██▍       | 103/430 [2:19:51<7:18:57, 80.54s/it]
 24%|██▍       | 104/430 [2:21:12<7:17:41, 80.56s/it]
 24%|██▍       | 105/430 [2:22:32<7:16:17, 80.55s/it]
                                                     
+
 24%|██▍       | 105/430 [2:22:32<7:16:17, 80.55s/it]
 25%|██▍       | 106/430 [2:23:53<7:15:01, 80.56s/it]
 25%|██▍       | 107/430 [2:25:14<7:13:51, 80.59s/it]
 25%|██▌       | 108/430 [2:26:34<7:12:27, 80.58s/it]
 25%|██▌       | 109/430 [2:27:55<7:11:11, 80.60s/it]
 26%|██▌       | 110/430 [2:29:15<7:09:47, 80.59s/it]
                                                     
+
 26%|██▌       | 110/430 [2:29:15<7:09:47, 80.59s/it]
 26%|██▌       | 111/430 [2:30:36<7:08:21, 80.57s/it]
 26%|██▌       | 112/430 [2:31:57<7:07:20, 80.63s/it]
 26%|██▋       | 113/430 [2:33:17<7:05:49, 80.60s/it]
 27%|██▋       | 114/430 [2:34:38<7:04:22, 80.58s/it]
 27%|██▋       | 115/430 [2:35:58<7:02:59, 80.57s/it]
                                                     
+
 27%|██▋       | 115/430 [2:35:58<7:02:59, 80.57s/it]
 27%|██▋       | 116/430 [2:37:20<7:02:34, 80.75s/it]
 27%|██▋       | 117/430 [2:38:40<7:00:55, 80.69s/it]
 27%|██▋       | 118/430 [2:40:01<6:59:23, 80.65s/it]
 28%|██▊       | 119/430 [2:41:21<6:57:54, 80.63s/it]
 28%|██▊       | 120/430 [2:42:42<6:56:26, 80.60s/it]
                                                     
+
 28%|██▊       | 120/430 [2:42:42<6:56:26, 80.60s/it]
 28%|██▊       | 121/430 [2:44:02<6:55:00, 80.58s/it]
 28%|██▊       | 122/430 [2:45:23<6:53:30, 80.55s/it]
 29%|██▊       | 123/430 [2:46:43<6:52:17, 80.58s/it]
 29%|██▉       | 124/430 [2:48:04<6:51:04, 80.60s/it]
 29%|██▉       | 125/430 [2:49:25<6:49:38, 80.59s/it]
                                                     
+
 29%|██▉       | 125/430 [2:49:25<6:49:38, 80.59s/it]
 29%|██▉       | 126/430 [2:50:45<6:48:24, 80.61s/it]
 30%|██▉       | 127/430 [2:52:06<6:47:00, 80.60s/it]
 30%|██▉       | 128/430 [2:53:27<6:46:32, 80.77s/it]
 30%|███       | 129/430 [2:54:48<6:44:56, 80.72s/it]
 30%|███       | 130/430 [2:56:57<7:56:43, 95.35s/it]
                                                     
+
 30%|███       | 130/430 [2:56:57<7:56:43, 95.35s/it]
 30%|███       | 131/430 [2:58:18<7:32:58, 90.90s/it]
 31%|███       | 132/430 [2:59:38<7:15:53, 87.76s/it]
 31%|███       | 133/430 [3:00:59<7:03:46, 85.61s/it]
 31%|███       | 134/430 [3:02:19<6:54:52, 84.10s/it]
 31%|███▏      | 135/430 [3:03:40<6:48:13, 83.03s/it]
                                                     
+
 31%|███▏      | 135/430 [3:03:40<6:48:13, 83.03s/it]
 32%|███▏      | 136/430 [3:05:00<6:43:08, 82.27s/it]
 32%|███▏      | 137/430 [3:06:21<6:39:15, 81.76s/it]
 32%|███▏      | 138/430 [3:07:41<6:36:06, 81.39s/it]
 32%|███▏      | 139/430 [3:09:02<6:33:50, 81.20s/it]
 33%|███▎      | 140/430 [3:10:23<6:31:28, 81.00s/it]
                                                     
+
 33%|███▎      | 140/430 [3:10:23<6:31:28, 81.00s/it]
 33%|███▎      | 141/430 [3:11:43<6:29:27, 80.86s/it]
 33%|███▎      | 142/430 [3:13:04<6:27:32, 80.74s/it]
 33%|███▎      | 143/430 [3:14:24<6:26:05, 80.72s/it]
 33%|███▎      | 144/430 [3:15:45<6:24:25, 80.65s/it]
 34%|███▎      | 145/430 [3:17:05<6:22:43, 80.57s/it]
                                                     
+
 34%|███▎      | 145/430 [3:17:05<6:22:43, 80.57s/it]
 34%|███▍      | 146/430 [3:18:26<6:21:11, 80.53s/it]
 34%|███▍      | 147/430 [3:19:46<6:19:49, 80.53s/it]
 34%|███▍      | 148/430 [3:21:07<6:18:31, 80.54s/it]
 35%|███▍      | 149/430 [3:22:27<6:17:16, 80.56s/it]
 35%|███▍      | 150/430 [3:23:48<6:15:53, 80.55s/it]
                                                     
+
 35%|███▍      | 150/430 [3:23:48<6:15:53, 80.55s/it]Saving model checkpoint to ./results/checkpoint-150
+Configuration saved in ./results/checkpoint-150/config.json
+Model weights saved in ./results/checkpoint-150/pytorch_model.bin
+tokenizer config file saved in ./results/checkpoint-150/tokenizer_config.json
+Special tokens file saved in ./results/checkpoint-150/special_tokens_map.json
+[2023-02-20 20:30:17,356] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step151 is begin to save!
+/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
+  warnings.warn(
+/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
+  warnings.warn(
+/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
+  warnings.warn(
+/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
+  warnings.warn(
+/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
+  warnings.warn(
+/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
+  warnings.warn(
+/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
+  warnings.warn(
+[2023-02-20 20:30:17,359] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-150/global_step151/mp_rank_00_model_states.pt
+[2023-02-20 20:30:17,359] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-150/global_step151/mp_rank_00_model_states.pt...
+[2023-02-20 20:30:18,018] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-150/global_step151/mp_rank_00_model_states.pt.
+[2023-02-20 20:30:18,019] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-150/global_step151/zero_pp_rank_0_mp_rank_00_optim_states.pt...
+[2023-02-20 20:30:18,223] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-150/global_step151/zero_pp_rank_0_mp_rank_00_optim_states.pt.
+[2023-02-20 20:30:18,223] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-150/global_step151/zero_pp_rank_0_mp_rank_00_optim_states.pt
+[2023-02-20 20:30:18,224] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step151 is ready now!
+Deleting older checkpoint [results/checkpoint-45] due to args.save_total_limit
+
 35%|███▌      | 151/430 [3:25:10<6:16:59, 81.07s/it]
 35%|███▌      | 152/430 [3:26:31<6:14:42, 80.87s/it]
 36%|███▌      | 153/430 [3:27:51<6:12:50, 80.76s/it]
 36%|███▌      | 154/430 [3:29:11<6:11:03, 80.67s/it]
 36%|███▌      | 155/430 [3:30:32<6:09:25, 80.60s/it]
                                                     
+
 36%|███▌      | 155/430 [3:30:32<6:09:25, 80.60s/it]
 36%|███▋      | 156/430 [3:31:52<6:07:55, 80.57s/it]
 37%|███▋      | 157/430 [3:33:13<6:07:08, 80.69s/it]
 37%|███▋      | 158/430 [3:34:34<6:05:33, 80.64s/it]
 37%|███▋      | 159/430 [3:35:54<6:04:05, 80.61s/it]
 37%|███▋      | 160/430 [3:37:15<6:02:37, 80.58s/it]
                                                     
+
 37%|███▋      | 160/430 [3:37:15<6:02:37, 80.58s/it]
 37%|███▋      | 161/430 [3:38:36<6:01:14, 80.58s/it]
 38%|███▊      | 162/430 [3:39:56<6:00:01, 80.60s/it]
 38%|███▊      | 163/430 [3:41:17<5:59:32, 80.80s/it]
 38%|███▊      | 164/430 [3:42:38<5:58:27, 80.86s/it]
 38%|███▊      | 165/430 [3:43:59<5:56:38, 80.75s/it]
                                                     
+
 38%|███▊      | 165/430 [3:43:59<5:56:38, 80.75s/it]
 39%|███▊      | 166/430 [3:45:20<5:55:05, 80.70s/it]
 39%|███▉      | 167/430 [3:46:40<5:53:32, 80.66s/it]
 39%|███▉      | 168/430 [3:48:01<5:52:04, 80.63s/it]
 39%|███▉      | 169/430 [3:49:21<5:50:54, 80.67s/it]
 40%|███▉      | 170/430 [3:50:42<5:49:25, 80.64s/it]
                                                     
+
 40%|███▉      | 170/430 [3:50:42<5:49:25, 80.64s/it]
 40%|███▉      | 171/430 [3:52:03<5:48:04, 80.64s/it]
 40%|████      | 172/430 [3:53:23<5:46:33, 80.59s/it]
 40%|████      | 173/430 [3:55:33<6:48:29, 95.37s/it]
 40%|████      | 174/430 [3:56:53<6:27:52, 90.91s/it]
 41%|████      | 175/430 [3:58:14<6:13:17, 87.83s/it]
                                                     
+
 41%|████      | 175/430 [3:58:14<6:13:17, 87.83s/it]
 41%|████      | 176/430 [3:59:35<6:02:43, 85.68s/it]
 41%|████      | 177/430 [4:00:55<5:54:43, 84.12s/it]
 41%|████▏     | 178/430 [4:02:16<5:48:45, 83.04s/it]
 42%|████▏     | 179/430 [4:03:36<5:44:09, 82.27s/it]
 42%|████▏     | 180/430 [4:04:57<5:40:41, 81.77s/it]
                                                     
+
 42%|████▏     | 180/430 [4:04:57<5:40:41, 81.77s/it]
 42%|████▏     | 181/430 [4:06:17<5:37:50, 81.41s/it]
 42%|████▏     | 182/430 [4:07:38<5:35:22, 81.14s/it]
 43%|████▎     | 183/430 [4:08:58<5:33:16, 80.96s/it]
 43%|████▎     | 184/430 [4:10:19<5:31:25, 80.83s/it]
 43%|████▎     | 185/430 [4:11:40<5:29:40, 80.74s/it]
                                                     
+
 43%|████▎     | 185/430 [4:11:40<5:29:40, 80.74s/it]
 43%|████▎     | 186/430 [4:13:00<5:28:03, 80.67s/it]
 43%|████▎     | 187/430 [4:14:21<5:26:44, 80.68s/it]
 44%|████▎     | 188/430 [4:15:41<5:25:24, 80.68s/it]
 44%|████▍     | 189/430 [4:17:02<5:23:51, 80.63s/it]
 44%|████▍     | 190/430 [4:18:22<5:22:20, 80.58s/it]
                                                     
+
 44%|████▍     | 190/430 [4:18:22<5:22:20, 80.58s/it]
 44%|████▍     | 191/430 [4:19:43<5:20:53, 80.56s/it]
 45%|████▍     | 192/430 [4:21:03<5:19:33, 80.56s/it]
 45%|████▍     | 193/430 [4:22:24<5:18:45, 80.70s/it]
 45%|████▌     | 194/430 [4:23:45<5:17:14, 80.65s/it]
 45%|████▌     | 195/430 [4:25:06<5:15:47, 80.63s/it]
                                                     
+
 45%|████▌     | 195/430 [4:25:06<5:15:47, 80.63s/it]
 46%|████▌     | 196/430 [4:26:26<5:14:14, 80.57s/it]
 46%|████▌     | 197/430 [4:27:47<5:12:46, 80.54s/it]
 46%|████▌     | 198/430 [4:29:07<5:11:27, 80.55s/it]
 46%|████▋     | 199/430 [4:30:28<5:10:06, 80.55s/it]
 47%|████▋     | 200/430 [4:31:48<5:08:45, 80.55s/it]
                                                     
+
 47%|████▋     | 200/430 [4:31:48<5:08:45, 80.55s/it]
 47%|████▋     | 201/430 [4:33:09<5:07:25, 80.55s/it]
 47%|████▋     | 202/430 [4:34:29<5:06:03, 80.54s/it]
 47%|████▋     | 203/430 [4:35:50<5:04:47, 80.56s/it]
 47%|████▋     | 204/430 [4:37:10<5:03:25, 80.55s/it]
 48%|████▊     | 205/430 [4:38:31<5:02:05, 80.56s/it]
                                                     
+
 48%|████▊     | 205/430 [4:38:31<5:02:05, 80.56s/it]
 48%|████▊     | 206/430 [4:39:52<5:00:48, 80.57s/it]
 48%|████▊     | 207/430 [4:41:12<4:59:23, 80.55s/it]
 48%|████▊     | 208/430 [4:42:33<4:58:04, 80.56s/it]
 49%|████▊     | 209/430 [4:43:53<4:56:40, 80.54s/it]
 49%|████▉     | 210/430 [4:45:14<4:55:18, 80.54s/it]
                                                     
+
 49%|████▉     | 210/430 [4:45:14<4:55:18, 80.54s/it]
 49%|████▉     | 211/430 [4:46:35<4:54:27, 80.67s/it]
 49%|████▉     | 212/430 [4:47:56<4:53:37, 80.81s/it]
 50%|████▉     | 213/430 [4:49:16<4:52:03, 80.75s/it]
 50%|████▉     | 214/430 [4:50:37<4:50:43, 80.75s/it]
 50%|█████     | 215/430 [4:51:58<4:49:04, 80.67s/it]
                                                     
+
 50%|█████     | 215/430 [4:51:58<4:49:04, 80.67s/it]
 50%|█████     | 216/430 [4:54:08<5:40:27, 95.45s/it]
 50%|█████     | 217/430 [4:55:28<5:22:58, 90.98s/it]
 51%|█████     | 218/430 [4:56:49<5:10:22, 87.84s/it]
 51%|█████     | 219/430 [4:58:09<5:01:10, 85.64s/it]
 51%|█████     | 220/430 [4:59:30<4:54:31, 84.15s/it]
                                                     
+
 51%|█████     | 220/430 [4:59:30<4:54:31, 84.15s/it]
 51%|█████▏    | 221/430 [5:00:50<4:49:24, 83.09s/it]
 52%|█████▏    | 222/430 [5:02:11<4:45:37, 82.39s/it]
 52%|█████▏    | 223/430 [5:03:32<4:42:20, 81.84s/it]
 52%|█████▏    | 224/430 [5:04:52<4:39:35, 81.44s/it]
 52%|█████▏    | 225/430 [5:06:13<4:37:20, 81.17s/it]
                                                     
+
 52%|█████▏    | 225/430 [5:06:13<4:37:20, 81.17s/it]
 53%|█████▎    | 226/430 [5:07:33<4:35:24, 81.00s/it]
 53%|█████▎    | 227/430 [5:08:54<4:33:33, 80.85s/it]
 53%|█████▎    | 228/430 [5:10:14<4:31:47, 80.73s/it]
 53%|█████▎    | 229/430 [5:11:35<4:30:14, 80.67s/it]
 53%|█████▎    | 230/430 [5:12:55<4:28:43, 80.62s/it]
                                                     
+
 53%|█████▎    | 230/430 [5:12:55<4:28:43, 80.62s/it]
 54%|█████▎    | 231/430 [5:14:16<4:27:16, 80.58s/it]
 54%|█████▍    | 232/430 [5:15:36<4:25:49, 80.55s/it]
 54%|█████▍    | 233/430 [5:16:57<4:24:23, 80.53s/it]
 54%|█████▍    | 234/430 [5:18:17<4:22:58, 80.50s/it]
 55%|█████▍    | 235/430 [5:19:38<4:21:32, 80.48s/it]
                                                     
+
 55%|█████▍    | 235/430 [5:19:38<4:21:32, 80.48s/it]
 55%|█████▍    | 236/430 [5:20:58<4:20:16, 80.50s/it]
 55%|█████▌    | 237/430 [5:22:19<4:18:53, 80.49s/it]
 55%|█████▌    | 238/430 [5:23:39<4:17:33, 80.49s/it]
 56%|█████▌    | 239/430 [5:25:00<4:16:10, 80.47s/it]
 56%|█████▌    | 240/430 [5:26:20<4:14:47, 80.46s/it]
                                                     
+
 56%|█████▌    | 240/430 [5:26:20<4:14:47, 80.46s/it]
 56%|█████▌    | 241/430 [5:27:40<4:13:24, 80.45s/it]
 56%|█████▋    | 242/430 [5:29:01<4:12:06, 80.46s/it]
 57%|█████▋    | 243/430 [5:30:21<4:10:48, 80.47s/it]
 57%|█████▋    | 244/430 [5:31:42<4:09:29, 80.48s/it]
 57%|█████▋    | 245/430 [5:33:02<4:08:08, 80.48s/it]
                                                     
+
 57%|█████▋    | 245/430 [5:33:02<4:08:08, 80.48s/it]
 57%|█████▋    | 246/430 [5:34:23<4:06:46, 80.47s/it]
 57%|█████▋    | 247/430 [5:35:44<4:05:59, 80.65s/it]
 58%|█████▊    | 248/430 [5:37:04<4:04:28, 80.59s/it]
 58%|█████▊    | 249/430 [5:38:25<4:03:06, 80.59s/it]
 58%|█████▊    | 250/430 [5:39:46<4:01:45, 80.59s/it]
                                                     
+
 58%|█████▊    | 250/430 [5:39:46<4:01:45, 80.59s/it]
 58%|█████▊    | 251/430 [5:41:06<4:00:32, 80.63s/it]
 59%|█████▊    | 252/430 [5:42:27<3:59:37, 80.77s/it]
 59%|█████▉    | 253/430 [5:43:48<3:58:03, 80.70s/it]
 59%|█████▉    | 254/430 [5:45:08<3:56:32, 80.64s/it]
 59%|█████▉    | 255/430 [5:46:29<3:55:05, 80.60s/it]
                                                     
+
 59%|█████▉    | 255/430 [5:46:29<3:55:05, 80.60s/it]
 60%|█████▉    | 256/430 [5:47:50<3:54:05, 80.72s/it]
 60%|█████▉    | 257/430 [5:49:10<3:52:33, 80.65s/it]
 60%|██████    | 258/430 [5:50:31<3:51:09, 80.64s/it]
 60%|██████    | 259/430 [5:52:40<4:31:29, 95.26s/it]
 60%|██████    | 260/430 [5:54:01<4:17:26, 90.86s/it]
                                                     
+
 60%|██████    | 260/430 [5:54:01<4:17:26, 90.86s/it]
 61%|██████    | 261/430 [5:55:22<4:07:10, 87.75s/it]
 61%|██████    | 262/430 [5:56:42<3:59:47, 85.64s/it]
 61%|██████    | 263/430 [5:58:03<3:54:02, 84.09s/it]
 61%|██████▏   | 264/430 [5:59:23<3:49:37, 83.00s/it]
 62%|██████▏   | 265/430 [6:00:44<3:46:08, 82.23s/it]
                                                     
+
 62%|██████▏   | 265/430 [6:00:44<3:46:08, 82.23s/it]
 62%|██████▏   | 266/430 [6:02:04<3:43:19, 81.70s/it]
 62%|██████▏   | 267/430 [6:03:25<3:40:59, 81.34s/it]
 62%|██████▏   | 268/430 [6:04:45<3:39:08, 81.16s/it]
 63%|██████▎   | 269/430 [6:06:06<3:37:16, 80.97s/it]
 63%|██████▎   | 270/430 [6:07:26<3:35:33, 80.84s/it]
                                                     
+
 63%|██████▎   | 270/430 [6:07:26<3:35:33, 80.84s/it]
 63%|██████▎   | 271/430 [6:08:47<3:34:06, 80.80s/it]
 63%|██████▎   | 272/430 [6:10:08<3:32:33, 80.72s/it]
 63%|██████▎   | 273/430 [6:11:28<3:31:01, 80.65s/it]
 64%|██████▎   | 274/430 [6:12:49<3:29:32, 80.60s/it]
 64%|██████▍   | 275/430 [6:14:09<3:28:03, 80.54s/it]
                                                     
+
 64%|██████▍   | 275/430 [6:14:09<3:28:03, 80.54s/it]
 64%|██████▍   | 276/430 [6:15:29<3:26:41, 80.53s/it]
 64%|██████▍   | 277/430 [6:16:50<3:25:18, 80.51s/it]
 65%|██████▍   | 278/430 [6:18:10<3:23:55, 80.50s/it]
 65%|██████▍   | 279/430 [6:19:31<3:22:38, 80.52s/it]
 65%|██████▌   | 280/430 [6:20:52<3:21:30, 80.60s/it]
                                                     
+
 65%|██████▌   | 280/430 [6:20:52<3:21:30, 80.60s/it]
 65%|██████▌   | 281/430 [6:22:12<3:20:06, 80.58s/it]
 66%|██████▌   | 282/430 [6:23:33<3:18:45, 80.58s/it]
 66%|██████▌   | 283/430 [6:24:54<3:17:25, 80.58s/it]
 66%|██████▌   | 284/430 [6:26:14<3:16:05, 80.59s/it]
 66%|██████▋   | 285/430 [6:27:35<3:14:40, 80.56s/it]
                                                     
+
 66%|██████▋   | 285/430 [6:27:35<3:14:40, 80.56s/it]
 67%|██████▋   | 286/430 [6:28:55<3:13:19, 80.55s/it]
 67%|██████▋   | 287/430 [6:30:16<3:11:55, 80.53s/it]
 67%|██████▋   | 288/430 [6:31:37<3:10:52, 80.65s/it]
 67%|██████▋   | 289/430 [6:32:57<3:09:26, 80.62s/it]
 67%|██████▋   | 290/430 [6:34:18<3:07:59, 80.57s/it]
                                                     
+
 67%|██████▋   | 290/430 [6:34:18<3:07:59, 80.57s/it]
 68%|██████▊   | 291/430 [6:35:39<3:06:56, 80.69s/it]
 68%|██████▊   | 292/430 [6:36:59<3:05:26, 80.63s/it]
 68%|██████▊   | 293/430 [6:38:20<3:04:03, 80.61s/it]
 68%|██████▊   | 294/430 [6:39:40<3:02:37, 80.57s/it]
 69%|██████▊   | 295/430 [6:41:01<3:01:16, 80.56s/it]
                                                     
+
 69%|██████▊   | 295/430 [6:41:01<3:01:16, 80.56s/it]
 69%|██████▉   | 296/430 [6:42:21<2:59:53, 80.55s/it]
 69%|██████▉   | 297/430 [6:43:42<2:58:55, 80.72s/it]
 69%|██████▉   | 298/430 [6:45:03<2:57:26, 80.66s/it]
 70%|██████▉   | 299/430 [6:46:23<2:55:57, 80.59s/it]
 70%|██████▉   | 300/430 [6:47:44<2:54:31, 80.55s/it]
                                                     
+
 70%|██████▉   | 300/430 [6:47:44<2:54:31, 80.55s/it]Saving model checkpoint to ./results/checkpoint-300
+Configuration saved in ./results/checkpoint-300/config.json
+Model weights saved in ./results/checkpoint-300/pytorch_model.bin
+tokenizer config file saved in ./results/checkpoint-300/tokenizer_config.json
+Special tokens file saved in ./results/checkpoint-300/special_tokens_map.json
+[2023-02-20 23:54:13,030] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step303 is begin to save!
+[2023-02-20 23:54:13,032] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-300/global_step303/mp_rank_00_model_states.pt
+[2023-02-20 23:54:13,032] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-300/global_step303/mp_rank_00_model_states.pt...
+[2023-02-20 23:54:13,586] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-300/global_step303/mp_rank_00_model_states.pt.
+[2023-02-20 23:54:13,587] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-300/global_step303/zero_pp_rank_0_mp_rank_00_optim_states.pt...
+[2023-02-20 23:54:13,739] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-300/global_step303/zero_pp_rank_0_mp_rank_00_optim_states.pt.
+[2023-02-20 23:54:13,740] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-300/global_step303/zero_pp_rank_0_mp_rank_00_optim_states.pt
+[2023-02-20 23:54:13,740] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step303 is ready now!
+Deleting older checkpoint [results/checkpoint-150] due to args.save_total_limit
+
 70%|███████   | 301/430 [6:49:06<2:54:08, 80.99s/it]
 70%|███████   | 302/430 [6:51:15<3:24:02, 95.64s/it]
 70%|███████   | 303/430 [6:52:36<3:12:48, 91.09s/it]
 71%|███████   | 304/430 [6:53:56<3:04:36, 87.91s/it]
 71%|███████   | 305/430 [6:55:17<2:58:30, 85.68s/it]
                                                     
+
 71%|███████   | 305/430 [6:55:17<2:58:30, 85.68s/it]
 71%|███████   | 306/430 [6:56:38<2:54:00, 84.19s/it]
 71%|███████▏  | 307/430 [6:57:59<2:50:44, 83.29s/it]
 72%|███████▏  | 308/430 [6:59:19<2:47:40, 82.46s/it]
 72%|███████▏  | 309/430 [7:00:40<2:45:10, 81.91s/it]
 72%|███████▏  | 310/430 [7:02:00<2:42:57, 81.48s/it]
                                                     
+
 72%|███████▏  | 310/430 [7:02:00<2:42:57, 81.48s/it]
 72%|███████▏  | 311/430 [7:03:21<2:41:01, 81.18s/it]
 73%|███████▎  | 312/430 [7:04:41<2:39:14, 80.97s/it]
 73%|███████▎  | 313/430 [7:06:02<2:37:35, 80.82s/it]
 73%|███████▎  | 314/430 [7:07:22<2:36:02, 80.71s/it]
 73%|███████▎  | 315/430 [7:08:43<2:34:33, 80.64s/it]
                                                     
+
 73%|███████▎  | 315/430 [7:08:43<2:34:33, 80.64s/it]
 73%|███████▎  | 316/430 [7:10:03<2:33:10, 80.62s/it]
 74%|███████▎  | 317/430 [7:11:24<2:31:46, 80.59s/it]
 74%|███████▍  | 318/430 [7:12:44<2:30:23, 80.56s/it]
 74%|███████▍  | 319/430 [7:14:05<2:29:01, 80.56s/it]
 74%|███████▍  | 320/430 [7:15:26<2:27:42, 80.57s/it]
                                                     
+
 74%|███████▍  | 320/430 [7:15:26<2:27:42, 80.57s/it]
 75%|███████▍  | 321/430 [7:16:46<2:26:19, 80.54s/it]
 75%|███████▍  | 322/430 [7:18:07<2:24:58, 80.54s/it]
 75%|███████▌  | 323/430 [7:19:27<2:23:39, 80.55s/it]
 75%|███████▌  | 324/430 [7:20:48<2:22:15, 80.53s/it]
 76%|███████▌  | 325/430 [7:22:08<2:20:55, 80.53s/it]
                                                     
+
 76%|███████▌  | 325/430 [7:22:08<2:20:55, 80.53s/it]
 76%|███████▌  | 326/430 [7:23:29<2:19:34, 80.52s/it]
 76%|███████▌  | 327/430 [7:24:49<2:18:11, 80.50s/it]
 76%|███████▋  | 328/430 [7:26:10<2:16:52, 80.52s/it]
 77%|███████▋  | 329/430 [7:27:30<2:15:32, 80.52s/it]
 77%|███████▋  | 330/430 [7:28:51<2:14:13, 80.53s/it]
                                                     
+
 77%|███████▋  | 330/430 [7:28:51<2:14:13, 80.53s/it]
 77%|███████▋  | 331/430 [7:30:11<2:12:57, 80.58s/it]
 77%|███████▋  | 332/430 [7:31:32<2:11:44, 80.66s/it]
 77%|███████▋  | 333/430 [7:32:53<2:10:32, 80.75s/it]
 78%|███████▊  | 334/430 [7:34:14<2:09:04, 80.67s/it]
 78%|███████▊  | 335/430 [7:35:34<2:07:39, 80.62s/it]
                                                     
+
 78%|███████▊  | 335/430 [7:35:34<2:07:39, 80.62s/it]
 78%|███████▊  | 336/430 [7:36:55<2:06:30, 80.75s/it]
 78%|███████▊  | 337/430 [7:38:16<2:05:08, 80.73s/it]
 79%|███████▊  | 338/430 [7:39:36<2:03:41, 80.66s/it]
 79%|███████▉  | 339/430 [7:40:57<2:02:15, 80.61s/it]
 79%|███████▉  | 340/430 [7:42:17<2:00:50, 80.56s/it]
                                                     
+
 79%|███████▉  | 340/430 [7:42:17<2:00:50, 80.56s/it]
 79%|███████▉  | 341/430 [7:43:38<1:59:27, 80.53s/it]
 80%|███████▉  | 342/430 [7:44:58<1:58:06, 80.53s/it]
 80%|███████▉  | 343/430 [7:46:19<1:56:47, 80.55s/it]
 80%|████████  | 344/430 [7:47:40<1:55:30, 80.59s/it]
 80%|████████  | 345/430 [7:49:49<2:14:57, 95.26s/it]
                                                     
+
 80%|████████  | 345/430 [7:49:49<2:14:57, 95.26s/it]
 80%|████████  | 346/430 [7:51:10<2:07:10, 90.84s/it]
 81%|████████  | 347/430 [7:52:30<2:01:21, 87.73s/it]
 81%|████████  | 348/430 [7:53:51<1:56:55, 85.56s/it]
 81%|████████  | 349/430 [7:55:11<1:53:26, 84.03s/it]
 81%|████████▏ | 350/430 [7:56:32<1:50:39, 83.00s/it]
                                                     
+
 81%|████████▏ | 350/430 [7:56:32<1:50:39, 83.00s/it]
 82%|████████▏ | 351/430 [7:57:52<1:48:17, 82.24s/it]
 82%|████████▏ | 352/430 [7:59:13<1:46:14, 81.72s/it]
 82%|████████▏ | 353/430 [8:00:34<1:44:40, 81.56s/it]
 82%|████████▏ | 354/430 [8:01:54<1:42:53, 81.22s/it]
 83%|████████▎ | 355/430 [8:03:15<1:41:14, 80.99s/it]
                                                     
+
 83%|████████▎ | 355/430 [8:03:15<1:41:14, 80.99s/it]
 83%|████████▎ | 356/430 [8:04:36<1:39:57, 81.05s/it]
 83%|████████▎ | 357/430 [8:05:56<1:38:25, 80.90s/it]
 83%|████████▎ | 358/430 [8:07:17<1:36:55, 80.77s/it]
 83%|████████▎ | 359/430 [8:08:38<1:35:30, 80.71s/it]
 84%|████████▎ | 360/430 [8:09:58<1:34:03, 80.63s/it]
                                                     
+
 84%|████████▎ | 360/430 [8:09:58<1:34:03, 80.63s/it]
 84%|████████▍ | 361/430 [8:11:18<1:32:39, 80.57s/it]
 84%|████████▍ | 362/430 [8:12:39<1:31:17, 80.55s/it]
 84%|████████▍ | 363/430 [8:13:59<1:29:56, 80.54s/it]
 85%|████████▍ | 364/430 [8:15:20<1:28:34, 80.52s/it]
 85%|████████▍ | 365/430 [8:16:40<1:27:12, 80.50s/it]
                                                     
+
 85%|████████▍ | 365/430 [8:16:40<1:27:12, 80.50s/it]
 85%|████████▌ | 366/430 [8:18:01<1:25:49, 80.47s/it]
 85%|████████▌ | 367/430 [8:19:21<1:24:27, 80.44s/it]
 86%|████████▌ | 368/430 [8:20:42<1:23:10, 80.49s/it]
 86%|████████▌ | 369/430 [8:22:02<1:21:48, 80.47s/it]
 86%|████████▌ | 370/430 [8:23:23<1:20:29, 80.49s/it]
                                                     
+
 86%|████████▌ | 370/430 [8:23:23<1:20:29, 80.49s/it]
 86%|████████▋ | 371/430 [8:24:43<1:19:09, 80.51s/it]
 87%|████████▋ | 372/430 [8:26:04<1:17:55, 80.61s/it]
 87%|████████▋ | 373/430 [8:27:25<1:16:32, 80.57s/it]
 87%|████████▋ | 374/430 [8:28:45<1:15:10, 80.55s/it]
 87%|████████▋ | 375/430 [8:30:06<1:13:49, 80.54s/it]
                                                     
+
 87%|████████▋ | 375/430 [8:30:06<1:13:49, 80.54s/it]
 87%|████████▋ | 376/430 [8:31:26<1:12:29, 80.54s/it]
 88%|████████▊ | 377/430 [8:32:47<1:11:07, 80.53s/it]
 88%|████████▊ | 378/430 [8:34:07<1:09:47, 80.53s/it]
 88%|████████▊ | 379/430 [8:35:28<1:08:26, 80.52s/it]
 88%|████████▊ | 380/430 [8:36:48<1:07:07, 80.55s/it]
                                                     
+
 88%|████████▊ | 380/430 [8:36:48<1:07:07, 80.55s/it]
 89%|████████▊ | 381/430 [8:38:09<1:05:47, 80.56s/it]
 89%|████████▉ | 382/430 [8:39:29<1:04:25, 80.53s/it]
 89%|████████▉ | 383/430 [8:40:50<1:03:04, 80.53s/it]
 89%|████████▉ | 384/430 [8:42:10<1:01:43, 80.51s/it]
 90%|████████▉ | 385/430 [8:43:31<1:00:22, 80.49s/it]
                                                     
+
 90%|████████▉ | 385/430 [8:43:31<1:00:22, 80.49s/it]
 90%|████████▉ | 386/430 [8:44:52<59:06, 80.61s/it]  
 90%|█████████ | 387/430 [8:46:12<57:43, 80.55s/it]
 90%|█████████ | 388/430 [8:48:22<1:06:42, 95.31s/it]
 90%|█████████ | 389/430 [8:49:42<1:02:05, 90.85s/it]
 91%|█████████ | 390/430 [8:51:03<58:29, 87.73s/it]  
                                                   
+
 91%|█████████ | 390/430 [8:51:03<58:29, 87.73s/it]
 91%|█████████ | 391/430 [8:52:23<55:37, 85.58s/it]
 91%|█████████ | 392/430 [8:53:44<53:14, 84.05s/it]
 91%|█████████▏| 393/430 [8:55:04<51:10, 82.98s/it]
 92%|█████████▏| 394/430 [8:56:25<49:25, 82.39s/it]
 92%|█████████▏| 395/430 [8:57:46<47:43, 81.82s/it]
                                                   
+
 92%|█████████▏| 395/430 [8:57:46<47:43, 81.82s/it]
 92%|█████████▏| 396/430 [8:59:06<46:09, 81.45s/it]
 92%|█████████▏| 397/430 [9:00:27<44:37, 81.15s/it]
 93%|█████████▎| 398/430 [9:01:47<43:09, 80.93s/it]
 93%|█████████▎| 399/430 [9:03:08<41:44, 80.80s/it]
 93%|█████████▎| 400/430 [9:04:28<40:20, 80.70s/it]
                                                   
+
 93%|█████████▎| 400/430 [9:04:28<40:20, 80.70s/it]
 93%|█████████▎| 401/430 [9:05:49<38:58, 80.65s/it]
 93%|█████████▎| 402/430 [9:07:09<37:38, 80.65s/it]
 94%|█████████▎| 403/430 [9:08:30<36:16, 80.59s/it]
 94%|█████████▍| 404/430 [9:09:50<34:54, 80.56s/it]
 94%|█████████▍| 405/430 [9:11:11<33:33, 80.54s/it]
                                                   
+
 94%|█████████▍| 405/430 [9:11:11<33:33, 80.54s/it]
 94%|█████████▍| 406/430 [9:12:32<32:14, 80.61s/it]
 95%|█████████▍| 407/430 [9:13:52<30:52, 80.56s/it]
 95%|█████████▍| 408/430 [9:15:12<29:31, 80.54s/it]
 95%|█████████▌| 409/430 [9:16:33<28:11, 80.54s/it]
 95%|█████████▌| 410/430 [9:17:53<26:50, 80.51s/it]
                                                   
+
 95%|█████████▌| 410/430 [9:17:53<26:50, 80.51s/it]
 96%|█████████▌| 411/430 [9:19:15<25:33, 80.70s/it]
 96%|█████████▌| 412/430 [9:20:35<24:11, 80.65s/it]
 96%|█████████▌| 413/430 [9:21:56<22:50, 80.61s/it]
 96%|█████████▋| 414/430 [9:23:16<21:28, 80.56s/it]
 97%|█████████▋| 415/430 [9:24:37<20:09, 80.62s/it]
                                                   
+
 97%|█████████▋| 415/430 [9:24:37<20:09, 80.62s/it]
 97%|█████████▋| 416/430 [9:25:57<18:48, 80.58s/it]
 97%|█████████▋| 417/430 [9:27:18<17:27, 80.54s/it]
 97%|█████████▋| 418/430 [9:28:38<16:06, 80.54s/it]
 97%|█████████▋| 419/430 [9:29:59<14:45, 80.51s/it]
 98%|█████████▊| 420/430 [9:31:20<13:26, 80.65s/it]
                                                   
+
 98%|█████████▊| 420/430 [9:31:20<13:26, 80.65s/it]
 98%|█████████▊| 421/430 [9:32:40<12:05, 80.59s/it]
 98%|█████████▊| 422/430 [9:34:01<10:44, 80.56s/it]
 98%|█████████▊| 423/430 [9:35:21<09:23, 80.53s/it]
 99%|█████████▊| 424/430 [9:36:42<08:03, 80.54s/it]
 99%|█████████▉| 425/430 [9:38:02<06:42, 80.52s/it]
                                                   
+
 99%|█████████▉| 425/430 [9:38:02<06:42, 80.52s/it]
 99%|█████████▉| 426/430 [9:39:23<05:22, 80.52s/it]
 99%|█████████▉| 427/430 [9:40:43<04:01, 80.52s/it]
                                                   
+
+
+Training completed. Do not forget to share your model on huggingface.co/models =)
+
+
+Time: 35085.85Time: 35085.80
+Time: 35085.78
+Samples/second: 33.41Time: 35085.88
+Samples/second: 33.41
+
+Samples/second: 33.41
+Samples/second: 33.41
+
+
                                                   
+Time: 35085.79
+Time: 35085.76Samples/second: 33.41
+
+Samples/second: 33.41
+
+GPU memory occupied: 43825 MB.
+GPU memory occupied: 43825 MB.
+GPU memory occupied: 43825 MB.
+GPU memory occupied: 43825 MB.
+GPU memory occupied: 43825 MB.
+
+Time: 35085.49
+Samples/second: 33.41
+GPU memory occupied: 43825 MB.
+Configuration saved in experiments/2023-02-21-b0010c97cb1f06debca911602ea05b6ff85a8270fb9487d27b3d52eb4eb29e9e/final_checkpoint/config.json
+Model weights saved in experiments/2023-02-21-b0010c97cb1f06debca911602ea05b6ff85a8270fb9487d27b3d52eb4eb29e9e/final_checkpoint/pytorch_model.bin
+tokenizer config file saved in experiments/2023-02-21-b0010c97cb1f06debca911602ea05b6ff85a8270fb9487d27b3d52eb4eb29e9e/final_checkpoint/tokenizer/tokenizer_config.json
+Special tokens file saved in experiments/2023-02-21-b0010c97cb1f06debca911602ea05b6ff85a8270fb9487d27b3d52eb4eb29e9e/final_checkpoint/tokenizer/special_tokens_map.json
+Saving model checkpoint to experiments/2023-02-21-b0010c97cb1f06debca911602ea05b6ff85a8270fb9487d27b3d52eb4eb29e9e/trainer_final_checkpoint
+Configuration saved in experiments/2023-02-21-b0010c97cb1f06debca911602ea05b6ff85a8270fb9487d27b3d52eb4eb29e9e/trainer_final_checkpoint/config.json
+Model weights saved in experiments/2023-02-21-b0010c97cb1f06debca911602ea05b6ff85a8270fb9487d27b3d52eb4eb29e9e/trainer_final_checkpoint/pytorch_model.bin
+tokenizer config file saved in experiments/2023-02-21-b0010c97cb1f06debca911602ea05b6ff85a8270fb9487d27b3d52eb4eb29e9e/trainer_final_checkpoint/tokenizer_config.json
+Special tokens file saved in experiments/2023-02-21-b0010c97cb1f06debca911602ea05b6ff85a8270fb9487d27b3d52eb4eb29e9e/trainer_final_checkpoint/special_tokens_map.json
+Traceback (most recent call last):
+  File "tune_gpt.py", line 227, in <module>
+    trainer.save_state(trainer_save_dir)
+TypeError: save_state() takes 1 positional argument but 2 were given
+[2023-02-21 02:51:15,357] [INFO] [launch.py:350:main] Process 31459 exits successfully.
+[2023-02-21 02:51:15,358] [INFO] [launch.py:350:main] Process 31463 exits successfully.
+[2023-02-21 02:51:16,360] [INFO] [launch.py:350:main] Process 31486 exits successfully.
+[2023-02-21 02:51:16,360] [INFO] [launch.py:350:main] Process 31471 exits successfully.
+[2023-02-21 02:51:16,360] [INFO] [launch.py:350:main] Process 31478 exits successfully.
+[2023-02-21 02:51:16,361] [INFO] [launch.py:350:main] Process 31490 exits successfully.
+[2023-02-21 02:51:17,362] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 31458
+[2023-02-21 02:51:17,363] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 31459
+[2023-02-21 02:51:17,363] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 31463
+[2023-02-21 02:51:17,363] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 31471
+[2023-02-21 02:51:17,363] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 31478
+[2023-02-21 02:51:17,363] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 31486
+[2023-02-21 02:51:17,363] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 31490
+[2023-02-21 02:51:17,364] [ERROR] [launch.py:324:sigkill_handler] ['/opt/conda/bin/python3', '-u', 'tune_gpt.py', '--local_rank=6', '--deepspeed', 'deepspeed.json', '--upload-experiment'] exits with return code = 1
+/opt/conda/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
+  from pandas import MultiIndex, Int64Index