nohup: ignoring input
[2023-02-21 19:51:35,707] [WARNING] [runner.py:186:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2023-02-21 19:51:35,765] [INFO] [runner.py:548:main] cmd = /opt/conda/bin/python3 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMiwgMywgNCwgNSwgNiwgN119 --master_addr=127.0.0.1 --master_port=29500 --enable_each_rank_log=None tune_gpt.py --deepspeed deepspeed.json --upload-experiment
/opt/conda/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  from pandas import MultiIndex, Int64Index
[2023-02-21 19:51:38,485] [INFO] [launch.py:135:main] 0 NCCL_VERSION=2.11.4
[2023-02-21 19:51:38,485] [INFO] [launch.py:142:main] WORLD INFO DICT: {'localhost': [0, 1, 2, 3, 4, 5, 6, 7]}
[2023-02-21 19:51:38,485] [INFO] [launch.py:148:main] nnodes=1, num_local_procs=8, node_rank=0
[2023-02-21 19:51:38,485] [INFO] [launch.py:161:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2, 3, 4, 5, 6, 7]})
[2023-02-21 19:51:38,485] [INFO] [launch.py:162:main] dist_world_size=8
[2023-02-21 19:51:38,485] [INFO] [launch.py:164:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
/opt/conda/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  from pandas import MultiIndex, Int64Index
/opt/conda/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  from pandas import MultiIndex, Int64Index
/opt/conda/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  from pandas import MultiIndex, Int64Index
/opt/conda/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  from pandas import MultiIndex, Int64Index
/opt/conda/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  from pandas import MultiIndex, Int64Index
/opt/conda/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  from pandas import MultiIndex, Int64Index
/opt/conda/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  from pandas import MultiIndex, Int64Index
/opt/conda/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  from pandas import MultiIndex, Int64Index
No config specified, defaulting to: apps/all
Found cached dataset apps (/home/.cache/huggingface/datasets/codeparrot___apps/all/0.0.0/04ac807715d07d6e5cc580f59cdc8213cd7dc4529d0bb819cca72c9f8e8c1aa5)
No config specified, defaulting to: apps/all
Found cached dataset apps (/home/.cache/huggingface/datasets/codeparrot___apps/all/0.0.0/04ac807715d07d6e5cc580f59cdc8213cd7dc4529d0bb819cca72c9f8e8c1aa5)
No config specified, defaulting to: apps/all
Found cached dataset apps (/home/.cache/huggingface/datasets/codeparrot___apps/all/0.0.0/04ac807715d07d6e5cc580f59cdc8213cd7dc4529d0bb819cca72c9f8e8c1aa5)
No config specified, defaulting to: apps/all
No config specified, defaulting to: apps/all
Found cached dataset apps (/home/.cache/huggingface/datasets/codeparrot___apps/all/0.0.0/04ac807715d07d6e5cc580f59cdc8213cd7dc4529d0bb819cca72c9f8e8c1aa5)
No config specified, defaulting to: apps/all
Found cached dataset apps (/home/.cache/huggingface/datasets/codeparrot___apps/all/0.0.0/04ac807715d07d6e5cc580f59cdc8213cd7dc4529d0bb819cca72c9f8e8c1aa5)
Found cached dataset apps (/home/.cache/huggingface/datasets/codeparrot___apps/all/0.0.0/04ac807715d07d6e5cc580f59cdc8213cd7dc4529d0bb819cca72c9f8e8c1aa5)
No config specified, defaulting to: apps/all
Found cached dataset apps (/home/.cache/huggingface/datasets/codeparrot___apps/all/0.0.0/04ac807715d07d6e5cc580f59cdc8213cd7dc4529d0bb819cca72c9f8e8c1aa5)
No config specified, defaulting to: apps/all
Found cached dataset apps (/home/.cache/huggingface/datasets/codeparrot___apps/all/0.0.0/04ac807715d07d6e5cc580f59cdc8213cd7dc4529d0bb819cca72c9f8e8c1aa5)
Max length: 2048
PyTorch: setting up devices
[2023-02-21 19:51:59,246] [INFO] [comm.py:657:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
Max length: 2048
PyTorch: setting up devices
Max length: 2048
PyTorch: setting up devices
Max length: 2048
PyTorch: setting up devices
Max length: 2048Max length: 2048

PyTorch: setting up devices
PyTorch: setting up devices
Max length: 2048
PyTorch: setting up devices
Max length: 2048
PyTorch: setting up devices
The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
GPU memory occupied: 7801 MB.
GPU memory occupied: 7801 MB.
[2023-02-21 19:51:59,917] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed info: version=0.8.1, git-hash=unknown, git-branch=unknown
GPU memory occupied: 7801 MB.
GPU memory occupied: 7801 MB.
GPU memory occupied: 7801 MB.
GPU memory occupied: 7801 MB.
GPU memory occupied: 7801 MB.
GPU memory occupied: 7801 MB.
[2023-02-21 19:52:02,386] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
Installed CUDA version 11.6 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.6 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.6 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.6 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.6 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.6 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.6 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.6 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/.cache/torch_extensions/py38_cu117/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
ninja: no work to do.
Loading extension module cpu_adam...
Time to load cpu_adam op: 3.051539182662964 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.8587288856506348 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.8896634578704834 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.8358540534973145 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.7415549755096436 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.8722808361053467 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.8915891647338867 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 3.0029048919677734 seconds
Adam Optimizer #0 is created with AVX512 arithmetic capability.
Config: alpha=0.000100, betas=(0.900000, 0.999000), weight_decay=0.100000, adam_w=1
[2023-02-21 19:52:09,340] [INFO] [logging.py:75:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer
[2023-02-21 19:52:09,347] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
[2023-02-21 19:52:09,347] [INFO] [utils.py:53:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
[2023-02-21 19:52:09,347] [INFO] [logging.py:75:log_dist] [Rank 0] Creating torch.float32 ZeRO stage 2 optimizer
[2023-02-21 19:52:09,348] [INFO] [stage_1_and_2.py:144:__init__] Reduce bucket size 500000000
[2023-02-21 19:52:09,348] [INFO] [stage_1_and_2.py:145:__init__] Allgather bucket size 500000000
[2023-02-21 19:52:09,348] [INFO] [stage_1_and_2.py:146:__init__] CPU Offload: True
[2023-02-21 19:52:09,348] [INFO] [stage_1_and_2.py:147:__init__] Round robin gradient partitioning: False
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Emitting ninja build file /home/.cache/torch_extensions/py38_cu117/utils/build.ninja...
Building extension module utils...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module utils...
Time to load utils op: 0.5143241882324219 seconds
Loading extension module utils...
Time to load utils op: 0.4041626453399658 seconds
Loading extension module utils...
Time to load utils op: 0.40406346321105957 seconds
Loading extension module utils...
Time to load utils op: 0.40425753593444824 seconds
Loading extension module utils...
Time to load utils op: 0.40483736991882324 seconds
Loading extension module utils...
Loading extension module utils...
Time to load utils op: 0.4042036533355713 seconds
Time to load utils op: 0.4030454158782959 seconds
Loading extension module utils...
Time to load utils op: 0.4040055274963379 seconds
Rank: 5 partition count [8] and sizes[(15649824, False)] 
Rank: 0 partition count [8] and sizes[(15649824, False)] 
Rank: 2 partition count [8] and sizes[(15649824, False)] 
Rank: 3 partition count [8] and sizes[(15649824, False)] 
Rank: 4 partition count [8] and sizes[(15649824, False)] 
Rank: 6 partition count [8] and sizes[(15649824, False)] 
Rank: 1 partition count [8] and sizes[(15649824, False)] 
Rank: 7 partition count [8] and sizes[(15649824, False)] 
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.0007538795471191406 seconds
[2023-02-21 19:52:14,395] [INFO] [utils.py:825:see_memory_usage] Before initializing optimizer states
[2023-02-21 19:52:14,405] [INFO] [utils.py:826:see_memory_usage] MA 0.66 GB         Max_MA 0.66 GB         CA 0.85 GB         Max_CA 1 GB 
[2023-02-21 19:52:14,405] [INFO] [utils.py:834:see_memory_usage] CPU Virtual Memory:  used = 42.5 GB, percent = 16.9%
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.005951404571533203 seconds
You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.0007529258728027344 seconds
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.020640850067138672 seconds
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.0007100105285644531 seconds
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.0004687309265136719 seconds
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.0029997825622558594 seconds
You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
[2023-02-21 19:52:14,635] [INFO] [utils.py:825:see_memory_usage] After initializing optimizer states
[2023-02-21 19:52:14,635] [INFO] [utils.py:826:see_memory_usage] MA 0.66 GB         Max_MA 0.66 GB         CA 0.85 GB         Max_CA 1 GB 
[2023-02-21 19:52:14,636] [INFO] [utils.py:834:see_memory_usage] CPU Virtual Memory:  used = 42.46 GB, percent = 16.9%
[2023-02-21 19:52:14,636] [INFO] [stage_1_and_2.py:527:__init__] optimizer state initialized
[2023-02-21 19:52:14,719] [INFO] [utils.py:825:see_memory_usage] After initializing ZeRO optimizer
[2023-02-21 19:52:14,720] [INFO] [utils.py:826:see_memory_usage] MA 0.66 GB         Max_MA 0.66 GB         CA 0.85 GB         Max_CA 1 GB 
[2023-02-21 19:52:14,720] [INFO] [utils.py:834:see_memory_usage] CPU Virtual Memory:  used = 42.46 GB, percent = 16.9%
[2023-02-21 19:52:14,721] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw
[2023-02-21 19:52:14,721] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed using configured LR scheduler = WarmupLR
[2023-02-21 19:52:14,721] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed LR Scheduler = <deepspeed.runtime.lr_schedules.WarmupLR object at 0x7f3a85b76cd0>
[2023-02-21 19:52:14,721] [INFO] [logging.py:75:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0001], mom=[[0.9, 0.999]]
[2023-02-21 19:52:14,722] [INFO] [config.py:1009:print] DeepSpeedEngine configuration:
[2023-02-21 19:52:14,722] [INFO] [config.py:1013:print]   activation_checkpointing_config  {
    "partition_activations": false, 
    "contiguous_memory_optimization": false, 
    "cpu_checkpointing": false, 
    "number_checkpoints": null, 
    "synchronize_checkpoint_boundary": false, 
    "profile": false
}
[2023-02-21 19:52:14,722] [INFO] [config.py:1013:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}
[2023-02-21 19:52:14,722] [INFO] [config.py:1013:print]   amp_enabled .................. False
[2023-02-21 19:52:14,722] [INFO] [config.py:1013:print]   amp_params ................... False
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   autotuning_config ............ {
    "enabled": false, 
    "start_step": null, 
    "end_step": null, 
    "metric_path": null, 
    "arg_mappings": null, 
    "metric": "throughput", 
    "model_info": null, 
    "results_dir": "autotuning_results", 
    "exps_dir": "autotuning_exps", 
    "overwrite": true, 
    "fast": true, 
    "start_profile_step": 3, 
    "end_profile_step": 5, 
    "tuner_type": "gridsearch", 
    "tuner_early_stopping": 5, 
    "tuner_num_trials": 50, 
    "model_info_path": null, 
    "mp_size": 1, 
    "max_train_batch_size": null, 
    "min_train_batch_size": 1, 
    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
    "min_train_micro_batch_size_per_gpu": 1, 
    "num_tuning_micro_batch_sizes": 3
}
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   bfloat16_enabled ............. False
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   checkpoint_parallel_write_pipeline  False
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   checkpoint_tag_validation_enabled  True
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   checkpoint_tag_validation_fail  False
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7f3a85b76250>
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   communication_data_type ...... None
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   curriculum_enabled_legacy .... False
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   curriculum_params_legacy ..... False
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   data_efficiency_enabled ...... False
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   dataloader_drop_last ......... False
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   disable_allgather ............ False
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   dump_state ................... False
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   dynamic_loss_scale_args ...... None
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   eigenvalue_enabled ........... False
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   eigenvalue_gas_boundary_resolution  1
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   eigenvalue_layer_num ......... 0
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   eigenvalue_max_iter .......... 100
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   eigenvalue_stability ......... 1e-06
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   eigenvalue_tol ............... 0.01
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   eigenvalue_verbose ........... False
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   elasticity_enabled ........... False
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   flops_profiler_config ........ {
    "enabled": false, 
    "profile_step": 1, 
    "module_depth": -1, 
    "top_modules": 1, 
    "detailed": true, 
    "output_file": null
}
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   fp16_auto_cast ............... None
[2023-02-21 19:52:14,723] [INFO] [config.py:1013:print]   fp16_enabled ................. False
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   fp16_master_weights_and_gradients  False
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   global_rank .................. 0
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   grad_accum_dtype ............. None
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   gradient_accumulation_steps .. 4
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   gradient_clipping ............ 1.0
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   gradient_predivide_factor .... 1.0
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   initial_dynamic_scale ........ 65536
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   load_universal_checkpoint .... False
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   loss_scale ................... 0
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   memory_breakdown ............. False
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=True, output_path='logs/', job_name='train_neo') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=True
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   nebula_config ................ {
    "enabled": false, 
    "persistent_storage_path": null, 
    "persistent_time_interval": 100, 
    "num_of_version_in_retention": 2, 
    "enable_nebula_load": true, 
    "load_path": null
}
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   optimizer_legacy_fusion ...... False
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   optimizer_name ............... adamw
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   optimizer_params ............. {'lr': 0.0001, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.1}
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   pld_enabled .................. False
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   pld_params ................... False
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   prescale_gradients ........... False
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   scheduler_name ............... WarmupLR
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   scheduler_params ............. {'warmup_min_lr': 1e-09, 'warmup_max_lr': 0.0001, 'warmup_num_steps': 1000}
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   sparse_attention ............. None
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   sparse_gradients_enabled ..... False
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   steps_per_print .............. 2000
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   train_batch_size ............. 192
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   train_micro_batch_size_per_gpu  6
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   use_node_local_storage ....... False
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   wall_clock_breakdown ......... False
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   world_size ................... 8
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   zero_allow_untested_optimizer  True
[2023-02-21 19:52:14,724] [INFO] [config.py:1013:print]   zero_config .................. stage=2 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500000000 allgather_partitions=True allgather_bucket_size=500000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='cpu', nvme_path=None, buffer_count=4, pin_memory=True, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False) sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False
[2023-02-21 19:52:14,725] [INFO] [config.py:1013:print]   zero_enabled ................. True
[2023-02-21 19:52:14,725] [INFO] [config.py:1013:print]   zero_optimization_stage ...... 2
[2023-02-21 19:52:14,725] [INFO] [config.py:998:print_user_config]   json = {
    "optimizer": {
        "type": "AdamW", 
        "params": {
            "lr": 0.0001, 
            "betas": [0.9, 0.999], 
            "eps": 1e-08, 
            "weight_decay": 0.1
        }
    }, 
    "scheduler": {
        "type": "WarmupLR", 
        "params": {
            "warmup_min_lr": 1e-09, 
            "warmup_max_lr": 0.0001, 
            "warmup_num_steps": 1000
        }
    }, 
    "zero_optimization": {
        "stage": 2, 
        "offload_optimizer": {
            "device": "cpu", 
            "pin_memory": true
        }, 
        "allgather_partitions": true, 
        "allgather_bucket_size": 5.000000e+08, 
        "overlap_comm": true, 
        "reduce_scatter": true, 
        "reduce_bucket_size": 5.000000e+08, 
        "contiguous_gradients": true
    }, 
    "tensorboard": {
        "enabled": true, 
        "output_path": "logs/", 
        "job_name": "train_neo"
    }, 
    "zero_allow_untested_optimizer": true, 
    "gradient_accumulation_steps": 4, 
    "gradient_clipping": 1.0, 
    "steps_per_print": 2.000000e+03, 
    "train_batch_size": 192, 
    "train_micro_batch_size_per_gpu": 6, 
    "wall_clock_breakdown": false
}
Using /home/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.00039505958557128906 seconds
***** Running training *****
  Num examples = 117232
  Num Epochs = 10
  Instantaneous batch size per device = 6
  Total train batch size (w. parallel, distributed & accumulation) = 192
  Gradient Accumulation steps = 4
  Total optimization steps = 6100
  Number of trainable parameters = 125198592

  0%|          | 0/6100 [00:00<?, ?it/s]You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.

  0%|          | 1/6100 [00:06<10:10:47,  6.01s/it]
                                                   
{'loss': 6.6728, 'learning_rate': 1e-09, 'epoch': 0.0}

  0%|          | 1/6100 [00:06<10:10:47,  6.01s/it]
  0%|          | 2/6100 [00:11<9:28:59,  5.60s/it] 
  0%|          | 3/6100 [00:16<9:17:57,  5.49s/it]
  0%|          | 4/6100 [00:21<9:10:21,  5.42s/it]
  0%|          | 5/6100 [00:27<9:05:42,  5.37s/it]
                                                  
{'loss': 2.1107, 'learning_rate': 2.3299767154532515e-05, 'epoch': 0.01}

  0%|          | 5/6100 [00:27<9:05:42,  5.37s/it]
  0%|          | 6/6100 [00:32<9:12:11,  5.44s/it]
  0%|          | 7/6100 [00:38<9:08:20,  5.40s/it]
  0%|          | 8/6100 [00:43<9:05:20,  5.37s/it]
  0%|          | 9/6100 [00:48<9:04:00,  5.36s/it]
  0%|          | 10/6100 [00:54<9:03:22,  5.35s/it]
                                                   
{'loss': 0.6282, 'learning_rate': 3.333400000000001e-05, 'epoch': 0.02}

  0%|          | 10/6100 [00:54<9:03:22,  5.35s/it]
  0%|          | 11/6100 [00:59<9:02:43,  5.35s/it]
  0%|          | 12/6100 [01:04<9:01:54,  5.34s/it]
  0%|          | 13/6100 [01:10<9:01:28,  5.34s/it]
  0%|          | 14/6100 [01:15<9:01:25,  5.34s/it]
  0%|          | 15/6100 [01:20<9:01:01,  5.33s/it]
                                                   
{'loss': 0.5024, 'learning_rate': 3.920364993810303e-05, 'epoch': 0.02}

  0%|          | 15/6100 [01:20<9:01:01,  5.33s/it]
  0%|          | 16/6100 [01:26<9:01:25,  5.34s/it]
  0%|          | 17/6100 [01:31<9:01:42,  5.34s/it]
  0%|          | 18/6100 [01:36<9:01:44,  5.34s/it]
  0%|          | 19/6100 [01:42<9:01:43,  5.35s/it]
  0%|          | 20/6100 [01:47<9:02:31,  5.35s/it]
                                                   
{'loss': 0.4809, 'learning_rate': 4.336823284546749e-05, 'epoch': 0.03}

  0%|          | 20/6100 [01:47<9:02:31,  5.35s/it]
  0%|          | 21/6100 [01:52<9:02:23,  5.35s/it]
  0%|          | 22/6100 [01:58<9:02:34,  5.36s/it]
  0%|          | 23/6100 [02:03<9:02:24,  5.36s/it]
  0%|          | 24/6100 [02:08<9:02:04,  5.35s/it]
  0%|          | 25/6100 [02:14<9:01:50,  5.35s/it]
                                                   
{'loss': 0.4818, 'learning_rate': 4.6598534309065034e-05, 'epoch': 0.04}

  0%|          | 25/6100 [02:14<9:01:50,  5.35s/it]
  0%|          | 26/6100 [02:19<9:01:35,  5.35s/it]
  0%|          | 27/6100 [02:25<9:01:40,  5.35s/it]
  0%|          | 28/6100 [02:30<9:01:51,  5.35s/it]
  0%|          | 29/6100 [02:35<9:02:26,  5.36s/it]
  0%|          | 30/6100 [02:41<9:02:30,  5.36s/it]
                                                   
{'loss': 0.4752, 'learning_rate': 4.923788278357051e-05, 'epoch': 0.05}

  0%|          | 30/6100 [02:41<9:02:30,  5.36s/it]
  1%|          | 31/6100 [02:46<9:02:55,  5.37s/it]
  1%|          | 32/6100 [02:51<9:02:59,  5.37s/it]
  1%|          | 33/6100 [02:57<9:02:56,  5.37s/it]
  1%|          | 34/6100 [03:02<9:03:07,  5.37s/it]
  1%|          | 35/6100 [03:08<9:03:12,  5.37s/it]
                                                   
{'loss': 0.4508, 'learning_rate': 5.1469420122327746e-05, 'epoch': 0.06}

  1%|          | 35/6100 [03:08<9:03:12,  5.37s/it]
  1%|          | 36/6100 [03:13<9:03:21,  5.38s/it]
  1%|          | 37/6100 [03:18<9:03:34,  5.38s/it]
  1%|          | 38/6100 [03:24<9:03:43,  5.38s/it]
  1%|          | 39/6100 [03:29<9:03:23,  5.38s/it]
  1%|          | 40/6100 [03:34<9:03:34,  5.38s/it]
                                                   
{'loss': 0.4376, 'learning_rate': 5.340246569093497e-05, 'epoch': 0.07}

  1%|          | 40/6100 [03:34<9:03:34,  5.38s/it]
  1%|          | 41/6100 [03:40<9:03:22,  5.38s/it]
  1%|          | 42/6100 [03:45<9:03:16,  5.38s/it]
  1%|          | 43/6100 [03:51<9:03:40,  5.39s/it]
  1%|          | 44/6100 [03:56<9:03:59,  5.39s/it]
  1%|          | 45/6100 [04:01<9:03:38,  5.39s/it]
                                                   
{'loss': 0.4475, 'learning_rate': 5.5107532721673536e-05, 'epoch': 0.07}

  1%|          | 45/6100 [04:01<9:03:38,  5.39s/it]
  1%|          | 46/6100 [04:07<9:03:54,  5.39s/it]
  1%|          | 47/6100 [04:12<9:03:57,  5.39s/it]
  1%|          | 48/6100 [04:18<9:03:45,  5.39s/it]
  1%|          | 49/6100 [04:23<9:04:01,  5.39s/it]
  1%|          | 50/6100 [04:28<9:04:13,  5.40s/it]
                                                   
{'loss': 0.4304, 'learning_rate': 5.6632767154532515e-05, 'epoch': 0.08}

  1%|          | 50/6100 [04:28<9:04:13,  5.40s/it]
  1%|          | 51/6100 [04:34<9:04:26,  5.40s/it]
  1%|          | 52/6100 [04:39<9:04:05,  5.40s/it]
  1%|          | 53/6100 [04:45<9:04:01,  5.40s/it]
  1%|          | 54/6100 [04:50<9:04:09,  5.40s/it]
  1%|          | 55/6100 [04:55<9:04:08,  5.40s/it]
                                                   
{'loss': 0.423, 'learning_rate': 5.801250952891164e-05, 'epoch': 0.09}

  1%|          | 55/6100 [04:55<9:04:08,  5.40s/it]
  1%|          | 56/6100 [05:01<9:04:37,  5.41s/it]
  1%|          | 57/6100 [05:06<9:04:59,  5.41s/it]
  1%|          | 58/6100 [05:12<9:04:38,  5.41s/it]
  1%|          | 59/6100 [05:17<9:04:14,  5.41s/it]
  1%|          | 60/6100 [05:22<9:04:09,  5.41s/it]
                                                   
{'loss': 0.4178, 'learning_rate': 5.9272115629038e-05, 'epoch': 0.1}

  1%|          | 60/6100 [05:22<9:04:09,  5.41s/it]
  1%|          | 61/6100 [05:28<9:04:21,  5.41s/it]
  1%|          | 62/6100 [05:33<9:03:52,  5.40s/it]
  1%|          | 63/6100 [05:39<9:03:55,  5.41s/it]
  1%|          | 64/6100 [05:44<9:03:41,  5.40s/it]
  1%|          | 65/6100 [05:49<9:03:38,  5.40s/it]
                                                   
{'loss': 0.4173, 'learning_rate': 6.043084091697631e-05, 'epoch': 0.11}

  1%|          | 65/6100 [05:49<9:03:38,  5.40s/it]
  1%|          | 66/6100 [05:55<9:04:12,  5.41s/it]
  1%|          | 67/6100 [06:00<9:04:56,  5.42s/it]
  1%|          | 68/6100 [06:06<9:05:12,  5.42s/it]
  1%|          | 69/6100 [06:11<9:04:43,  5.42s/it]
  1%|          | 70/6100 [06:17<9:04:07,  5.41s/it]
                                                   
{'loss': 0.3981, 'learning_rate': 6.150365296779523e-05, 'epoch': 0.11}

  1%|          | 70/6100 [06:17<9:04:07,  5.41s/it]
  1%|          | 71/6100 [06:22<9:04:10,  5.42s/it]
  1%|          | 72/6100 [06:27<9:03:59,  5.41s/it]
  1%|          | 73/6100 [06:33<9:03:14,  5.41s/it]
  1%|          | 74/6100 [06:38<9:02:54,  5.41s/it]
  1%|          | 75/6100 [06:44<9:03:12,  5.41s/it]
                                                   
{'loss': 0.3819, 'learning_rate': 6.250241709263554e-05, 'epoch': 0.12}

  1%|          | 75/6100 [06:44<9:03:12,  5.41s/it]
  1%|          | 76/6100 [06:49<9:03:07,  5.41s/it]
  1%|▏         | 77/6100 [06:54<9:02:52,  5.41s/it]
  1%|▏         | 78/6100 [07:00<9:03:14,  5.41s/it]
  1%|▏         | 79/6100 [07:05<9:02:29,  5.41s/it]
  1%|▏         | 80/6100 [07:11<9:02:25,  5.41s/it]
                                                   
{'loss': 0.4089, 'learning_rate': 6.343669853640246e-05, 'epoch': 0.13}

  1%|▏         | 80/6100 [07:11<9:02:25,  5.41s/it]
  1%|▏         | 81/6100 [07:16<9:02:42,  5.41s/it]
  1%|▏         | 82/6100 [07:21<9:02:26,  5.41s/it]
  1%|▏         | 83/6100 [07:27<9:02:10,  5.41s/it]
  1%|▏         | 84/6100 [07:32<9:02:02,  5.41s/it]
  1%|▏         | 85/6100 [07:38<9:02:41,  5.41s/it]
                                                   
{'loss': 0.402, 'learning_rate': 6.431432105083453e-05, 'epoch': 0.14}

  1%|▏         | 85/6100 [07:38<9:02:41,  5.41s/it]
  1%|▏         | 86/6100 [07:44<9:17:25,  5.56s/it]
  1%|▏         | 87/6100 [07:49<9:12:36,  5.51s/it]
  1%|▏         | 88/6100 [07:54<9:09:20,  5.48s/it]
  1%|▏         | 89/6100 [08:00<9:07:22,  5.46s/it]
  1%|▏         | 90/6100 [08:05<9:05:40,  5.45s/it]
                                                   
{'loss': 0.3907, 'learning_rate': 6.514176556714102e-05, 'epoch': 0.15}

  1%|▏         | 90/6100 [08:05<9:05:40,  5.45s/it]
  1%|▏         | 91/6100 [08:11<9:04:43,  5.44s/it]
  2%|▏         | 92/6100 [08:16<9:03:27,  5.43s/it]
  2%|▏         | 93/6100 [08:21<9:02:36,  5.42s/it]
  2%|▏         | 94/6100 [08:27<9:02:17,  5.42s/it]
  2%|▏         | 95/6100 [08:32<9:02:04,  5.42s/it]
                                                   
{'loss': 0.3979, 'learning_rate': 6.592446093509317e-05, 'epoch': 0.16}

  2%|▏         | 95/6100 [08:32<9:02:04,  5.42s/it]
  2%|▏         | 96/6100 [08:38<9:01:56,  5.42s/it]
  2%|▏         | 97/6100 [08:43<9:01:31,  5.41s/it]
  2%|▏         | 98/6100 [08:49<9:01:23,  5.41s/it]
  2%|▏         | 99/6100 [08:54<9:01:03,  5.41s/it]
  2%|▏         | 100/6100 [08:59<9:01:16,  5.41s/it]
                                                    
{'loss': 0.403, 'learning_rate': 6.666700000000002e-05, 'epoch': 0.16}

  2%|▏         | 100/6100 [08:59<9:01:16,  5.41s/it]
  2%|▏         | 101/6100 [09:05<9:01:28,  5.42s/it]
  2%|▏         | 102/6100 [09:10<9:01:15,  5.41s/it]
  2%|▏         | 103/6100 [09:16<9:01:02,  5.41s/it]
  2%|▏         | 104/6100 [09:21<9:00:53,  5.41s/it]
  2%|▏         | 105/6100 [09:26<9:00:16,  5.41s/it]
                                                    
{'loss': 0.3851, 'learning_rate': 6.737330290589825e-05, 'epoch': 0.17}

  2%|▏         | 105/6100 [09:26<9:00:16,  5.41s/it]
  2%|▏         | 106/6100 [09:32<9:00:08,  5.41s/it]
  2%|▏         | 107/6100 [09:37<9:00:34,  5.41s/it]
  2%|▏         | 108/6100 [09:43<9:00:51,  5.42s/it]
  2%|▏         | 109/6100 [09:48<9:00:17,  5.41s/it]
  2%|▏         | 110/6100 [09:53<9:00:22,  5.41s/it]
                                                    
{'loss': 0.3691, 'learning_rate': 6.804674237437913e-05, 'epoch': 0.18}

  2%|▏         | 110/6100 [09:53<9:00:22,  5.41s/it]
  2%|▏         | 111/6100 [09:59<9:00:34,  5.42s/it]
  2%|▏         | 112/6100 [10:04<9:00:27,  5.42s/it]
  2%|▏         | 113/6100 [10:10<8:59:59,  5.41s/it]
  2%|▏         | 114/6100 [10:15<8:59:42,  5.41s/it]
  2%|▏         | 115/6100 [10:21<9:00:03,  5.41s/it]
                                                    
{'loss': 0.3808, 'learning_rate': 6.869024111250695e-05, 'epoch': 0.19}

  2%|▏         | 115/6100 [10:21<9:00:03,  5.41s/it]
  2%|▏         | 116/6100 [10:26<9:00:04,  5.42s/it]
  2%|▏         | 117/6100 [10:31<9:00:13,  5.42s/it]
  2%|▏         | 118/6100 [10:37<8:59:41,  5.41s/it]
  2%|▏         | 119/6100 [10:42<8:59:18,  5.41s/it]
  2%|▏         | 120/6100 [10:48<8:58:58,  5.41s/it]
                                                    
{'loss': 0.3742, 'learning_rate': 6.930634847450548e-05, 'epoch': 0.2}

  2%|▏         | 120/6100 [10:48<8:58:58,  5.41s/it]
  2%|▏         | 121/6100 [10:53<9:01:07,  5.43s/it]
  2%|▏         | 122/6100 [10:58<9:00:07,  5.42s/it]
  2%|▏         | 123/6100 [11:04<8:59:29,  5.42s/it]
  2%|▏         | 124/6100 [11:09<8:59:05,  5.41s/it]
  2%|▏         | 125/6100 [11:15<8:58:46,  5.41s/it]
                                                    
{'loss': 0.3727, 'learning_rate': 6.989730146359756e-05, 'epoch': 0.2}

  2%|▏         | 125/6100 [11:15<8:58:46,  5.41s/it]
  2%|▏         | 126/6100 [11:20<8:58:35,  5.41s/it]
  2%|▏         | 127/6100 [11:25<8:58:06,  5.41s/it]
  2%|▏         | 128/6100 [11:31<8:58:15,  5.41s/it]
  2%|▏         | 129/6100 [11:36<8:58:22,  5.41s/it]
  2%|▏         | 130/6100 [11:42<8:58:17,  5.41s/it]
                                                    
{'loss': 0.3668, 'learning_rate': 7.04650737624438e-05, 'epoch': 0.21}

  2%|▏         | 130/6100 [11:42<8:58:17,  5.41s/it]
  2%|▏         | 131/6100 [11:47<8:58:31,  5.41s/it]
  2%|▏         | 132/6100 [11:53<8:58:05,  5.41s/it]
  2%|▏         | 133/6100 [11:58<8:57:48,  5.41s/it]
  2%|▏         | 134/6100 [12:03<8:57:30,  5.41s/it]
  2%|▏         | 135/6100 [12:09<8:57:37,  5.41s/it]
                                                    
{'loss': 0.3536, 'learning_rate': 7.101141550524405e-05, 'epoch': 0.22}

  2%|▏         | 135/6100 [12:09<8:57:37,  5.41s/it]
  2%|▏         | 136/6100 [12:14<8:57:48,  5.41s/it]
  2%|▏         | 137/6100 [12:20<8:57:40,  5.41s/it]
  2%|▏         | 138/6100 [12:25<8:57:08,  5.41s/it]
  2%|▏         | 139/6100 [12:30<8:57:17,  5.41s/it]
  2%|▏         | 140/6100 [12:36<8:57:07,  5.41s/it]
                                                    
{'loss': 0.3645, 'learning_rate': 7.153788581326271e-05, 'epoch': 0.23}

  2%|▏         | 140/6100 [12:36<8:57:07,  5.41s/it]
  2%|▏         | 141/6100 [12:41<8:56:59,  5.41s/it]
  2%|▏         | 142/6100 [12:47<8:56:39,  5.40s/it]
  2%|▏         | 143/6100 [12:52<8:56:49,  5.41s/it]
  2%|▏         | 144/6100 [12:57<8:57:03,  5.41s/it]
  2%|▏         | 145/6100 [13:03<8:58:51,  5.43s/it]
                                                    
{'loss': 0.3589, 'learning_rate': 7.204587961849842e-05, 'epoch': 0.24}

  2%|▏         | 145/6100 [13:03<8:58:51,  5.43s/it]
  2%|▏         | 146/6100 [13:08<8:58:07,  5.42s/it]
  2%|▏         | 147/6100 [13:14<8:57:25,  5.42s/it]
  2%|▏         | 148/6100 [13:19<8:56:51,  5.41s/it]
  2%|▏         | 149/6100 [13:25<8:56:36,  5.41s/it]
  2%|▏         | 150/6100 [13:30<8:56:58,  5.41s/it]
                                                    
{'loss': 0.3474, 'learning_rate': 7.253664993810302e-05, 'epoch': 0.25}

  2%|▏         | 150/6100 [13:30<8:56:58,  5.41s/it]Saving model checkpoint to ./results/checkpoint-150
Configuration saved in ./results/checkpoint-150/config.json
Model weights saved in ./results/checkpoint-150/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-150/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-150/special_tokens_map.json
[2023-02-21 20:05:46,142] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step150 is begin to save!
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
  warnings.warn(
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
  warnings.warn(
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
  warnings.warn(
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
  warnings.warn(
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
  warnings.warn(
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
  warnings.warn(
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
  warnings.warn(
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
  warnings.warn(
[2023-02-21 20:05:46,147] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-150/global_step150/mp_rank_00_model_states.pt
[2023-02-21 20:05:46,147] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-150/global_step150/mp_rank_00_model_states.pt...
[2023-02-21 20:05:46,973] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-150/global_step150/mp_rank_00_model_states.pt.
[2023-02-21 20:05:46,975] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-150/global_step150/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-21 20:05:47,150] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-150/global_step150/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-21 20:05:47,150] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-150/global_step150/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-21 20:05:47,150] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step150 is ready now!
Deleting older checkpoint [results/checkpoint-2700] due to args.save_total_limit

  2%|▏         | 151/6100 [13:38<10:01:28,  6.07s/it]
  2%|▏         | 152/6100 [13:43<9:41:17,  5.86s/it] 
  3%|▎         | 153/6100 [13:48<9:27:32,  5.73s/it]
  3%|▎         | 154/6100 [13:54<9:18:26,  5.64s/it]
  3%|▎         | 155/6100 [13:59<9:11:04,  5.56s/it]
                                                    
{'loss': 0.3508, 'learning_rate': 7.301132649511034e-05, 'epoch': 0.25}

  3%|▎         | 155/6100 [13:59<9:11:04,  5.56s/it]
  3%|▎         | 156/6100 [14:05<9:06:27,  5.52s/it]
  3%|▎         | 157/6100 [14:10<9:03:11,  5.48s/it]
  3%|▎         | 158/6100 [14:16<9:05:53,  5.51s/it]
  3%|▎         | 159/6100 [14:21<9:02:46,  5.48s/it]
  3%|▎         | 160/6100 [14:26<9:00:27,  5.46s/it]
                                                    
{'loss': 0.3452, 'learning_rate': 7.347093138186994e-05, 'epoch': 0.26}

  3%|▎         | 160/6100 [14:26<9:00:27,  5.46s/it]
  3%|▎         | 161/6100 [14:32<8:58:37,  5.44s/it]
  3%|▎         | 162/6100 [14:37<8:57:15,  5.43s/it]
  3%|▎         | 163/6100 [14:43<8:56:41,  5.42s/it]
  3%|▎         | 164/6100 [14:48<8:56:19,  5.42s/it]
  3%|▎         | 165/6100 [14:53<8:55:57,  5.42s/it]
                                                    
{'loss': 0.3527, 'learning_rate': 7.391639231248214e-05, 'epoch': 0.27}

  3%|▎         | 165/6100 [14:53<8:55:57,  5.42s/it]
  3%|▎         | 166/6100 [14:59<8:56:03,  5.42s/it]
  3%|▎         | 167/6100 [15:04<8:55:48,  5.42s/it]
  3%|▎         | 168/6100 [15:10<8:55:14,  5.41s/it]
  3%|▎         | 169/6100 [15:15<8:55:26,  5.42s/it]
  3%|▎         | 170/6100 [15:20<8:55:07,  5.41s/it]
                                                    
{'loss': 0.3468, 'learning_rate': 7.434855389630202e-05, 'epoch': 0.28}

  3%|▎         | 170/6100 [15:20<8:55:07,  5.41s/it]
  3%|▎         | 171/6100 [15:26<8:54:31,  5.41s/it]
  3%|▎         | 172/6100 [15:31<8:54:26,  5.41s/it]
  3%|▎         | 173/6100 [15:37<8:54:25,  5.41s/it]
  3%|▎         | 174/6100 [15:42<8:54:11,  5.41s/it]
  3%|▎         | 175/6100 [15:48<8:54:08,  5.41s/it]
                                                    
{'loss': 0.3358, 'learning_rate': 7.476818727686027e-05, 'epoch': 0.29}

  3%|▎         | 175/6100 [15:48<8:54:08,  5.41s/it]
  3%|▎         | 176/6100 [15:53<8:53:53,  5.41s/it]
  3%|▎         | 177/6100 [15:58<8:53:34,  5.41s/it]
  3%|▎         | 178/6100 [16:04<8:53:39,  5.41s/it]
  3%|▎         | 179/6100 [16:09<8:53:39,  5.41s/it]
  3%|▎         | 180/6100 [16:15<8:53:43,  5.41s/it]
                                                    
{'loss': 0.3437, 'learning_rate': 7.51759984126085e-05, 'epoch': 0.29}

  3%|▎         | 180/6100 [16:15<8:53:43,  5.41s/it]
  3%|▎         | 181/6100 [16:20<8:53:49,  5.41s/it]
  3%|▎         | 182/6100 [16:25<8:53:44,  5.41s/it]
  3%|▎         | 183/6100 [16:31<8:53:22,  5.41s/it]
  3%|▎         | 184/6100 [16:36<8:53:00,  5.41s/it]
  3%|▎         | 185/6100 [16:42<8:52:55,  5.41s/it]
                                                    
{'loss': 0.3263, 'learning_rate': 7.557263522285766e-05, 'epoch': 0.3}

  3%|▎         | 185/6100 [16:42<8:52:55,  5.41s/it]
  3%|▎         | 186/6100 [16:47<8:54:41,  5.42s/it]
  3%|▎         | 187/6100 [16:52<8:54:16,  5.42s/it]
  3%|▎         | 188/6100 [16:58<8:53:51,  5.42s/it]
  3%|▎         | 189/6100 [17:03<8:53:42,  5.42s/it]
  3%|▎         | 190/6100 [17:09<8:53:11,  5.41s/it]
                                                    
{'loss': 0.3288, 'learning_rate': 7.595869378056065e-05, 'epoch': 0.31}

  3%|▎         | 190/6100 [17:09<8:53:11,  5.41s/it]
  3%|▎         | 191/6100 [17:14<8:53:15,  5.41s/it]
  3%|▎         | 192/6100 [17:20<8:52:57,  5.41s/it]
  3%|▎         | 193/6100 [17:25<8:52:44,  5.41s/it]
  3%|▎         | 194/6100 [17:30<8:52:42,  5.41s/it]
  3%|▎         | 195/6100 [17:36<8:57:32,  5.46s/it]
                                                    
{'loss': 0.3333, 'learning_rate': 7.633472370054681e-05, 'epoch': 0.32}

  3%|▎         | 195/6100 [17:36<8:57:32,  5.46s/it]
  3%|▎         | 196/6100 [17:41<8:55:44,  5.44s/it]
  3%|▎         | 197/6100 [17:47<8:54:14,  5.43s/it]
  3%|▎         | 198/6100 [17:52<8:53:49,  5.43s/it]
  3%|▎         | 199/6100 [17:58<8:52:58,  5.42s/it]
  3%|▎         | 200/6100 [18:03<8:52:58,  5.42s/it]
                                                    
{'loss': 0.3206, 'learning_rate': 7.670123284546749e-05, 'epoch': 0.33}

  3%|▎         | 200/6100 [18:03<8:52:58,  5.42s/it]
  3%|▎         | 201/6100 [18:08<8:52:42,  5.42s/it]
  3%|▎         | 202/6100 [18:14<8:52:35,  5.42s/it]
  3%|▎         | 203/6100 [18:19<8:52:01,  5.41s/it]
  3%|▎         | 204/6100 [18:25<8:51:30,  5.41s/it]
  3%|▎         | 205/6100 [18:30<8:51:09,  5.41s/it]
                                                    
{'loss': 0.3263, 'learning_rate': 7.705869145057148e-05, 'epoch': 0.34}

  3%|▎         | 205/6100 [18:30<8:51:09,  5.41s/it]
  3%|▎         | 206/6100 [18:35<8:50:57,  5.41s/it]
  3%|▎         | 207/6100 [18:41<8:50:27,  5.40s/it]
  3%|▎         | 208/6100 [18:46<8:50:18,  5.40s/it]
  3%|▎         | 209/6100 [18:52<8:50:40,  5.40s/it]
  3%|▎         | 210/6100 [18:57<8:50:19,  5.40s/it]
                                                    
{'loss': 0.3222, 'learning_rate': 7.740753575136574e-05, 'epoch': 0.34}

  3%|▎         | 210/6100 [18:57<8:50:19,  5.40s/it]
  3%|▎         | 211/6100 [19:02<8:50:44,  5.41s/it]
  3%|▎         | 212/6100 [19:08<8:50:36,  5.41s/it]
  3%|▎         | 213/6100 [19:13<8:50:14,  5.40s/it]
  4%|▎         | 214/6100 [19:19<8:50:26,  5.41s/it]
  4%|▎         | 215/6100 [19:24<8:50:13,  5.41s/it]
                                                    
{'loss': 0.3196, 'learning_rate': 7.774817118436687e-05, 'epoch': 0.35}

  4%|▎         | 215/6100 [19:24<8:50:13,  5.41s/it]
  4%|▎         | 216/6100 [19:29<8:50:16,  5.41s/it]
  4%|▎         | 217/6100 [19:35<8:50:16,  5.41s/it]
  4%|▎         | 218/6100 [19:40<8:50:08,  5.41s/it]
  4%|▎         | 219/6100 [19:46<8:49:49,  5.41s/it]
  4%|▎         | 220/6100 [19:51<8:50:35,  5.41s/it]
                                                    
{'loss': 0.3333, 'learning_rate': 7.808097521984661e-05, 'epoch': 0.36}

  4%|▎         | 220/6100 [19:51<8:50:35,  5.41s/it]
  4%|▎         | 221/6100 [19:57<8:51:12,  5.42s/it]
  4%|▎         | 222/6100 [20:02<8:50:57,  5.42s/it]
  4%|▎         | 223/6100 [20:07<8:50:42,  5.42s/it]
  4%|▎         | 224/6100 [20:13<8:50:11,  5.41s/it]
  4%|▎         | 225/6100 [20:18<8:50:17,  5.42s/it]
                                                    
{'loss': 0.3144, 'learning_rate': 7.840629987620606e-05, 'epoch': 0.37}

  4%|▎         | 225/6100 [20:18<8:50:17,  5.42s/it]
  4%|▎         | 226/6100 [20:24<8:50:13,  5.42s/it]
  4%|▎         | 227/6100 [20:29<8:49:48,  5.41s/it]
  4%|▎         | 228/6100 [20:34<8:50:07,  5.42s/it]
  4%|▍         | 229/6100 [20:40<8:49:55,  5.42s/it]
  4%|▍         | 230/6100 [20:45<8:49:34,  5.41s/it]
                                                    
{'loss': 0.3054, 'learning_rate': 7.872447395797443e-05, 'epoch': 0.38}

  4%|▍         | 230/6100 [20:45<8:49:34,  5.41s/it]
  4%|▍         | 231/6100 [20:51<8:49:42,  5.42s/it]
  4%|▍         | 232/6100 [20:56<8:49:15,  5.41s/it]
  4%|▍         | 233/6100 [21:01<8:49:12,  5.41s/it]
  4%|▍         | 234/6100 [21:07<8:49:28,  5.42s/it]
  4%|▍         | 235/6100 [21:12<8:49:28,  5.42s/it]
                                                    
{'loss': 0.3082, 'learning_rate': 7.903580505310379e-05, 'epoch': 0.38}

  4%|▍         | 235/6100 [21:12<8:49:28,  5.42s/it]
  4%|▍         | 236/6100 [21:18<8:49:14,  5.42s/it]
  4%|▍         | 237/6100 [21:23<8:48:59,  5.41s/it]
  4%|▍         | 238/6100 [21:29<8:48:30,  5.41s/it]
  4%|▍         | 239/6100 [21:34<8:48:51,  5.41s/it]
  4%|▍         | 240/6100 [21:39<8:48:22,  5.41s/it]
                                                    
{'loss': 0.3209, 'learning_rate': 7.934058131997296e-05, 'epoch': 0.39}

  4%|▍         | 240/6100 [21:39<8:48:22,  5.41s/it]
  4%|▍         | 241/6100 [21:45<8:48:20,  5.41s/it]
  4%|▍         | 242/6100 [21:50<8:48:20,  5.41s/it]
  4%|▍         | 243/6100 [21:56<8:48:10,  5.41s/it]
  4%|▍         | 244/6100 [22:01<8:47:50,  5.41s/it]
  4%|▍         | 245/6100 [22:06<8:47:26,  5.41s/it]
                                                    
{'loss': 0.3009, 'learning_rate': 7.963907309012297e-05, 'epoch': 0.4}

  4%|▍         | 245/6100 [22:06<8:47:26,  5.41s/it]
  4%|▍         | 246/6100 [22:12<8:47:23,  5.41s/it]
  4%|▍         | 247/6100 [22:17<8:47:38,  5.41s/it]
  4%|▍         | 248/6100 [22:23<8:47:39,  5.41s/it]
  4%|▍         | 249/6100 [22:28<8:47:10,  5.41s/it]
  4%|▍         | 250/6100 [22:33<8:47:00,  5.41s/it]
                                                    
{'loss': 0.3059, 'learning_rate': 7.993153430906503e-05, 'epoch': 0.41}

  4%|▍         | 250/6100 [22:33<8:47:00,  5.41s/it]
  4%|▍         | 251/6100 [22:39<8:46:46,  5.40s/it]
  4%|▍         | 252/6100 [22:44<8:46:39,  5.40s/it]
  4%|▍         | 253/6100 [22:50<8:46:53,  5.41s/it]
  4%|▍         | 254/6100 [22:55<8:46:46,  5.41s/it]
  4%|▍         | 255/6100 [23:00<8:46:42,  5.41s/it]
                                                    
{'loss': 0.2991, 'learning_rate': 8.021820383440503e-05, 'epoch': 0.42}

  4%|▍         | 255/6100 [23:00<8:46:42,  5.41s/it]
  4%|▍         | 256/6100 [23:06<8:46:41,  5.41s/it]
  4%|▍         | 257/6100 [23:11<8:46:45,  5.41s/it]
  4%|▍         | 258/6100 [23:17<8:46:49,  5.41s/it]
  4%|▍         | 259/6100 [23:22<8:46:51,  5.41s/it]
  4%|▍         | 260/6100 [23:28<8:46:44,  5.41s/it]
                                                    
{'loss': 0.3081, 'learning_rate': 8.049930660791128e-05, 'epoch': 0.43}

  4%|▍         | 260/6100 [23:28<8:46:44,  5.41s/it]
  4%|▍         | 261/6100 [23:33<8:47:10,  5.42s/it]
  4%|▍         | 262/6100 [23:38<8:46:49,  5.41s/it]
  4%|▍         | 263/6100 [23:44<8:46:59,  5.42s/it]
  4%|▍         | 264/6100 [23:49<8:46:28,  5.41s/it]
  4%|▍         | 265/6100 [23:55<8:45:45,  5.41s/it]
                                                    
{'loss': 0.2892, 'learning_rate': 8.077505471593562e-05, 'epoch': 0.43}

  4%|▍         | 265/6100 [23:55<8:45:45,  5.41s/it]
  4%|▍         | 266/6100 [24:00<8:45:54,  5.41s/it]
  4%|▍         | 267/6100 [24:05<8:46:26,  5.42s/it]
  4%|▍         | 268/6100 [24:11<8:46:31,  5.42s/it]
  4%|▍         | 269/6100 [24:16<8:46:21,  5.42s/it]
  4%|▍         | 270/6100 [24:22<8:46:26,  5.42s/it]
                                                    
{'loss': 0.3073, 'learning_rate': 8.104564835071153e-05, 'epoch': 0.44}

  4%|▍         | 270/6100 [24:22<8:46:26,  5.42s/it]
  4%|▍         | 271/6100 [24:27<8:46:07,  5.42s/it]
  4%|▍         | 272/6100 [24:33<8:45:56,  5.41s/it]
  4%|▍         | 273/6100 [24:38<8:45:56,  5.42s/it]
  4%|▍         | 274/6100 [24:43<8:45:38,  5.41s/it]
  5%|▍         | 275/6100 [24:49<8:45:20,  5.41s/it]
                                                    
{'loss': 0.2909, 'learning_rate': 8.131127668344416e-05, 'epoch': 0.45}

  5%|▍         | 275/6100 [24:49<8:45:20,  5.41s/it]
  5%|▍         | 276/6100 [24:54<8:45:04,  5.41s/it]
  5%|▍         | 277/6100 [25:00<8:45:11,  5.41s/it]
  5%|▍         | 278/6100 [25:05<8:45:03,  5.41s/it]
  5%|▍         | 279/6100 [25:10<8:44:54,  5.41s/it]
  5%|▍         | 280/6100 [25:16<8:44:54,  5.41s/it]
                                                    
{'loss': 0.2801, 'learning_rate': 8.157211865873019e-05, 'epoch': 0.46}

  5%|▍         | 280/6100 [25:16<8:44:54,  5.41s/it]
  5%|▍         | 281/6100 [25:21<8:44:59,  5.41s/it]
  5%|▍         | 282/6100 [25:27<8:44:58,  5.41s/it]
  5%|▍         | 283/6100 [25:32<8:44:55,  5.41s/it]
  5%|▍         | 284/6100 [25:37<8:44:41,  5.41s/it]
  5%|▍         | 285/6100 [25:43<8:44:28,  5.41s/it]
                                                    
{'loss': 0.2923, 'learning_rate': 8.182834371866368e-05, 'epoch': 0.47}

  5%|▍         | 285/6100 [25:43<8:44:28,  5.41s/it]
  5%|▍         | 286/6100 [25:48<8:44:28,  5.41s/it]
  5%|▍         | 287/6100 [25:54<8:44:16,  5.41s/it]
  5%|▍         | 288/6100 [25:59<8:44:01,  5.41s/it]
  5%|▍         | 289/6100 [26:05<8:43:59,  5.41s/it]
  5%|▍         | 290/6100 [26:10<8:43:49,  5.41s/it]
                                                    
{'loss': 0.2938, 'learning_rate': 8.20801124639659e-05, 'epoch': 0.48}

  5%|▍         | 290/6100 [26:10<8:43:49,  5.41s/it]
  5%|▍         | 291/6100 [26:15<8:43:45,  5.41s/it]
  5%|▍         | 292/6100 [26:21<8:43:52,  5.41s/it]
  5%|▍         | 293/6100 [26:26<8:44:02,  5.41s/it]
  5%|▍         | 294/6100 [26:32<8:43:45,  5.41s/it]
  5%|▍         | 295/6100 [26:37<8:43:28,  5.41s/it]
                                                    
{'loss': 0.2916, 'learning_rate': 8.232757725860012e-05, 'epoch': 0.48}

  5%|▍         | 295/6100 [26:37<8:43:28,  5.41s/it]
  5%|▍         | 296/6100 [26:42<8:43:31,  5.41s/it]
  5%|▍         | 297/6100 [26:48<8:43:07,  5.41s/it]
  5%|▍         | 298/6100 [26:53<8:43:13,  5.41s/it]
  5%|▍         | 299/6100 [26:59<8:43:13,  5.41s/it]
  5%|▍         | 300/6100 [27:04<8:42:56,  5.41s/it]
                                                    
{'loss': 0.2867, 'learning_rate': 8.257088278357052e-05, 'epoch': 0.49}

  5%|▍         | 300/6100 [27:04<8:42:56,  5.41s/it]Saving model checkpoint to ./results/checkpoint-300
Configuration saved in ./results/checkpoint-300/config.json
Model weights saved in ./results/checkpoint-300/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-300/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-300/special_tokens_map.json
[2023-02-21 20:19:20,215] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step300 is begin to save!
[2023-02-21 20:19:20,218] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-300/global_step300/mp_rank_00_model_states.pt
[2023-02-21 20:19:20,218] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-300/global_step300/mp_rank_00_model_states.pt...
[2023-02-21 20:19:21,040] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-300/global_step300/mp_rank_00_model_states.pt.
[2023-02-21 20:19:21,042] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-300/global_step300/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-21 20:19:21,219] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-300/global_step300/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-21 20:19:21,219] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-300/global_step300/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-21 20:19:21,219] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step300 is ready now!
Deleting older checkpoint [results/checkpoint-150] due to args.save_total_limit

  5%|▍         | 301/6100 [27:12<9:44:41,  6.05s/it]
  5%|▍         | 302/6100 [27:17<9:25:55,  5.86s/it]
  5%|▍         | 303/6100 [27:22<9:13:09,  5.73s/it]
  5%|▍         | 304/6100 [27:28<9:20:39,  5.80s/it]
  5%|▌         | 305/6100 [27:34<9:09:37,  5.69s/it]
                                                    
{'loss': 0.294, 'learning_rate': 8.281016654494643e-05, 'epoch': 0.5}

  5%|▌         | 305/6100 [27:34<9:09:37,  5.69s/it]
  5%|▌         | 306/6100 [27:39<9:01:09,  5.60s/it]
  5%|▌         | 307/6100 [27:45<8:55:29,  5.55s/it]
  5%|▌         | 308/6100 [27:50<8:51:15,  5.50s/it]
  5%|▌         | 309/6100 [27:55<8:48:26,  5.48s/it]
  5%|▌         | 310/6100 [28:01<8:46:24,  5.46s/it]
                                                    
{'loss': 0.2848, 'learning_rate': 8.304555934057783e-05, 'epoch': 0.51}

  5%|▌         | 310/6100 [28:01<8:46:24,  5.46s/it]
  5%|▌         | 311/6100 [28:06<8:44:50,  5.44s/it]
  5%|▌         | 312/6100 [28:12<8:43:38,  5.43s/it]
  5%|▌         | 313/6100 [28:17<8:43:07,  5.42s/it]
  5%|▌         | 314/6100 [28:22<8:42:41,  5.42s/it]
  5%|▌         | 315/6100 [28:28<8:42:20,  5.42s/it]
                                                    
{'loss': 0.2729, 'learning_rate': 8.327718568946876e-05, 'epoch': 0.52}

  5%|▌         | 315/6100 [28:28<8:42:20,  5.42s/it]
  5%|▌         | 316/6100 [28:33<8:42:12,  5.42s/it]
  5%|▌         | 317/6100 [28:39<8:41:50,  5.41s/it]
  5%|▌         | 318/6100 [28:44<8:41:44,  5.41s/it]
  5%|▌         | 319/6100 [28:50<8:41:23,  5.41s/it]
  5%|▌         | 320/6100 [28:55<8:41:27,  5.41s/it]
                                                    
{'loss': 0.2803, 'learning_rate': 8.350516422733743e-05, 'epoch': 0.52}

  5%|▌         | 320/6100 [28:55<8:41:27,  5.41s/it]
  5%|▌         | 321/6100 [29:00<8:41:40,  5.42s/it]
  5%|▌         | 322/6100 [29:06<8:41:38,  5.42s/it]
  5%|▌         | 323/6100 [29:11<8:42:04,  5.42s/it]
  5%|▌         | 324/6100 [29:17<8:41:55,  5.42s/it]
  5%|▌         | 325/6100 [29:22<8:41:27,  5.42s/it]
                                                    
{'loss': 0.2757, 'learning_rate': 8.372960807150883e-05, 'epoch': 0.53}

  5%|▌         | 325/6100 [29:22<8:41:27,  5.42s/it]
  5%|▌         | 326/6100 [29:27<8:41:23,  5.42s/it]
  5%|▌         | 327/6100 [29:33<8:41:20,  5.42s/it]
  5%|▌         | 328/6100 [29:38<8:40:52,  5.41s/it]
  5%|▌         | 329/6100 [29:44<8:42:29,  5.43s/it]
  5%|▌         | 330/6100 [29:49<8:41:26,  5.42s/it]
                                                    
{'loss': 0.276, 'learning_rate': 8.395062515794963e-05, 'epoch': 0.54}

  5%|▌         | 330/6100 [29:49<8:41:26,  5.42s/it]
  5%|▌         | 331/6100 [29:55<8:41:13,  5.42s/it]
  5%|▌         | 332/6100 [30:00<8:40:44,  5.42s/it]
  5%|▌         | 333/6100 [30:05<8:40:13,  5.41s/it]
  5%|▌         | 334/6100 [30:11<8:40:00,  5.41s/it]
  5%|▌         | 335/6100 [30:16<8:40:12,  5.41s/it]
                                                    
{'loss': 0.2687, 'learning_rate': 8.416831855295917e-05, 'epoch': 0.55}

  5%|▌         | 335/6100 [30:16<8:40:12,  5.41s/it]
  6%|▌         | 336/6100 [30:22<8:40:02,  5.41s/it]
  6%|▌         | 337/6100 [30:27<8:40:03,  5.41s/it]
  6%|▌         | 338/6100 [30:32<8:39:49,  5.41s/it]
  6%|▌         | 339/6100 [30:38<8:39:44,  5.41s/it]
  6%|▌         | 340/6100 [30:43<8:40:14,  5.42s/it]
                                                    
{'loss': 0.2654, 'learning_rate': 8.43827867417695e-05, 'epoch': 0.56}

  6%|▌         | 340/6100 [30:43<8:40:14,  5.42s/it]
  6%|▌         | 341/6100 [30:49<8:40:10,  5.42s/it]
  6%|▌         | 342/6100 [30:54<8:39:42,  5.42s/it]
  6%|▌         | 343/6100 [31:00<8:39:17,  5.41s/it]
  6%|▌         | 344/6100 [31:05<8:39:03,  5.41s/it]
  6%|▌         | 345/6100 [31:10<8:38:53,  5.41s/it]
                                                    
{'loss': 0.2728, 'learning_rate': 8.459412389607746e-05, 'epoch': 0.57}

  6%|▌         | 345/6100 [31:10<8:38:53,  5.41s/it]
  6%|▌         | 346/6100 [31:16<8:38:39,  5.41s/it]
  6%|▌         | 347/6100 [31:21<8:39:06,  5.41s/it]
  6%|▌         | 348/6100 [31:27<8:38:50,  5.41s/it]
  6%|▌         | 349/6100 [31:32<8:39:01,  5.41s/it]
  6%|▌         | 350/6100 [31:37<8:38:58,  5.42s/it]
                                                    
{'loss': 0.2717, 'learning_rate': 8.480242012232773e-05, 'epoch': 0.57}

  6%|▌         | 350/6100 [31:37<8:38:58,  5.42s/it]
  6%|▌         | 351/6100 [31:43<8:38:42,  5.41s/it]
  6%|▌         | 352/6100 [31:48<8:38:43,  5.41s/it]
  6%|▌         | 353/6100 [31:54<8:38:54,  5.42s/it]
  6%|▌         | 354/6100 [31:59<8:38:46,  5.42s/it]
  6%|▌         | 355/6100 [32:05<8:38:25,  5.41s/it]
                                                    
{'loss': 0.2547, 'learning_rate': 8.500776169238547e-05, 'epoch': 0.58}

  6%|▌         | 355/6100 [32:05<8:38:25,  5.41s/it]
  6%|▌         | 356/6100 [32:10<8:38:20,  5.41s/it]
  6%|▌         | 357/6100 [32:15<8:38:08,  5.41s/it]
  6%|▌         | 358/6100 [32:21<8:38:00,  5.41s/it]
  6%|▌         | 359/6100 [32:26<8:37:20,  5.41s/it]
  6%|▌         | 360/6100 [32:32<8:37:17,  5.41s/it]
                                                    
{'loss': 0.2625, 'learning_rate': 8.5210231258076e-05, 'epoch': 0.59}

  6%|▌         | 360/6100 [32:32<8:37:17,  5.41s/it]
  6%|▌         | 361/6100 [32:37<8:37:16,  5.41s/it]
  6%|▌         | 362/6100 [32:42<8:37:10,  5.41s/it]
  6%|▌         | 363/6100 [32:48<8:37:02,  5.41s/it]
  6%|▌         | 364/6100 [32:53<8:37:18,  5.41s/it]
  6%|▌         | 365/6100 [32:59<8:37:01,  5.41s/it]
                                                    
{'loss': 0.2614, 'learning_rate': 8.540990805092769e-05, 'epoch': 0.6}

  6%|▌         | 365/6100 [32:59<8:37:01,  5.41s/it]
  6%|▌         | 366/6100 [33:04<8:36:56,  5.41s/it]
  6%|▌         | 367/6100 [33:09<8:36:59,  5.41s/it]
  6%|▌         | 368/6100 [33:15<8:37:14,  5.41s/it]
  6%|▌         | 369/6100 [33:20<8:37:05,  5.41s/it]
  6%|▌         | 370/6100 [33:26<8:36:52,  5.41s/it]
                                                    
{'loss': 0.2571, 'learning_rate': 8.560686806832515e-05, 'epoch': 0.61}

  6%|▌         | 370/6100 [33:26<8:36:52,  5.41s/it]
  6%|▌         | 371/6100 [33:31<8:36:41,  5.41s/it]
  6%|▌         | 372/6100 [33:37<8:36:47,  5.41s/it]
  6%|▌         | 373/6100 [33:42<8:37:06,  5.42s/it]
  6%|▌         | 374/6100 [33:47<8:36:54,  5.42s/it]
  6%|▌         | 375/6100 [33:53<8:36:37,  5.41s/it]
                                                    
{'loss': 0.2423, 'learning_rate': 8.580118424716806e-05, 'epoch': 0.61}

  6%|▌         | 375/6100 [33:53<8:36:37,  5.41s/it]
  6%|▌         | 376/6100 [33:58<8:36:46,  5.42s/it]
  6%|▌         | 377/6100 [34:04<8:36:32,  5.42s/it]
  6%|▌         | 378/6100 [34:09<8:36:23,  5.41s/it]
  6%|▌         | 379/6100 [34:14<8:36:10,  5.41s/it]
  6%|▌         | 380/6100 [34:20<8:36:16,  5.42s/it]
                                                    
{'loss': 0.265, 'learning_rate': 8.599292662602815e-05, 'epoch': 0.62}

  6%|▌         | 380/6100 [34:20<8:36:16,  5.42s/it]
  6%|▌         | 381/6100 [34:25<8:36:30,  5.42s/it]
  6%|▋         | 382/6100 [34:31<8:36:09,  5.42s/it]
  6%|▋         | 383/6100 [34:36<8:36:07,  5.42s/it]
  6%|▋         | 384/6100 [34:42<8:36:16,  5.42s/it]
  6%|▋         | 385/6100 [34:47<8:35:43,  5.41s/it]
                                                    
{'loss': 0.2617, 'learning_rate': 8.618216249670687e-05, 'epoch': 0.63}

  6%|▋         | 385/6100 [34:47<8:35:43,  5.41s/it]
  6%|▋         | 386/6100 [34:52<8:35:25,  5.41s/it]
  6%|▋         | 387/6100 [34:58<8:35:30,  5.41s/it]
  6%|▋         | 388/6100 [35:03<8:35:20,  5.41s/it]
  6%|▋         | 389/6100 [35:09<8:35:03,  5.41s/it]
  6%|▋         | 390/6100 [35:14<8:34:44,  5.41s/it]
                                                    
{'loss': 0.2424, 'learning_rate': 8.636895654601431e-05, 'epoch': 0.64}

  6%|▋         | 390/6100 [35:14<8:34:44,  5.41s/it]
  6%|▋         | 391/6100 [35:19<8:35:06,  5.41s/it]
  6%|▋         | 392/6100 [35:25<8:34:53,  5.41s/it]
  6%|▋         | 393/6100 [35:30<8:34:45,  5.41s/it]
  6%|▋         | 394/6100 [35:36<8:34:20,  5.41s/it]
  6%|▋         | 395/6100 [35:41<8:34:22,  5.41s/it]
                                                    
{'loss': 0.2412, 'learning_rate': 8.655337098851681e-05, 'epoch': 0.65}

  6%|▋         | 395/6100 [35:41<8:34:22,  5.41s/it]
  6%|▋         | 396/6100 [35:46<8:34:18,  5.41s/it]
  7%|▋         | 397/6100 [35:52<8:34:08,  5.41s/it]
  7%|▋         | 398/6100 [35:57<8:34:11,  5.41s/it]
  7%|▋         | 399/6100 [36:03<8:33:52,  5.41s/it]
  7%|▋         | 400/6100 [36:08<8:34:03,  5.41s/it]
                                                    
{'loss': 0.2375, 'learning_rate': 8.673546569093498e-05, 'epoch': 0.66}

  7%|▋         | 400/6100 [36:08<8:34:03,  5.41s/it]
  7%|▋         | 401/6100 [36:13<8:34:06,  5.41s/it]
  7%|▋         | 402/6100 [36:19<8:33:43,  5.41s/it]
  7%|▋         | 403/6100 [36:24<8:33:49,  5.41s/it]
  7%|▋         | 404/6100 [36:30<8:34:03,  5.41s/it]
  7%|▋         | 405/6100 [36:35<8:33:32,  5.41s/it]
                                                    
{'loss': 0.2504, 'learning_rate': 8.691529828881455e-05, 'epoch': 0.66}

  7%|▋         | 405/6100 [36:35<8:33:32,  5.41s/it]
  7%|▋         | 406/6100 [36:41<8:40:25,  5.48s/it]
  7%|▋         | 407/6100 [36:46<8:37:48,  5.46s/it]
  7%|▋         | 408/6100 [36:52<8:36:35,  5.45s/it]
  7%|▋         | 409/6100 [36:57<8:35:28,  5.43s/it]
  7%|▋         | 410/6100 [37:02<8:34:37,  5.43s/it]
                                                    
{'loss': 0.2618, 'learning_rate': 8.709292429603896e-05, 'epoch': 0.67}

  7%|▋         | 410/6100 [37:02<8:34:37,  5.43s/it]
  7%|▋         | 411/6100 [37:08<8:33:53,  5.42s/it]
  7%|▋         | 412/6100 [37:13<8:33:49,  5.42s/it]
  7%|▋         | 413/6100 [37:19<8:33:34,  5.42s/it]
  7%|▋         | 414/6100 [37:24<8:33:23,  5.42s/it]
  7%|▋         | 415/6100 [37:29<8:33:12,  5.42s/it]
                                                    
{'loss': 0.2524, 'learning_rate': 8.726839720770418e-05, 'epoch': 0.68}

  7%|▋         | 415/6100 [37:29<8:33:12,  5.42s/it]
  7%|▋         | 416/6100 [37:35<8:33:15,  5.42s/it]
  7%|▋         | 417/6100 [37:40<8:33:11,  5.42s/it]
  7%|▋         | 418/6100 [37:46<8:32:39,  5.41s/it]
  7%|▋         | 419/6100 [37:51<8:32:11,  5.41s/it]
  7%|▋         | 420/6100 [37:57<8:32:00,  5.41s/it]
                                                    
{'loss': 0.2446, 'learning_rate': 8.744176859683322e-05, 'epoch': 0.69}

  7%|▋         | 420/6100 [37:57<8:32:00,  5.41s/it]
  7%|▋         | 421/6100 [38:02<8:32:03,  5.41s/it]
  7%|▋         | 422/6100 [38:07<8:32:01,  5.41s/it]
  7%|▋         | 423/6100 [38:13<8:31:57,  5.41s/it]
  7%|▋         | 424/6100 [38:18<8:31:48,  5.41s/it]
  7%|▋         | 425/6100 [38:24<8:31:32,  5.41s/it]
                                                    
{'loss': 0.2537, 'learning_rate': 8.761308820536704e-05, 'epoch': 0.7}

  7%|▋         | 425/6100 [38:24<8:31:32,  5.41s/it]
  7%|▋         | 426/6100 [38:29<8:31:34,  5.41s/it]
  7%|▋         | 427/6100 [38:34<8:31:22,  5.41s/it]
  7%|▋         | 428/6100 [38:40<8:34:28,  5.44s/it]
  7%|▋         | 429/6100 [38:45<8:33:33,  5.43s/it]
  7%|▋         | 430/6100 [38:51<8:32:52,  5.43s/it]
                                                    
{'loss': 0.2565, 'learning_rate': 8.778240402983436e-05, 'epoch': 0.7}

  7%|▋         | 430/6100 [38:51<8:32:52,  5.43s/it]
  7%|▋         | 431/6100 [38:56<8:33:26,  5.43s/it]
  7%|▋         | 432/6100 [39:02<8:32:28,  5.42s/it]
  7%|▋         | 433/6100 [39:07<8:31:45,  5.42s/it]
  7%|▋         | 434/6100 [39:12<8:31:23,  5.42s/it]
  7%|▋         | 435/6100 [39:18<8:31:19,  5.42s/it]
                                                    
{'loss': 0.242, 'learning_rate': 8.794976240206893e-05, 'epoch': 0.71}

  7%|▋         | 435/6100 [39:18<8:31:19,  5.42s/it]
  7%|▋         | 436/6100 [39:23<8:30:54,  5.41s/it]
  7%|▋         | 437/6100 [39:29<8:30:41,  5.41s/it]
  7%|▋         | 438/6100 [39:34<8:30:54,  5.41s/it]
  7%|▋         | 439/6100 [39:39<8:30:36,  5.41s/it]
  7%|▋         | 440/6100 [39:45<8:30:48,  5.41s/it]
                                                    
{'loss': 0.2435, 'learning_rate': 8.811520806531409e-05, 'epoch': 0.72}

  7%|▋         | 440/6100 [39:45<8:30:48,  5.41s/it]
  7%|▋         | 441/6100 [39:50<8:30:50,  5.42s/it]
  7%|▋         | 442/6100 [39:56<8:30:37,  5.41s/it]
  7%|▋         | 443/6100 [40:01<8:30:45,  5.42s/it]
  7%|▋         | 444/6100 [40:07<8:30:45,  5.42s/it]
  7%|▋         | 445/6100 [40:12<8:30:18,  5.41s/it]
                                                    
{'loss': 0.2344, 'learning_rate': 8.82787842460274e-05, 'epoch': 0.73}

  7%|▋         | 445/6100 [40:12<8:30:18,  5.41s/it]
  7%|▋         | 446/6100 [40:17<8:29:56,  5.41s/it]
  7%|▋         | 447/6100 [40:23<8:30:04,  5.41s/it]
  7%|▋         | 448/6100 [40:28<8:30:11,  5.42s/it]
  7%|▋         | 449/6100 [40:34<8:30:04,  5.42s/it]
  7%|▋         | 450/6100 [40:39<8:30:25,  5.42s/it]
                                                    
{'loss': 0.2421, 'learning_rate': 8.844053272167354e-05, 'epoch': 0.74}

  7%|▋         | 450/6100 [40:39<8:30:25,  5.42s/it]Saving model checkpoint to ./results/checkpoint-450
Configuration saved in ./results/checkpoint-450/config.json
Model weights saved in ./results/checkpoint-450/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-450/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-450/special_tokens_map.json
[2023-02-21 20:32:55,221] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step450 is begin to save!
[2023-02-21 20:32:55,225] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-450/global_step450/mp_rank_00_model_states.pt
[2023-02-21 20:32:55,225] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-450/global_step450/mp_rank_00_model_states.pt...
[2023-02-21 20:32:56,041] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-450/global_step450/mp_rank_00_model_states.pt.
[2023-02-21 20:32:56,044] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-450/global_step450/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-21 20:32:56,218] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-450/global_step450/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-21 20:32:56,218] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-450/global_step450/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-21 20:32:56,218] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step450 is ready now!
Deleting older checkpoint [results/checkpoint-300] due to args.save_total_limit

  7%|▋         | 451/6100 [40:47<9:29:33,  6.05s/it]
  7%|▋         | 452/6100 [40:52<9:11:14,  5.86s/it]
  7%|▋         | 453/6100 [40:57<8:58:18,  5.72s/it]
  7%|▋         | 454/6100 [41:03<8:49:24,  5.63s/it]
  7%|▋         | 455/6100 [41:08<8:42:51,  5.56s/it]
                                                    
{'loss': 0.2408, 'learning_rate': 8.860049388477153e-05, 'epoch': 0.75}

  7%|▋         | 455/6100 [41:08<8:42:51,  5.56s/it]
  7%|▋         | 456/6100 [41:14<8:38:34,  5.51s/it]
  7%|▋         | 457/6100 [41:19<8:36:05,  5.49s/it]
  8%|▊         | 458/6100 [41:24<8:33:36,  5.46s/it]
  8%|▊         | 459/6100 [41:30<8:32:04,  5.45s/it]
  8%|▊         | 460/6100 [41:35<8:30:52,  5.43s/it]
                                                    
{'loss': 0.2272, 'learning_rate': 8.875870680344193e-05, 'epoch': 0.75}

  8%|▊         | 460/6100 [41:35<8:30:52,  5.43s/it]
  8%|▊         | 461/6100 [41:41<8:30:01,  5.43s/it]
  8%|▊         | 462/6100 [41:46<8:29:09,  5.42s/it]
  8%|▊         | 463/6100 [41:52<8:30:42,  5.44s/it]
  8%|▊         | 464/6100 [41:57<8:29:53,  5.43s/it]
  8%|▊         | 465/6100 [42:02<8:29:25,  5.42s/it]
                                                    
{'loss': 0.2225, 'learning_rate': 8.891520927868084e-05, 'epoch': 0.76}

  8%|▊         | 465/6100 [42:02<8:29:25,  5.42s/it]
  8%|▊         | 466/6100 [42:08<8:29:06,  5.42s/it]
  8%|▊         | 467/6100 [42:13<8:28:42,  5.42s/it]
  8%|▊         | 468/6100 [42:19<8:28:29,  5.42s/it]
  8%|▊         | 469/6100 [42:24<8:28:11,  5.42s/it]
  8%|▊         | 470/6100 [42:29<8:27:58,  5.41s/it]
                                                    
{'loss': 0.2305, 'learning_rate': 8.907003789857127e-05, 'epoch': 0.77}

  8%|▊         | 470/6100 [42:29<8:27:58,  5.41s/it]
  8%|▊         | 471/6100 [42:35<8:27:57,  5.41s/it]
  8%|▊         | 472/6100 [42:40<8:27:49,  5.41s/it]
  8%|▊         | 473/6100 [42:46<8:28:31,  5.42s/it]
  8%|▊         | 474/6100 [42:51<8:27:57,  5.42s/it]
  8%|▊         | 475/6100 [42:57<8:27:26,  5.41s/it]
                                                    
{'loss': 0.2316, 'learning_rate': 8.922322808962569e-05, 'epoch': 0.78}

  8%|▊         | 475/6100 [42:57<8:27:26,  5.41s/it]
  8%|▊         | 476/6100 [43:02<8:27:17,  5.41s/it]
  8%|▊         | 477/6100 [43:07<8:26:58,  5.41s/it]
  8%|▊         | 478/6100 [43:13<8:26:57,  5.41s/it]
  8%|▊         | 479/6100 [43:18<8:26:40,  5.41s/it]
  8%|▊         | 480/6100 [43:24<8:26:33,  5.41s/it]
                                                    
{'loss': 0.2278, 'learning_rate': 8.937481416544046e-05, 'epoch': 0.79}

  8%|▊         | 480/6100 [43:24<8:26:33,  5.41s/it]
  8%|▊         | 481/6100 [43:29<8:26:15,  5.41s/it]
  8%|▊         | 482/6100 [43:34<8:27:08,  5.42s/it]
  8%|▊         | 483/6100 [43:40<8:27:11,  5.42s/it]
  8%|▊         | 484/6100 [43:45<8:27:19,  5.42s/it]
  8%|▊         | 485/6100 [43:51<8:27:41,  5.42s/it]
                                                    
{'loss': 0.2258, 'learning_rate': 8.952482937282926e-05, 'epoch': 0.79}

  8%|▊         | 485/6100 [43:51<8:27:41,  5.42s/it]
  8%|▊         | 486/6100 [43:56<8:27:03,  5.42s/it]
  8%|▊         | 487/6100 [44:01<8:27:00,  5.42s/it]
  8%|▊         | 488/6100 [44:07<8:26:55,  5.42s/it]
  8%|▊         | 489/6100 [44:12<8:26:34,  5.42s/it]
  8%|▊         | 490/6100 [44:18<8:26:43,  5.42s/it]
                                                    
{'loss': 0.2259, 'learning_rate': 8.967330593559045e-05, 'epoch': 0.8}

  8%|▊         | 490/6100 [44:18<8:26:43,  5.42s/it]
  8%|▊         | 491/6100 [44:23<8:26:38,  5.42s/it]
  8%|▊         | 492/6100 [44:29<8:26:17,  5.42s/it]
  8%|▊         | 493/6100 [44:34<8:26:11,  5.42s/it]
  8%|▊         | 494/6100 [44:39<8:25:48,  5.41s/it]
  8%|▊         | 495/6100 [44:45<8:25:37,  5.41s/it]
                                                    
{'loss': 0.2273, 'learning_rate': 8.982027509605266e-05, 'epoch': 0.81}

  8%|▊         | 495/6100 [44:45<8:25:37,  5.41s/it]
  8%|▊         | 496/6100 [44:50<8:25:23,  5.41s/it]
  8%|▊         | 497/6100 [44:56<8:25:33,  5.41s/it]
  8%|▊         | 498/6100 [45:01<8:25:11,  5.41s/it]
  8%|▊         | 499/6100 [45:06<8:25:05,  5.41s/it]
  8%|▊         | 500/6100 [45:12<8:25:04,  5.41s/it]
                                                    
{'loss': 0.2288, 'learning_rate': 8.996576715453252e-05, 'epoch': 0.82}

  8%|▊         | 500/6100 [45:12<8:25:04,  5.41s/it]
  8%|▊         | 501/6100 [45:17<8:25:13,  5.41s/it]
  8%|▊         | 502/6100 [45:23<8:25:06,  5.41s/it]
  8%|▊         | 503/6100 [45:28<8:24:53,  5.41s/it]
  8%|▊         | 504/6100 [45:34<8:25:03,  5.42s/it]
  8%|▊         | 505/6100 [45:39<8:25:07,  5.42s/it]
                                                    
{'loss': 0.2328, 'learning_rate': 9.010981150682935e-05, 'epoch': 0.83}

  8%|▊         | 505/6100 [45:39<8:25:07,  5.42s/it]
  8%|▊         | 506/6100 [45:44<8:24:52,  5.42s/it]
  8%|▊         | 507/6100 [45:50<8:24:26,  5.41s/it]
  8%|▊         | 508/6100 [45:55<8:24:38,  5.41s/it]
  8%|▊         | 509/6100 [46:01<8:24:23,  5.41s/it]
  8%|▊         | 510/6100 [46:06<8:24:25,  5.41s/it]
                                                    
{'loss': 0.2331, 'learning_rate': 9.025243667987251e-05, 'epoch': 0.84}

  8%|▊         | 510/6100 [46:06<8:24:25,  5.41s/it]
  8%|▊         | 511/6100 [46:11<8:24:28,  5.42s/it]
  8%|▊         | 512/6100 [46:17<8:24:00,  5.41s/it]
  8%|▊         | 513/6100 [46:22<8:23:52,  5.41s/it]
  8%|▊         | 514/6100 [46:28<8:23:23,  5.41s/it]
  8%|▊         | 515/6100 [46:33<8:23:31,  5.41s/it]
                                                    
{'loss': 0.2136, 'learning_rate': 9.039367036563003e-05, 'epoch': 0.84}

  8%|▊         | 515/6100 [46:33<8:23:31,  5.41s/it]
  8%|▊         | 516/6100 [46:39<8:24:25,  5.42s/it]
  8%|▊         | 517/6100 [46:44<8:24:27,  5.42s/it]
  8%|▊         | 518/6100 [46:49<8:24:05,  5.42s/it]
  9%|▊         | 519/6100 [46:55<8:23:55,  5.42s/it]
  9%|▊         | 520/6100 [47:00<8:23:45,  5.42s/it]
                                                    
{'loss': 0.2139, 'learning_rate': 9.053353945337877e-05, 'epoch': 0.85}

  9%|▊         | 520/6100 [47:00<8:23:45,  5.42s/it]
  9%|▊         | 521/6100 [47:06<8:23:40,  5.42s/it]
  9%|▊         | 522/6100 [47:11<8:23:59,  5.42s/it]
  9%|▊         | 523/6100 [47:16<8:23:38,  5.42s/it]
  9%|▊         | 524/6100 [47:22<8:23:51,  5.42s/it]
  9%|▊         | 525/6100 [47:27<8:23:30,  5.42s/it]
                                                    
{'loss': 0.2256, 'learning_rate': 9.067207006043077e-05, 'epoch': 0.86}

  9%|▊         | 525/6100 [47:27<8:23:30,  5.42s/it]
  9%|▊         | 526/6100 [47:33<8:23:36,  5.42s/it]
  9%|▊         | 527/6100 [47:38<8:23:23,  5.42s/it]
  9%|▊         | 528/6100 [47:44<8:23:06,  5.42s/it]
  9%|▊         | 529/6100 [47:49<8:22:47,  5.42s/it]
  9%|▊         | 530/6100 [47:54<8:22:45,  5.42s/it]
                                                    
{'loss': 0.2334, 'learning_rate': 9.080928756140312e-05, 'epoch': 0.87}

  9%|▊         | 530/6100 [47:54<8:22:45,  5.42s/it]
  9%|▊         | 531/6100 [48:00<8:22:41,  5.42s/it]
  9%|▊         | 532/6100 [48:05<8:22:21,  5.41s/it]
  9%|▊         | 533/6100 [48:11<8:22:15,  5.41s/it]
  9%|▉         | 534/6100 [48:16<8:22:39,  5.42s/it]
  9%|▉         | 535/6100 [48:21<8:22:42,  5.42s/it]
                                                    
{'loss': 0.2244, 'learning_rate': 9.094521661611361e-05, 'epoch': 0.88}

  9%|▉         | 535/6100 [48:21<8:22:42,  5.42s/it]
  9%|▉         | 536/6100 [48:27<8:22:10,  5.42s/it]
  9%|▉         | 537/6100 [48:32<8:21:52,  5.41s/it]
  9%|▉         | 538/6100 [48:38<8:21:27,  5.41s/it]
  9%|▉         | 539/6100 [48:43<8:21:50,  5.41s/it]
  9%|▉         | 540/6100 [48:49<8:23:00,  5.43s/it]
                                                    
{'loss': 0.2096, 'learning_rate': 9.107988119617903e-05, 'epoch': 0.88}

  9%|▉         | 540/6100 [48:49<8:23:00,  5.43s/it]
  9%|▉         | 541/6100 [48:54<8:22:44,  5.43s/it]
  9%|▉         | 542/6100 [48:59<8:22:27,  5.42s/it]
  9%|▉         | 543/6100 [49:05<8:22:00,  5.42s/it]
  9%|▉         | 544/6100 [49:10<8:21:44,  5.42s/it]
  9%|▉         | 545/6100 [49:16<8:21:21,  5.42s/it]
                                                    
{'loss': 0.2171, 'learning_rate': 9.121330461038732e-05, 'epoch': 0.89}

  9%|▉         | 545/6100 [49:16<8:21:21,  5.42s/it]
  9%|▉         | 546/6100 [49:21<8:21:22,  5.42s/it]
  9%|▉         | 547/6100 [49:26<8:21:24,  5.42s/it]
  9%|▉         | 548/6100 [49:32<8:21:20,  5.42s/it]
  9%|▉         | 549/6100 [49:37<8:21:02,  5.42s/it]
  9%|▉         | 550/6100 [49:43<8:21:15,  5.42s/it]
                                                    
{'loss': 0.2181, 'learning_rate': 9.134550952891163e-05, 'epoch': 0.9}

  9%|▉         | 550/6100 [49:43<8:21:15,  5.42s/it]
  9%|▉         | 551/6100 [49:48<8:21:08,  5.42s/it]
  9%|▉         | 552/6100 [49:54<8:20:23,  5.41s/it]
  9%|▉         | 553/6100 [49:59<8:20:59,  5.42s/it]
  9%|▉         | 554/6100 [50:04<8:20:32,  5.42s/it]
  9%|▉         | 555/6100 [50:10<8:20:28,  5.42s/it]
                                                    
{'loss': 0.2072, 'learning_rate': 9.147651800642817e-05, 'epoch': 0.91}

  9%|▉         | 555/6100 [50:10<8:20:28,  5.42s/it]
  9%|▉         | 556/6100 [50:15<8:20:50,  5.42s/it]
  9%|▉         | 557/6100 [50:21<8:20:24,  5.42s/it]
  9%|▉         | 558/6100 [50:26<8:20:13,  5.42s/it]
  9%|▉         | 559/6100 [50:31<8:20:00,  5.41s/it]
  9%|▉         | 560/6100 [50:37<8:19:58,  5.41s/it]
                                                    
{'loss': 0.2126, 'learning_rate': 9.160635150419769e-05, 'epoch': 0.92}

  9%|▉         | 560/6100 [50:37<8:19:58,  5.41s/it]
  9%|▉         | 561/6100 [50:42<8:20:00,  5.42s/it]
  9%|▉         | 562/6100 [50:48<8:19:48,  5.42s/it]
  9%|▉         | 563/6100 [50:53<8:19:39,  5.41s/it]
  9%|▉         | 564/6100 [50:59<8:19:50,  5.42s/it]
  9%|▉         | 565/6100 [51:04<8:20:17,  5.42s/it]
                                                    
{'loss': 0.2108, 'learning_rate': 9.173503091116537e-05, 'epoch': 0.93}

  9%|▉         | 565/6100 [51:04<8:20:17,  5.42s/it]
  9%|▉         | 566/6100 [51:09<8:20:14,  5.42s/it]
  9%|▉         | 567/6100 [51:15<8:19:56,  5.42s/it]
  9%|▉         | 568/6100 [51:20<8:19:25,  5.42s/it]
  9%|▉         | 569/6100 [51:26<8:19:29,  5.42s/it]
  9%|▉         | 570/6100 [51:31<8:19:11,  5.42s/it]
                                                    
{'loss': 0.2211, 'learning_rate': 9.186257656413118e-05, 'epoch': 0.93}

  9%|▉         | 570/6100 [51:31<8:19:11,  5.42s/it]
  9%|▉         | 571/6100 [51:36<8:18:57,  5.41s/it]
  9%|▉         | 572/6100 [51:42<8:19:20,  5.42s/it]
  9%|▉         | 573/6100 [51:47<8:18:57,  5.42s/it]
  9%|▉         | 574/6100 [51:53<8:38:56,  5.63s/it]
  9%|▉         | 575/6100 [51:59<8:32:39,  5.57s/it]
                                                    
{'loss': 0.2014, 'learning_rate': 9.198900826703947e-05, 'epoch': 0.94}

  9%|▉         | 575/6100 [51:59<8:32:39,  5.57s/it]
  9%|▉         | 576/6100 [52:04<8:28:40,  5.53s/it]
  9%|▉         | 577/6100 [52:10<8:25:22,  5.49s/it]
  9%|▉         | 578/6100 [52:15<8:23:06,  5.47s/it]
  9%|▉         | 579/6100 [52:21<8:21:52,  5.45s/it]
 10%|▉         | 580/6100 [52:26<8:20:34,  5.44s/it]
                                                    
{'loss': 0.2169, 'learning_rate': 9.21143453094334e-05, 'epoch': 0.95}

 10%|▉         | 580/6100 [52:26<8:20:34,  5.44s/it]
 10%|▉         | 581/6100 [52:31<8:19:57,  5.44s/it]
 10%|▉         | 582/6100 [52:37<8:19:19,  5.43s/it]
 10%|▉         | 583/6100 [52:42<8:19:07,  5.43s/it]
 10%|▉         | 584/6100 [52:48<8:18:30,  5.42s/it]
 10%|▉         | 585/6100 [52:53<8:18:14,  5.42s/it]
                                                    
{'loss': 0.2121, 'learning_rate': 9.223860648411733e-05, 'epoch': 0.96}

 10%|▉         | 585/6100 [52:53<8:18:14,  5.42s/it]
 10%|▉         | 586/6100 [52:58<8:18:07,  5.42s/it]
 10%|▉         | 587/6100 [53:04<8:17:43,  5.42s/it]
 10%|▉         | 588/6100 [53:09<8:17:50,  5.42s/it]
 10%|▉         | 589/6100 [53:15<8:17:20,  5.41s/it]
 10%|▉         | 590/6100 [53:20<8:17:15,  5.41s/it]
                                                    
{'loss': 0.2039, 'learning_rate': 9.23618101040676e-05, 'epoch': 0.97}

 10%|▉         | 590/6100 [53:20<8:17:15,  5.41s/it]
 10%|▉         | 591/6100 [53:26<8:17:16,  5.42s/it]
 10%|▉         | 592/6100 [53:31<8:16:57,  5.41s/it]
 10%|▉         | 593/6100 [53:36<8:16:27,  5.41s/it]
 10%|▉         | 594/6100 [53:42<8:16:53,  5.41s/it]
 10%|▉         | 595/6100 [53:47<8:16:31,  5.41s/it]
                                                    
{'loss': 0.1986, 'learning_rate': 9.248397401862976e-05, 'epoch': 0.97}

 10%|▉         | 595/6100 [53:47<8:16:31,  5.41s/it]
 10%|▉         | 596/6100 [53:53<8:16:23,  5.41s/it]
 10%|▉         | 597/6100 [53:58<8:15:42,  5.40s/it]
 10%|▉         | 598/6100 [54:03<8:16:07,  5.41s/it]
 10%|▉         | 599/6100 [54:09<8:16:21,  5.41s/it]
 10%|▉         | 600/6100 [54:14<8:16:09,  5.41s/it]
                                                    
{'loss': 0.2, 'learning_rate': 9.2605115629038e-05, 'epoch': 0.98}

 10%|▉         | 600/6100 [54:14<8:16:09,  5.41s/it]Saving model checkpoint to ./results/checkpoint-600
Configuration saved in ./results/checkpoint-600/config.json
Model weights saved in ./results/checkpoint-600/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-600/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-600/special_tokens_map.json
[2023-02-21 20:46:30,422] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step600 is begin to save!
[2023-02-21 20:46:30,425] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-600/global_step600/mp_rank_00_model_states.pt
[2023-02-21 20:46:30,426] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-600/global_step600/mp_rank_00_model_states.pt...
[2023-02-21 20:46:31,253] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-600/global_step600/mp_rank_00_model_states.pt.
[2023-02-21 20:46:31,255] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-600/global_step600/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-21 20:46:31,430] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-600/global_step600/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-21 20:46:31,430] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-600/global_step600/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-21 20:46:31,431] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step600 is ready now!
Deleting older checkpoint [results/checkpoint-450] due to args.save_total_limit

 10%|▉         | 601/6100 [54:22<9:15:48,  6.06s/it]
 10%|▉         | 602/6100 [54:27<8:57:33,  5.87s/it]
 10%|▉         | 603/6100 [54:33<8:45:13,  5.73s/it]
 10%|▉         | 604/6100 [54:38<8:36:07,  5.63s/it]
 10%|▉         | 605/6100 [54:43<8:29:55,  5.57s/it]
                                                    
{'loss': 0.1991, 'learning_rate': 9.272525190329075e-05, 'epoch': 0.99}

 10%|▉         | 605/6100 [54:43<8:29:55,  5.57s/it]
 10%|▉         | 606/6100 [54:49<8:25:22,  5.52s/it]
 10%|▉         | 607/6100 [54:54<8:21:56,  5.48s/it]
 10%|▉         | 608/6100 [55:00<8:19:50,  5.46s/it]
 10%|▉         | 609/6100 [55:05<8:18:12,  5.44s/it]
 10%|█         | 610/6100 [55:10<8:17:30,  5.44s/it]
                                                    
{'loss': 0.2001, 'learning_rate': 9.284439939041391e-05, 'epoch': 1.0}

 10%|█         | 610/6100 [55:10<8:17:30,  5.44s/it]
 10%|█         | 611/6100 [55:19<9:28:21,  6.21s/it]
 10%|█         | 612/6100 [55:24<9:06:28,  5.97s/it]
 10%|█         | 613/6100 [55:29<8:50:24,  5.80s/it]
 10%|█         | 614/6100 [55:35<8:39:53,  5.69s/it]
 10%|█         | 615/6100 [55:40<8:32:20,  5.60s/it]
                                                    
{'loss': 0.2191, 'learning_rate': 9.296257423414198e-05, 'epoch': 1.01}

 10%|█         | 615/6100 [55:40<8:32:20,  5.60s/it]
 10%|█         | 616/6100 [55:46<8:27:06,  5.55s/it]
 10%|█         | 617/6100 [55:51<8:23:24,  5.51s/it]
 10%|█         | 618/6100 [55:56<8:21:02,  5.48s/it]
 10%|█         | 619/6100 [56:02<8:18:56,  5.46s/it]
 10%|█         | 620/6100 [56:07<8:17:24,  5.45s/it]
                                                    
{'loss': 0.1954, 'learning_rate': 9.307979218604531e-05, 'epoch': 1.02}

 10%|█         | 620/6100 [56:07<8:17:24,  5.45s/it]
 10%|█         | 621/6100 [56:13<8:16:29,  5.44s/it]
 10%|█         | 622/6100 [56:18<8:16:00,  5.43s/it]
 10%|█         | 623/6100 [56:23<8:14:59,  5.42s/it]
 10%|█         | 624/6100 [56:29<8:14:43,  5.42s/it]
 10%|█         | 625/6100 [56:34<8:14:03,  5.41s/it]
                                                    
{'loss': 0.1919, 'learning_rate': 9.319606861813007e-05, 'epoch': 1.02}

 10%|█         | 625/6100 [56:34<8:14:03,  5.41s/it]
 10%|█         | 626/6100 [56:40<8:13:39,  5.41s/it]
 10%|█         | 627/6100 [56:45<8:13:34,  5.41s/it]
 10%|█         | 628/6100 [56:51<8:13:26,  5.41s/it]
 10%|█         | 629/6100 [56:56<8:13:26,  5.41s/it]
 10%|█         | 630/6100 [57:01<8:13:13,  5.41s/it]
                                                    
{'loss': 0.1932, 'learning_rate': 9.331141853493626e-05, 'epoch': 1.03}

 10%|█         | 630/6100 [57:01<8:13:13,  5.41s/it]
 10%|█         | 631/6100 [57:07<8:13:33,  5.41s/it]
 10%|█         | 632/6100 [57:12<8:13:28,  5.41s/it]
 10%|█         | 633/6100 [57:18<8:13:52,  5.42s/it]
 10%|█         | 634/6100 [57:23<8:13:37,  5.42s/it]
 10%|█         | 635/6100 [57:28<8:13:08,  5.41s/it]
                                                    
{'loss': 0.1989, 'learning_rate': 9.342585658515743e-05, 'epoch': 1.04}

 10%|█         | 635/6100 [57:28<8:13:08,  5.41s/it]
 10%|█         | 636/6100 [57:34<8:13:23,  5.42s/it]
 10%|█         | 637/6100 [57:39<8:13:22,  5.42s/it]
 10%|█         | 638/6100 [57:45<8:13:03,  5.42s/it]
 10%|█         | 639/6100 [57:50<8:12:46,  5.41s/it]
 10%|█         | 640/6100 [57:55<8:12:33,  5.41s/it]
                                                    
{'loss': 0.2017, 'learning_rate': 9.353939707280492e-05, 'epoch': 1.05}

 10%|█         | 640/6100 [57:55<8:12:33,  5.41s/it]
 11%|█         | 641/6100 [58:01<8:12:34,  5.41s/it]
 11%|█         | 642/6100 [58:06<8:12:27,  5.41s/it]
 11%|█         | 643/6100 [58:12<8:12:35,  5.42s/it]
 11%|█         | 644/6100 [58:17<8:12:42,  5.42s/it]
 11%|█         | 645/6100 [58:23<8:12:15,  5.41s/it]
                                                    
{'loss': 0.1931, 'learning_rate': 9.36520539679374e-05, 'epoch': 1.06}

 11%|█         | 645/6100 [58:23<8:12:15,  5.41s/it]
 11%|█         | 646/6100 [58:28<8:12:02,  5.41s/it]
 11%|█         | 647/6100 [58:33<8:11:47,  5.41s/it]
 11%|█         | 648/6100 [58:39<8:12:01,  5.41s/it]
 11%|█         | 649/6100 [58:44<8:12:06,  5.42s/it]
 11%|█         | 650/6100 [58:50<8:11:54,  5.42s/it]
                                                    
{'loss': 0.1967, 'learning_rate': 9.376384091697631e-05, 'epoch': 1.07}

 11%|█         | 650/6100 [58:50<8:11:54,  5.42s/it]
 11%|█         | 651/6100 [58:55<8:11:43,  5.41s/it]
 11%|█         | 652/6100 [59:00<8:11:49,  5.42s/it]
 11%|█         | 653/6100 [59:06<8:28:07,  5.60s/it]
 11%|█         | 654/6100 [59:12<8:23:10,  5.54s/it]
 11%|█         | 655/6100 [59:17<8:19:24,  5.50s/it]
                                                    
{'loss': 0.1944, 'learning_rate': 9.387477125262611e-05, 'epoch': 1.07}

 11%|█         | 655/6100 [59:17<8:19:24,  5.50s/it]
 11%|█         | 656/6100 [59:23<8:16:50,  5.48s/it]
 11%|█         | 657/6100 [59:28<8:14:55,  5.46s/it]
 11%|█         | 658/6100 [59:34<8:14:00,  5.45s/it]
 11%|█         | 659/6100 [59:39<8:12:44,  5.43s/it]
 11%|█         | 660/6100 [59:44<8:12:04,  5.43s/it]
                                                    
{'loss': 0.183, 'learning_rate': 9.398485800341712e-05, 'epoch': 1.08}

 11%|█         | 660/6100 [59:44<8:12:04,  5.43s/it]
 11%|█         | 661/6100 [59:50<8:11:20,  5.42s/it]
 11%|█         | 662/6100 [59:55<8:11:17,  5.42s/it]
 11%|█         | 663/6100 [1:00:01<8:11:08,  5.42s/it]
 11%|█         | 664/6100 [1:00:06<8:10:51,  5.42s/it]
 11%|█         | 665/6100 [1:00:11<8:10:33,  5.42s/it]
                                                      
{'loss': 0.1973, 'learning_rate': 9.40941139028884e-05, 'epoch': 1.09}

 11%|█         | 665/6100 [1:00:11<8:10:33,  5.42s/it]
 11%|█         | 666/6100 [1:00:17<8:10:47,  5.42s/it]
 11%|█         | 667/6100 [1:00:22<8:10:44,  5.42s/it]
 11%|█         | 668/6100 [1:00:28<8:10:21,  5.42s/it]
 11%|█         | 669/6100 [1:00:33<8:10:12,  5.42s/it]
 11%|█         | 670/6100 [1:00:39<8:10:34,  5.42s/it]
                                                      
{'loss': 0.1889, 'learning_rate': 9.420255139842665e-05, 'epoch': 1.1}

 11%|█         | 670/6100 [1:00:39<8:10:34,  5.42s/it]
 11%|█         | 671/6100 [1:00:44<8:10:11,  5.42s/it]
 11%|█         | 672/6100 [1:00:49<8:09:56,  5.42s/it]
 11%|█         | 673/6100 [1:00:55<8:09:49,  5.42s/it]
 11%|█         | 674/6100 [1:01:00<8:09:52,  5.42s/it]
 11%|█         | 675/6100 [1:01:06<8:09:32,  5.41s/it]
                                                      
{'loss': 0.1958, 'learning_rate': 9.431018265977657e-05, 'epoch': 1.11}

 11%|█         | 675/6100 [1:01:06<8:09:32,  5.41s/it]
 11%|█         | 676/6100 [1:01:11<8:09:20,  5.41s/it]
 11%|█         | 677/6100 [1:01:16<8:09:17,  5.41s/it]
 11%|█         | 678/6100 [1:01:22<8:09:27,  5.42s/it]
 11%|█         | 679/6100 [1:01:27<8:09:44,  5.42s/it]
 11%|█         | 680/6100 [1:01:33<8:16:11,  5.49s/it]
                                                      
{'loss': 0.2007, 'learning_rate': 9.441701958723697e-05, 'epoch': 1.11}

 11%|█         | 680/6100 [1:01:33<8:16:11,  5.49s/it]
 11%|█         | 681/6100 [1:01:38<8:13:56,  5.47s/it]
 11%|█         | 682/6100 [1:01:44<8:12:23,  5.45s/it]
 11%|█         | 683/6100 [1:01:49<8:11:00,  5.44s/it]
 11%|█         | 684/6100 [1:01:55<8:10:25,  5.43s/it]
 11%|█         | 685/6100 [1:02:00<8:09:36,  5.42s/it]
                                                      
{'loss': 0.1826, 'learning_rate': 9.452307381955704e-05, 'epoch': 1.12}

 11%|█         | 685/6100 [1:02:00<8:09:36,  5.42s/it]
 11%|█         | 686/6100 [1:02:05<8:09:24,  5.42s/it]
 11%|█▏        | 687/6100 [1:02:11<8:09:06,  5.42s/it]
 11%|█▏        | 688/6100 [1:02:16<8:08:43,  5.42s/it]
 11%|█▏        | 689/6100 [1:02:22<8:08:37,  5.42s/it]
 11%|█▏        | 690/6100 [1:02:27<8:08:49,  5.42s/it]
                                                      
{'loss': 0.1843, 'learning_rate': 9.462835674154493e-05, 'epoch': 1.13}

 11%|█▏        | 690/6100 [1:02:27<8:08:49,  5.42s/it]
 11%|█▏        | 691/6100 [1:02:33<8:08:39,  5.42s/it]
 11%|█▏        | 692/6100 [1:02:38<8:08:26,  5.42s/it]
 11%|█▏        | 693/6100 [1:02:43<8:08:04,  5.42s/it]
 11%|█▏        | 694/6100 [1:02:49<8:07:54,  5.42s/it]
 11%|█▏        | 695/6100 [1:02:54<8:07:46,  5.41s/it]
                                                      
{'loss': 0.1796, 'learning_rate': 9.473287949140227e-05, 'epoch': 1.14}

 11%|█▏        | 695/6100 [1:02:54<8:07:46,  5.41s/it]
 11%|█▏        | 696/6100 [1:03:00<8:07:53,  5.42s/it]
 11%|█▏        | 697/6100 [1:03:05<8:07:29,  5.41s/it]
 11%|█▏        | 698/6100 [1:03:10<8:07:32,  5.42s/it]
 11%|█▏        | 699/6100 [1:03:16<8:07:14,  5.41s/it]
 11%|█▏        | 700/6100 [1:03:21<8:07:36,  5.42s/it]
                                                      
{'loss': 0.195, 'learning_rate': 9.483665296779523e-05, 'epoch': 1.15}

 11%|█▏        | 700/6100 [1:03:21<8:07:36,  5.42s/it]
 11%|█▏        | 701/6100 [1:03:27<8:07:39,  5.42s/it]
 12%|█▏        | 702/6100 [1:03:32<8:08:02,  5.42s/it]
 12%|█▏        | 703/6100 [1:03:38<8:07:57,  5.42s/it]
 12%|█▏        | 704/6100 [1:03:43<8:07:20,  5.42s/it]
 12%|█▏        | 705/6100 [1:03:48<8:07:12,  5.42s/it]
                                                      
{'loss': 0.1829, 'learning_rate': 9.493968783667432e-05, 'epoch': 1.16}

 12%|█▏        | 705/6100 [1:03:48<8:07:12,  5.42s/it]
 12%|█▏        | 706/6100 [1:03:54<8:06:56,  5.42s/it]
 12%|█▏        | 707/6100 [1:03:59<8:06:55,  5.42s/it]
 12%|█▏        | 708/6100 [1:04:05<8:09:41,  5.45s/it]
 12%|█▏        | 709/6100 [1:04:10<8:08:38,  5.44s/it]
 12%|█▏        | 710/6100 [1:04:16<8:07:53,  5.43s/it]
                                                      
{'loss': 0.1909, 'learning_rate': 9.504199453785295e-05, 'epoch': 1.16}

 12%|█▏        | 710/6100 [1:04:16<8:07:53,  5.43s/it]
 12%|█▏        | 711/6100 [1:04:21<8:07:27,  5.43s/it]
 12%|█▏        | 712/6100 [1:04:26<8:07:27,  5.43s/it]
 12%|█▏        | 713/6100 [1:04:32<8:07:16,  5.43s/it]
 12%|█▏        | 714/6100 [1:04:37<8:07:11,  5.43s/it]
 12%|█▏        | 715/6100 [1:04:43<8:07:06,  5.43s/it]
                                                      
{'loss': 0.1878, 'learning_rate': 9.514358329135542e-05, 'epoch': 1.17}

 12%|█▏        | 715/6100 [1:04:43<8:07:06,  5.43s/it]
 12%|█▏        | 716/6100 [1:04:48<8:06:39,  5.42s/it]
 12%|█▏        | 717/6100 [1:04:54<8:06:24,  5.42s/it]
 12%|█▏        | 718/6100 [1:04:59<8:06:14,  5.42s/it]
 12%|█▏        | 719/6100 [1:05:04<8:06:30,  5.42s/it]
 12%|█▏        | 720/6100 [1:05:10<8:06:01,  5.42s/it]
                                                      
{'loss': 0.1896, 'learning_rate': 9.524446410354348e-05, 'epoch': 1.18}

 12%|█▏        | 720/6100 [1:05:10<8:06:01,  5.42s/it]
 12%|█▏        | 721/6100 [1:05:15<8:05:56,  5.42s/it]
 12%|█▏        | 722/6100 [1:05:21<8:05:17,  5.41s/it]
 12%|█▏        | 723/6100 [1:05:26<8:05:09,  5.41s/it]
 12%|█▏        | 724/6100 [1:05:31<8:05:04,  5.41s/it]
 12%|█▏        | 725/6100 [1:05:37<8:05:49,  5.42s/it]
                                                      
{'loss': 0.1773, 'learning_rate': 9.534464677303094e-05, 'epoch': 1.19}

 12%|█▏        | 725/6100 [1:05:37<8:05:49,  5.42s/it]
 12%|█▏        | 726/6100 [1:05:42<8:05:31,  5.42s/it]
 12%|█▏        | 727/6100 [1:05:48<8:05:10,  5.42s/it]
 12%|█▏        | 728/6100 [1:05:53<8:05:03,  5.42s/it]
 12%|█▏        | 729/6100 [1:05:59<8:04:42,  5.41s/it]
 12%|█▏        | 730/6100 [1:06:04<8:04:16,  5.41s/it]
                                                      
{'loss': 0.1766, 'learning_rate': 9.544414089639517e-05, 'epoch': 1.2}

 12%|█▏        | 730/6100 [1:06:04<8:04:16,  5.41s/it]
 12%|█▏        | 731/6100 [1:06:09<8:04:21,  5.41s/it]
 12%|█▏        | 732/6100 [1:06:15<8:04:21,  5.41s/it]
 12%|█▏        | 733/6100 [1:06:20<8:04:49,  5.42s/it]
 12%|█▏        | 734/6100 [1:06:26<8:04:44,  5.42s/it]
 12%|█▏        | 735/6100 [1:06:31<8:04:39,  5.42s/it]
                                                      
{'loss': 0.1827, 'learning_rate': 9.554295587369347e-05, 'epoch': 1.2}

 12%|█▏        | 735/6100 [1:06:31<8:04:39,  5.42s/it]
 12%|█▏        | 736/6100 [1:06:36<8:04:13,  5.42s/it]
 12%|█▏        | 737/6100 [1:06:42<8:04:08,  5.42s/it]
 12%|█▏        | 738/6100 [1:06:48<8:10:24,  5.49s/it]
 12%|█▏        | 739/6100 [1:06:53<8:08:28,  5.47s/it]
 12%|█▏        | 740/6100 [1:06:58<8:07:23,  5.46s/it]
                                                      
{'loss': 0.1762, 'learning_rate': 9.564110091379264e-05, 'epoch': 1.21}

 12%|█▏        | 740/6100 [1:06:58<8:07:23,  5.46s/it]
 12%|█▏        | 741/6100 [1:07:04<8:06:31,  5.45s/it]
 12%|█▏        | 742/6100 [1:07:09<8:06:04,  5.44s/it]
 12%|█▏        | 743/6100 [1:07:15<8:05:24,  5.44s/it]
 12%|█▏        | 744/6100 [1:07:20<8:04:43,  5.43s/it]
 12%|█▏        | 745/6100 [1:07:25<8:04:18,  5.43s/it]
                                                      
{'loss': 0.1873, 'learning_rate': 9.573858503951885e-05, 'epoch': 1.22}

 12%|█▏        | 745/6100 [1:07:25<8:04:18,  5.43s/it]
 12%|█▏        | 746/6100 [1:07:31<8:04:10,  5.43s/it]
 12%|█▏        | 747/6100 [1:07:36<8:03:59,  5.42s/it]
 12%|█▏        | 748/6100 [1:07:42<8:03:23,  5.42s/it]
 12%|█▏        | 749/6100 [1:07:47<8:03:04,  5.42s/it]
 12%|█▏        | 750/6100 [1:07:53<8:03:07,  5.42s/it]
                                                      
{'loss': 0.1729, 'learning_rate': 9.583541709263554e-05, 'epoch': 1.23}

 12%|█▏        | 750/6100 [1:07:53<8:03:07,  5.42s/it]Saving model checkpoint to ./results/checkpoint-750
Configuration saved in ./results/checkpoint-750/config.json
Model weights saved in ./results/checkpoint-750/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-750/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-750/special_tokens_map.json
[2023-02-21 21:00:08,732] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step750 is begin to save!
[2023-02-21 21:00:08,735] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-750/global_step750/mp_rank_00_model_states.pt
[2023-02-21 21:00:08,735] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-750/global_step750/mp_rank_00_model_states.pt...
[2023-02-21 21:00:09,552] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-750/global_step750/mp_rank_00_model_states.pt.
[2023-02-21 21:00:09,554] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-750/global_step750/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-21 21:00:09,730] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-750/global_step750/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-21 21:00:09,730] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-750/global_step750/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-21 21:00:09,730] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step750 is ready now!
Deleting older checkpoint [results/checkpoint-600] due to args.save_total_limit

 12%|█▏        | 751/6100 [1:08:00<8:59:54,  6.06s/it]
 12%|█▏        | 752/6100 [1:08:06<8:42:31,  5.86s/it]
 12%|█▏        | 753/6100 [1:08:11<8:30:16,  5.73s/it]
 12%|█▏        | 754/6100 [1:08:16<8:22:13,  5.64s/it]
 12%|█▏        | 755/6100 [1:08:22<8:16:18,  5.57s/it]
                                                      
{'loss': 0.1787, 'learning_rate': 9.593160573865574e-05, 'epoch': 1.24}

 12%|█▏        | 755/6100 [1:08:22<8:16:18,  5.57s/it]
 12%|█▏        | 756/6100 [1:08:27<8:12:20,  5.53s/it]
 12%|█▏        | 757/6100 [1:08:33<8:09:37,  5.50s/it]
 12%|█▏        | 758/6100 [1:08:38<8:07:06,  5.47s/it]
 12%|█▏        | 759/6100 [1:08:43<8:05:15,  5.45s/it]
 12%|█▏        | 760/6100 [1:08:49<8:03:52,  5.44s/it]
                                                      
{'loss': 0.1824, 'learning_rate': 9.602715947149564e-05, 'epoch': 1.25}

 12%|█▏        | 760/6100 [1:08:49<8:03:52,  5.44s/it]
 12%|█▏        | 761/6100 [1:08:54<8:03:54,  5.44s/it]
 12%|█▏        | 762/6100 [1:09:00<8:03:07,  5.43s/it]
 13%|█▎        | 763/6100 [1:09:05<8:02:45,  5.43s/it]
 13%|█▎        | 764/6100 [1:09:11<8:02:32,  5.43s/it]
 13%|█▎        | 765/6100 [1:09:16<8:02:19,  5.42s/it]
                                                      
{'loss': 0.1782, 'learning_rate': 9.612208661797555e-05, 'epoch': 1.25}

 13%|█▎        | 765/6100 [1:09:16<8:02:19,  5.42s/it]
 13%|█▎        | 766/6100 [1:09:21<8:02:05,  5.42s/it]
 13%|█▎        | 767/6100 [1:09:27<8:01:43,  5.42s/it]
 13%|█▎        | 768/6100 [1:09:32<8:01:44,  5.42s/it]
 13%|█▎        | 769/6100 [1:09:38<8:01:37,  5.42s/it]
 13%|█▎        | 770/6100 [1:09:43<8:01:43,  5.42s/it]
                                                      
{'loss': 0.1718, 'learning_rate': 9.621639534217434e-05, 'epoch': 1.26}

 13%|█▎        | 770/6100 [1:09:43<8:01:43,  5.42s/it]
 13%|█▎        | 771/6100 [1:09:49<8:03:43,  5.45s/it]
 13%|█▎        | 772/6100 [1:09:54<8:02:31,  5.43s/it]
 13%|█▎        | 773/6100 [1:09:59<8:02:07,  5.43s/it]
 13%|█▎        | 774/6100 [1:10:05<8:01:25,  5.42s/it]
 13%|█▎        | 775/6100 [1:10:10<8:01:21,  5.42s/it]
                                                      
{'loss': 0.1759, 'learning_rate': 9.631009364964285e-05, 'epoch': 1.27}

 13%|█▎        | 775/6100 [1:10:10<8:01:21,  5.42s/it]
 13%|█▎        | 776/6100 [1:10:16<8:01:07,  5.42s/it]
 13%|█▎        | 777/6100 [1:10:21<8:01:22,  5.43s/it]
 13%|█▎        | 778/6100 [1:10:26<8:00:47,  5.42s/it]
 13%|█▎        | 779/6100 [1:10:32<8:00:17,  5.42s/it]
 13%|█▎        | 780/6100 [1:10:37<7:59:55,  5.41s/it]
                                                      
{'loss': 0.189, 'learning_rate': 9.640318939148179e-05, 'epoch': 1.28}

 13%|█▎        | 780/6100 [1:10:37<7:59:55,  5.41s/it]
 13%|█▎        | 781/6100 [1:10:43<7:59:52,  5.41s/it]
 13%|█▎        | 782/6100 [1:10:48<7:59:49,  5.41s/it]
 13%|█▎        | 783/6100 [1:10:54<7:59:37,  5.41s/it]
 13%|█▎        | 784/6100 [1:10:59<7:59:43,  5.41s/it]
 13%|█▎        | 785/6100 [1:11:04<7:59:39,  5.41s/it]
                                                      
{'loss': 0.1809, 'learning_rate': 9.64956902682895e-05, 'epoch': 1.29}

 13%|█▎        | 785/6100 [1:11:04<7:59:39,  5.41s/it]
 13%|█▎        | 786/6100 [1:11:10<7:59:37,  5.42s/it]
 13%|█▎        | 787/6100 [1:11:15<7:59:24,  5.41s/it]
 13%|█▎        | 788/6100 [1:11:21<7:59:11,  5.41s/it]
 13%|█▎        | 789/6100 [1:11:26<8:00:15,  5.43s/it]
 13%|█▎        | 790/6100 [1:11:31<8:00:07,  5.43s/it]
                                                      
{'loss': 0.1762, 'learning_rate': 9.65876038339843e-05, 'epoch': 1.29}

 13%|█▎        | 790/6100 [1:11:31<8:00:07,  5.43s/it]
 13%|█▎        | 791/6100 [1:11:37<7:59:46,  5.42s/it]
 13%|█▎        | 792/6100 [1:11:42<7:59:16,  5.42s/it]
 13%|█▎        | 793/6100 [1:11:48<7:59:03,  5.42s/it]
 13%|█▎        | 794/6100 [1:11:53<7:58:50,  5.41s/it]
 13%|█▎        | 795/6100 [1:11:59<7:58:56,  5.42s/it]
                                                      
{'loss': 0.1679, 'learning_rate': 9.667893749950614e-05, 'epoch': 1.3}

 13%|█▎        | 795/6100 [1:11:59<7:58:56,  5.42s/it]
 13%|█▎        | 796/6100 [1:12:04<7:58:50,  5.42s/it]
 13%|█▎        | 797/6100 [1:12:09<7:58:34,  5.41s/it]
 13%|█▎        | 798/6100 [1:12:15<7:58:38,  5.42s/it]
 13%|█▎        | 799/6100 [1:12:20<7:58:46,  5.42s/it]
 13%|█▎        | 800/6100 [1:12:26<7:58:13,  5.41s/it]
                                                      
{'loss': 0.1715, 'learning_rate': 9.676969853640246e-05, 'epoch': 1.31}

 13%|█▎        | 800/6100 [1:12:26<7:58:13,  5.41s/it]
 13%|█▎        | 801/6100 [1:12:31<7:58:14,  5.41s/it]
 13%|█▎        | 802/6100 [1:12:36<7:57:55,  5.41s/it]
 13%|█▎        | 803/6100 [1:12:42<7:58:04,  5.42s/it]
 13%|█▎        | 804/6100 [1:12:47<7:57:56,  5.41s/it]
 13%|█▎        | 805/6100 [1:12:53<7:57:49,  5.41s/it]
                                                      
{'loss': 0.1726, 'learning_rate': 9.685989408030217e-05, 'epoch': 1.32}

 13%|█▎        | 805/6100 [1:12:53<7:57:49,  5.41s/it]
 13%|█▎        | 806/6100 [1:12:58<7:58:14,  5.42s/it]
 13%|█▎        | 807/6100 [1:13:04<7:57:54,  5.42s/it]
 13%|█▎        | 808/6100 [1:13:09<7:57:49,  5.42s/it]
 13%|█▎        | 809/6100 [1:13:14<7:58:09,  5.42s/it]
 13%|█▎        | 810/6100 [1:13:20<7:58:07,  5.42s/it]
                                                      
{'loss': 0.1646, 'learning_rate': 9.694953113428203e-05, 'epoch': 1.33}

 13%|█▎        | 810/6100 [1:13:20<7:58:07,  5.42s/it]
 13%|█▎        | 811/6100 [1:13:25<7:57:55,  5.42s/it]
 13%|█▎        | 812/6100 [1:13:31<7:57:46,  5.42s/it]
 13%|█▎        | 813/6100 [1:13:36<7:59:00,  5.44s/it]
 13%|█▎        | 814/6100 [1:13:42<7:58:57,  5.44s/it]
 13%|█▎        | 815/6100 [1:13:47<7:58:19,  5.43s/it]
                                                      
{'loss': 0.1722, 'learning_rate': 9.703861657212966e-05, 'epoch': 1.34}

 13%|█▎        | 815/6100 [1:13:47<7:58:19,  5.43s/it]
 13%|█▎        | 816/6100 [1:13:52<7:57:39,  5.42s/it]
 13%|█▎        | 817/6100 [1:13:58<7:57:08,  5.42s/it]
 13%|█▎        | 818/6100 [1:14:03<7:56:51,  5.42s/it]
 13%|█▎        | 819/6100 [1:14:09<7:56:41,  5.42s/it]
 13%|█▎        | 820/6100 [1:14:14<7:58:42,  5.44s/it]
                                                      
{'loss': 0.1806, 'learning_rate': 9.712715714150643e-05, 'epoch': 1.34}

 13%|█▎        | 820/6100 [1:14:14<7:58:42,  5.44s/it]
 13%|█▎        | 821/6100 [1:14:20<7:58:03,  5.43s/it]
 13%|█▎        | 822/6100 [1:14:25<7:57:21,  5.43s/it]
 13%|█▎        | 823/6100 [1:14:30<7:56:41,  5.42s/it]
 14%|█▎        | 824/6100 [1:14:36<7:56:07,  5.41s/it]
 14%|█▎        | 825/6100 [1:14:41<7:56:09,  5.42s/it]
                                                      
{'loss': 0.1812, 'learning_rate': 9.721515946701466e-05, 'epoch': 1.35}

 14%|█▎        | 825/6100 [1:14:41<7:56:09,  5.42s/it]
 14%|█▎        | 826/6100 [1:14:47<7:56:58,  5.43s/it]
 14%|█▎        | 827/6100 [1:14:52<7:56:30,  5.42s/it]
 14%|█▎        | 828/6100 [1:14:57<7:57:12,  5.43s/it]
 14%|█▎        | 829/6100 [1:15:03<7:56:42,  5.43s/it]
 14%|█▎        | 830/6100 [1:15:08<7:56:33,  5.43s/it]
                                                      
{'loss': 0.1622, 'learning_rate': 9.730263005317167e-05, 'epoch': 1.36}

 14%|█▎        | 830/6100 [1:15:08<7:56:33,  5.43s/it]
 14%|█▎        | 831/6100 [1:15:14<7:56:00,  5.42s/it]
 14%|█▎        | 832/6100 [1:15:19<7:55:55,  5.42s/it]
 14%|█▎        | 833/6100 [1:15:25<7:55:56,  5.42s/it]
 14%|█▎        | 834/6100 [1:15:30<7:55:59,  5.42s/it]
 14%|█▎        | 835/6100 [1:15:35<7:55:33,  5.42s/it]
                                                      
{'loss': 0.1634, 'learning_rate': 9.738957528729492e-05, 'epoch': 1.37}

 14%|█▎        | 835/6100 [1:15:35<7:55:33,  5.42s/it]
 14%|█▎        | 836/6100 [1:15:41<7:55:39,  5.42s/it]
 14%|█▎        | 837/6100 [1:15:46<7:55:27,  5.42s/it]
 14%|█▎        | 838/6100 [1:15:52<7:55:02,  5.42s/it]
 14%|█▍        | 839/6100 [1:15:57<7:54:43,  5.41s/it]
 14%|█▍        | 840/6100 [1:16:02<7:54:25,  5.41s/it]
                                                      
{'loss': 0.1758, 'learning_rate': 9.747600144230072e-05, 'epoch': 1.38}

 14%|█▍        | 840/6100 [1:16:02<7:54:25,  5.41s/it]
 14%|█▍        | 841/6100 [1:16:08<7:54:19,  5.41s/it]
 14%|█▍        | 842/6100 [1:16:13<7:54:22,  5.41s/it]
 14%|█▍        | 843/6100 [1:16:19<7:54:16,  5.41s/it]
 14%|█▍        | 844/6100 [1:16:24<7:54:05,  5.41s/it]
 14%|█▍        | 845/6100 [1:16:30<7:53:48,  5.41s/it]
                                                      
{'loss': 0.1779, 'learning_rate': 9.75619146794201e-05, 'epoch': 1.38}

 14%|█▍        | 845/6100 [1:16:30<7:53:48,  5.41s/it]
 14%|█▍        | 846/6100 [1:16:35<7:54:03,  5.41s/it]
 14%|█▍        | 847/6100 [1:16:40<7:53:34,  5.41s/it]
 14%|█▍        | 848/6100 [1:16:46<7:53:41,  5.41s/it]
 14%|█▍        | 849/6100 [1:16:51<7:53:57,  5.42s/it]
 14%|█▍        | 850/6100 [1:16:57<7:53:37,  5.41s/it]
                                                      
{'loss': 0.1619, 'learning_rate': 9.764732105083454e-05, 'epoch': 1.39}

 14%|█▍        | 850/6100 [1:16:57<7:53:37,  5.41s/it]
 14%|█▍        | 851/6100 [1:17:02<7:53:46,  5.42s/it]
 14%|█▍        | 852/6100 [1:17:07<7:53:46,  5.42s/it]
 14%|█▍        | 853/6100 [1:17:13<7:53:44,  5.42s/it]
 14%|█▍        | 854/6100 [1:17:18<7:53:17,  5.41s/it]
 14%|█▍        | 855/6100 [1:17:24<7:53:18,  5.41s/it]
                                                      
{'loss': 0.1679, 'learning_rate': 9.773222650223418e-05, 'epoch': 1.4}

 14%|█▍        | 855/6100 [1:17:24<7:53:18,  5.41s/it]
 14%|█▍        | 856/6100 [1:17:29<7:53:35,  5.42s/it]
 14%|█▍        | 857/6100 [1:17:35<7:53:35,  5.42s/it]
 14%|█▍        | 858/6100 [1:17:40<7:53:28,  5.42s/it]
 14%|█▍        | 859/6100 [1:17:45<7:53:04,  5.42s/it]
 14%|█▍        | 860/6100 [1:17:51<7:52:54,  5.42s/it]
                                                      
{'loss': 0.1697, 'learning_rate': 9.781663687530184e-05, 'epoch': 1.41}

 14%|█▍        | 860/6100 [1:17:51<7:52:54,  5.42s/it]
 14%|█▍        | 861/6100 [1:17:56<7:52:49,  5.42s/it]
 14%|█▍        | 862/6100 [1:18:02<7:52:54,  5.42s/it]
 14%|█▍        | 863/6100 [1:18:07<7:52:49,  5.42s/it]
 14%|█▍        | 864/6100 [1:18:12<7:53:00,  5.42s/it]
 14%|█▍        | 865/6100 [1:18:18<7:52:40,  5.42s/it]
                                                      
{'loss': 0.171, 'learning_rate': 9.790055791012466e-05, 'epoch': 1.42}

 14%|█▍        | 865/6100 [1:18:18<7:52:40,  5.42s/it]
 14%|█▍        | 866/6100 [1:18:23<7:52:40,  5.42s/it]
 14%|█▍        | 867/6100 [1:18:29<7:52:18,  5.42s/it]
 14%|█▍        | 868/6100 [1:18:34<7:52:23,  5.42s/it]
 14%|█▍        | 869/6100 [1:18:40<7:52:20,  5.42s/it]
 14%|█▍        | 870/6100 [1:18:45<7:51:56,  5.41s/it]
                                                      
{'loss': 0.1725, 'learning_rate': 9.798399524753642e-05, 'epoch': 1.43}

 14%|█▍        | 870/6100 [1:18:45<7:51:56,  5.41s/it]
 14%|█▍        | 871/6100 [1:18:50<7:51:37,  5.41s/it]
 14%|█▍        | 872/6100 [1:18:56<7:51:54,  5.42s/it]
 14%|█▍        | 873/6100 [1:19:01<7:51:46,  5.42s/it]
 14%|█▍        | 874/6100 [1:19:07<7:51:34,  5.41s/it]
 14%|█▍        | 875/6100 [1:19:12<7:51:09,  5.41s/it]
                                                      
{'loss': 0.1615, 'learning_rate': 9.806695443139277e-05, 'epoch': 1.43}

 14%|█▍        | 875/6100 [1:19:12<7:51:09,  5.41s/it]
 14%|█▍        | 876/6100 [1:19:17<7:50:45,  5.41s/it]
 14%|█▍        | 877/6100 [1:19:23<7:50:47,  5.41s/it]
 14%|█▍        | 878/6100 [1:19:28<7:50:50,  5.41s/it]
 14%|█▍        | 879/6100 [1:19:34<7:51:10,  5.41s/it]
 14%|█▍        | 880/6100 [1:19:39<7:51:38,  5.42s/it]
                                                      
{'loss': 0.162, 'learning_rate': 9.814944091078157e-05, 'epoch': 1.44}

 14%|█▍        | 880/6100 [1:19:39<7:51:38,  5.42s/it]
 14%|█▍        | 881/6100 [1:19:45<7:51:35,  5.42s/it]
 14%|█▍        | 882/6100 [1:19:50<7:51:35,  5.42s/it]
 14%|█▍        | 883/6100 [1:19:55<7:51:14,  5.42s/it]
 14%|█▍        | 884/6100 [1:20:01<7:50:45,  5.42s/it]
 15%|█▍        | 885/6100 [1:20:06<7:55:29,  5.47s/it]
                                                      
{'loss': 0.1619, 'learning_rate': 9.823146004217062e-05, 'epoch': 1.45}

 15%|█▍        | 885/6100 [1:20:06<7:55:29,  5.47s/it]
 15%|█▍        | 886/6100 [1:20:12<7:54:37,  5.46s/it]
 15%|█▍        | 887/6100 [1:20:17<7:53:18,  5.45s/it]
 15%|█▍        | 888/6100 [1:20:23<7:52:24,  5.44s/it]
 15%|█▍        | 889/6100 [1:20:28<7:51:48,  5.43s/it]
 15%|█▍        | 890/6100 [1:20:33<7:50:59,  5.42s/it]
                                                      
{'loss': 0.1497, 'learning_rate': 9.831301709149488e-05, 'epoch': 1.46}

 15%|█▍        | 890/6100 [1:20:33<7:50:59,  5.42s/it]
 15%|█▍        | 891/6100 [1:20:39<7:50:46,  5.42s/it]
 15%|█▍        | 892/6100 [1:20:44<7:50:40,  5.42s/it]
 15%|█▍        | 893/6100 [1:20:50<7:49:58,  5.42s/it]
 15%|█▍        | 894/6100 [1:20:55<7:49:45,  5.41s/it]
 15%|█▍        | 895/6100 [1:21:01<7:49:36,  5.41s/it]
                                                      
{'loss': 0.1643, 'learning_rate': 9.83941172361853e-05, 'epoch': 1.47}

 15%|█▍        | 895/6100 [1:21:01<7:49:36,  5.41s/it]
 15%|█▍        | 896/6100 [1:21:06<7:49:56,  5.42s/it]
 15%|█▍        | 897/6100 [1:21:11<7:49:55,  5.42s/it]
 15%|█▍        | 898/6100 [1:21:17<7:50:17,  5.42s/it]
 15%|█▍        | 899/6100 [1:21:22<7:50:14,  5.42s/it]
 15%|█▍        | 900/6100 [1:21:28<7:49:45,  5.42s/it]
                                                      
{'loss': 0.1579, 'learning_rate': 9.847476556714102e-05, 'epoch': 1.48}

 15%|█▍        | 900/6100 [1:21:28<7:49:45,  5.42s/it]Saving model checkpoint to ./results/checkpoint-900
Configuration saved in ./results/checkpoint-900/config.json
Model weights saved in ./results/checkpoint-900/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-900/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-900/special_tokens_map.json
[2023-02-21 21:13:43,811] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step900 is begin to save!
[2023-02-21 21:13:43,814] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-900/global_step900/mp_rank_00_model_states.pt
[2023-02-21 21:13:43,814] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-900/global_step900/mp_rank_00_model_states.pt...
[2023-02-21 21:13:44,630] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-900/global_step900/mp_rank_00_model_states.pt.
[2023-02-21 21:13:44,632] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-900/global_step900/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-21 21:13:44,808] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-900/global_step900/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-21 21:13:44,808] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-900/global_step900/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-21 21:13:44,809] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step900 is ready now!
Deleting older checkpoint [results/checkpoint-750] due to args.save_total_limit

 15%|█▍        | 901/6100 [1:21:35<8:44:36,  6.05s/it]
 15%|█▍        | 902/6100 [1:21:41<8:27:22,  5.86s/it]
 15%|█▍        | 903/6100 [1:21:46<8:16:38,  5.73s/it]
 15%|█▍        | 904/6100 [1:21:51<8:08:10,  5.64s/it]
 15%|█▍        | 905/6100 [1:21:57<8:02:04,  5.57s/it]
                                                      
{'loss': 0.1618, 'learning_rate': 9.855496709064704e-05, 'epoch': 1.48}

 15%|█▍        | 905/6100 [1:21:57<8:02:04,  5.57s/it]
 15%|█▍        | 906/6100 [1:22:02<7:57:47,  5.52s/it]
 15%|█▍        | 907/6100 [1:22:08<7:55:05,  5.49s/it]
 15%|█▍        | 908/6100 [1:22:13<7:53:25,  5.47s/it]
 15%|█▍        | 909/6100 [1:22:19<7:52:00,  5.46s/it]
 15%|█▍        | 910/6100 [1:22:24<7:50:56,  5.44s/it]
                                                      
{'loss': 0.1564, 'learning_rate': 9.863472673023902e-05, 'epoch': 1.49}

 15%|█▍        | 910/6100 [1:22:24<7:50:56,  5.44s/it]
 15%|█▍        | 911/6100 [1:22:29<7:49:49,  5.43s/it]
 15%|█▍        | 912/6100 [1:22:35<7:49:22,  5.43s/it]
 15%|█▍        | 913/6100 [1:22:40<7:49:11,  5.43s/it]
 15%|█▍        | 914/6100 [1:22:46<7:48:36,  5.42s/it]
 15%|█▌        | 915/6100 [1:22:51<7:48:14,  5.42s/it]
                                                      
{'loss': 0.1603, 'learning_rate': 9.871404932851692e-05, 'epoch': 1.5}

 15%|█▌        | 915/6100 [1:22:51<7:48:14,  5.42s/it]
 15%|█▌        | 916/6100 [1:22:56<7:47:39,  5.41s/it]
 15%|█▌        | 917/6100 [1:23:02<7:47:38,  5.41s/it]
 15%|█▌        | 918/6100 [1:23:07<7:47:27,  5.41s/it]
 15%|█▌        | 919/6100 [1:23:13<7:47:39,  5.42s/it]
 15%|█▌        | 920/6100 [1:23:18<7:47:57,  5.42s/it]
                                                      
{'loss': 0.1555, 'learning_rate': 9.879293964890941e-05, 'epoch': 1.51}

 15%|█▌        | 920/6100 [1:23:18<7:47:57,  5.42s/it]
 15%|█▌        | 921/6100 [1:23:23<7:47:35,  5.42s/it]
 15%|█▌        | 922/6100 [1:23:29<7:47:42,  5.42s/it]
 15%|█▌        | 923/6100 [1:23:34<7:47:20,  5.42s/it]
 15%|█▌        | 924/6100 [1:23:40<7:46:47,  5.41s/it]
 15%|█▌        | 925/6100 [1:23:45<7:47:00,  5.41s/it]
                                                      
{'loss': 0.1562, 'learning_rate': 9.887140237739018e-05, 'epoch': 1.52}

 15%|█▌        | 925/6100 [1:23:45<7:47:00,  5.41s/it]
 15%|█▌        | 926/6100 [1:23:51<7:47:16,  5.42s/it]
 15%|█▌        | 927/6100 [1:23:56<7:46:50,  5.41s/it]
 15%|█▌        | 928/6100 [1:24:01<7:47:06,  5.42s/it]
 15%|█▌        | 929/6100 [1:24:07<7:46:33,  5.41s/it]
 15%|█▌        | 930/6100 [1:24:12<7:46:25,  5.41s/it]
                                                      
{'loss': 0.1639, 'learning_rate': 9.894944212414832e-05, 'epoch': 1.52}

 15%|█▌        | 930/6100 [1:24:12<7:46:25,  5.41s/it]
 15%|█▌        | 931/6100 [1:24:18<7:46:04,  5.41s/it]
 15%|█▌        | 932/6100 [1:24:23<7:45:43,  5.41s/it]
 15%|█▌        | 933/6100 [1:24:28<7:45:23,  5.40s/it]
 15%|█▌        | 934/6100 [1:24:34<7:45:20,  5.40s/it]
 15%|█▌        | 935/6100 [1:24:39<7:45:29,  5.41s/it]
                                                      
{'loss': 0.165, 'learning_rate': 9.902706342521365e-05, 'epoch': 1.53}

 15%|█▌        | 935/6100 [1:24:39<7:45:29,  5.41s/it]
 15%|█▌        | 936/6100 [1:24:45<7:45:21,  5.41s/it]
 15%|█▌        | 937/6100 [1:24:50<7:45:16,  5.41s/it]
 15%|█▌        | 938/6100 [1:24:55<7:45:17,  5.41s/it]
 15%|█▌        | 939/6100 [1:25:01<7:45:31,  5.41s/it]
 15%|█▌        | 940/6100 [1:25:06<7:45:47,  5.42s/it]
                                                      
{'loss': 0.1476, 'learning_rate': 9.910427074403877e-05, 'epoch': 1.54}

 15%|█▌        | 940/6100 [1:25:06<7:45:47,  5.42s/it]
 15%|█▌        | 941/6100 [1:25:12<7:45:47,  5.42s/it]
 15%|█▌        | 942/6100 [1:25:17<7:45:29,  5.41s/it]
 15%|█▌        | 943/6100 [1:25:23<7:45:15,  5.41s/it]
 15%|█▌        | 944/6100 [1:25:28<7:45:13,  5.41s/it]
 15%|█▌        | 945/6100 [1:25:33<7:45:10,  5.41s/it]
                                                      
{'loss': 0.1645, 'learning_rate': 9.918106847303928e-05, 'epoch': 1.55}

 15%|█▌        | 945/6100 [1:25:33<7:45:10,  5.41s/it]
 16%|█▌        | 946/6100 [1:25:39<7:45:10,  5.42s/it]
 16%|█▌        | 947/6100 [1:25:44<7:44:58,  5.41s/it]
 16%|█▌        | 948/6100 [1:25:50<7:44:26,  5.41s/it]
 16%|█▌        | 949/6100 [1:25:55<7:44:34,  5.41s/it]
 16%|█▌        | 950/6100 [1:26:00<7:44:39,  5.41s/it]
                                                      
{'loss': 0.165, 'learning_rate': 9.925746093509318e-05, 'epoch': 1.56}

 16%|█▌        | 950/6100 [1:26:00<7:44:39,  5.41s/it]
 16%|█▌        | 951/6100 [1:26:06<7:44:52,  5.42s/it]
 16%|█▌        | 952/6100 [1:26:11<7:44:40,  5.42s/it]
 16%|█▌        | 953/6100 [1:26:17<7:44:30,  5.41s/it]
 16%|█▌        | 954/6100 [1:26:22<7:44:04,  5.41s/it]
 16%|█▌        | 955/6100 [1:26:28<7:44:10,  5.41s/it]
                                                      
{'loss': 0.1445, 'learning_rate': 9.933345238500102e-05, 'epoch': 1.57}

 16%|█▌        | 955/6100 [1:26:28<7:44:10,  5.41s/it]
 16%|█▌        | 956/6100 [1:26:33<7:44:24,  5.42s/it]
 16%|█▌        | 957/6100 [1:26:38<7:44:15,  5.42s/it]
 16%|█▌        | 958/6100 [1:26:44<7:43:55,  5.41s/it]
 16%|█▌        | 959/6100 [1:26:49<7:43:58,  5.42s/it]
 16%|█▌        | 960/6100 [1:26:55<7:44:33,  5.42s/it]
                                                      
{'loss': 0.1547, 'learning_rate': 9.940904701090794e-05, 'epoch': 1.57}

 16%|█▌        | 960/6100 [1:26:55<7:44:33,  5.42s/it]
 16%|█▌        | 961/6100 [1:27:00<7:44:16,  5.42s/it]
 16%|█▌        | 962/6100 [1:27:05<7:44:26,  5.42s/it]
 16%|█▌        | 963/6100 [1:27:11<7:44:05,  5.42s/it]
 16%|█▌        | 964/6100 [1:27:16<7:43:33,  5.42s/it]
 16%|█▌        | 965/6100 [1:27:22<7:43:41,  5.42s/it]
                                                      
{'loss': 0.1629, 'learning_rate': 9.948424893568864e-05, 'epoch': 1.58}

 16%|█▌        | 965/6100 [1:27:22<7:43:41,  5.42s/it]
 16%|█▌        | 966/6100 [1:27:27<7:43:30,  5.42s/it]
 16%|█▌        | 967/6100 [1:27:33<7:43:46,  5.42s/it]
 16%|█▌        | 968/6100 [1:27:38<7:43:41,  5.42s/it]
 16%|█▌        | 969/6100 [1:27:43<7:43:10,  5.42s/it]
 16%|█▌        | 970/6100 [1:27:49<7:42:40,  5.41s/it]
                                                      
{'loss': 0.1659, 'learning_rate': 9.955906221829676e-05, 'epoch': 1.59}

 16%|█▌        | 970/6100 [1:27:49<7:42:40,  5.41s/it]
 16%|█▌        | 971/6100 [1:27:54<7:42:48,  5.41s/it]
 16%|█▌        | 972/6100 [1:28:00<7:42:32,  5.41s/it]
 16%|█▌        | 973/6100 [1:28:05<7:42:15,  5.41s/it]
 16%|█▌        | 974/6100 [1:28:10<7:42:11,  5.41s/it]
 16%|█▌        | 975/6100 [1:28:16<7:42:24,  5.41s/it]
                                                      
{'loss': 0.1513, 'learning_rate': 9.963349085507933e-05, 'epoch': 1.6}

 16%|█▌        | 975/6100 [1:28:16<7:42:24,  5.41s/it]
 16%|█▌        | 976/6100 [1:28:21<7:42:05,  5.41s/it]
 16%|█▌        | 977/6100 [1:28:27<7:42:20,  5.41s/it]
 16%|█▌        | 978/6100 [1:28:32<7:42:39,  5.42s/it]
 16%|█▌        | 979/6100 [1:28:38<7:42:08,  5.41s/it]
 16%|█▌        | 980/6100 [1:28:43<7:41:48,  5.41s/it]
                                                      
{'loss': 0.1613, 'learning_rate': 9.970753878105793e-05, 'epoch': 1.61}

 16%|█▌        | 980/6100 [1:28:43<7:41:48,  5.41s/it]
 16%|█▌        | 981/6100 [1:28:48<7:41:52,  5.41s/it]
 16%|█▌        | 982/6100 [1:28:54<7:41:33,  5.41s/it]
 16%|█▌        | 983/6100 [1:28:59<7:41:36,  5.41s/it]
 16%|█▌        | 984/6100 [1:29:05<7:41:44,  5.42s/it]
 16%|█▌        | 985/6100 [1:29:10<7:41:35,  5.41s/it]
                                                      
{'loss': 0.1448, 'learning_rate': 9.97812098711769e-05, 'epoch': 1.61}

 16%|█▌        | 985/6100 [1:29:10<7:41:35,  5.41s/it]
 16%|█▌        | 986/6100 [1:29:15<7:41:23,  5.41s/it]
 16%|█▌        | 987/6100 [1:29:21<7:41:24,  5.41s/it]
 16%|█▌        | 988/6100 [1:29:26<7:41:15,  5.41s/it]
 16%|█▌        | 989/6100 [1:29:32<7:41:06,  5.41s/it]
 16%|█▌        | 990/6100 [1:29:38<7:53:00,  5.55s/it]
                                                      
{'loss': 0.153, 'learning_rate': 9.985450794152014e-05, 'epoch': 1.62}

 16%|█▌        | 990/6100 [1:29:38<7:53:00,  5.55s/it]
 16%|█▌        | 991/6100 [1:29:43<7:49:28,  5.51s/it]
 16%|█▋        | 992/6100 [1:29:48<7:46:54,  5.48s/it]
 16%|█▋        | 993/6100 [1:29:54<7:45:04,  5.46s/it]
 16%|█▋        | 994/6100 [1:29:59<7:44:01,  5.45s/it]
 16%|█▋        | 995/6100 [1:30:05<7:42:59,  5.44s/it]
                                                      
{'loss': 0.1487, 'learning_rate': 9.992743675049728e-05, 'epoch': 1.63}

 16%|█▋        | 995/6100 [1:30:05<7:42:59,  5.44s/it]
 16%|█▋        | 996/6100 [1:30:10<7:42:19,  5.43s/it]
 16%|█▋        | 997/6100 [1:30:15<7:41:12,  5.42s/it]
 16%|█▋        | 998/6100 [1:30:21<7:41:08,  5.42s/it]
 16%|█▋        | 999/6100 [1:30:26<7:40:55,  5.42s/it]
 16%|█▋        | 1000/6100 [1:30:32<7:40:36,  5.42s/it]
                                                       
{'loss': 0.1479, 'learning_rate': 0.0001, 'epoch': 1.64}

 16%|█▋        | 1000/6100 [1:30:32<7:40:36,  5.42s/it]
 16%|█▋        | 1001/6100 [1:30:37<7:40:27,  5.42s/it]
 16%|█▋        | 1002/6100 [1:30:42<7:39:47,  5.41s/it]
 16%|█▋        | 1003/6100 [1:30:48<7:39:33,  5.41s/it]
 16%|█▋        | 1004/6100 [1:30:53<7:39:30,  5.41s/it]
 16%|█▋        | 1005/6100 [1:30:59<7:39:31,  5.41s/it]
                                                       
{'loss': 0.145, 'learning_rate': 0.0001, 'epoch': 1.65}

 16%|█▋        | 1005/6100 [1:30:59<7:39:31,  5.41s/it]
 16%|█▋        | 1006/6100 [1:31:04<7:39:44,  5.42s/it]
 17%|█▋        | 1007/6100 [1:31:10<7:39:30,  5.41s/it]
 17%|█▋        | 1008/6100 [1:31:15<7:39:28,  5.41s/it]
 17%|█▋        | 1009/6100 [1:31:20<7:39:03,  5.41s/it]
 17%|█▋        | 1010/6100 [1:31:26<7:39:11,  5.41s/it]
                                                       
{'loss': 0.1507, 'learning_rate': 0.0001, 'epoch': 1.66}

 17%|█▋        | 1010/6100 [1:31:26<7:39:11,  5.41s/it]
 17%|█▋        | 1011/6100 [1:31:31<7:39:10,  5.41s/it]
 17%|█▋        | 1012/6100 [1:31:37<7:39:02,  5.41s/it]
 17%|█▋        | 1013/6100 [1:31:42<7:39:00,  5.41s/it]
 17%|█▋        | 1014/6100 [1:31:47<7:38:43,  5.41s/it]
 17%|█▋        | 1015/6100 [1:31:53<7:38:45,  5.41s/it]
                                                       
{'loss': 0.1474, 'learning_rate': 0.0001, 'epoch': 1.66}

 17%|█▋        | 1015/6100 [1:31:53<7:38:45,  5.41s/it]
 17%|█▋        | 1016/6100 [1:31:58<7:38:35,  5.41s/it]
 17%|█▋        | 1017/6100 [1:32:04<7:38:57,  5.42s/it]
 17%|█▋        | 1018/6100 [1:32:09<7:38:51,  5.42s/it]
 17%|█▋        | 1019/6100 [1:32:15<7:39:02,  5.42s/it]
 17%|█▋        | 1020/6100 [1:32:20<7:38:46,  5.42s/it]
                                                       
{'loss': 0.1478, 'learning_rate': 0.0001, 'epoch': 1.67}

 17%|█▋        | 1020/6100 [1:32:20<7:38:46,  5.42s/it]
 17%|█▋        | 1021/6100 [1:32:25<7:38:36,  5.42s/it]
 17%|█▋        | 1022/6100 [1:32:31<7:38:20,  5.42s/it]
 17%|█▋        | 1023/6100 [1:32:36<7:38:18,  5.42s/it]
 17%|█▋        | 1024/6100 [1:32:42<7:38:18,  5.42s/it]
 17%|█▋        | 1025/6100 [1:32:47<7:38:10,  5.42s/it]
                                                       
{'loss': 0.1575, 'learning_rate': 0.0001, 'epoch': 1.68}

 17%|█▋        | 1025/6100 [1:32:47<7:38:10,  5.42s/it]
 17%|█▋        | 1026/6100 [1:32:52<7:38:00,  5.42s/it]
 17%|█▋        | 1027/6100 [1:32:58<7:38:23,  5.42s/it]
 17%|█▋        | 1028/6100 [1:33:03<7:37:56,  5.42s/it]
 17%|█▋        | 1029/6100 [1:33:09<7:37:53,  5.42s/it]
 17%|█▋        | 1030/6100 [1:33:14<7:37:31,  5.41s/it]
                                                       
{'loss': 0.1535, 'learning_rate': 0.0001, 'epoch': 1.69}

 17%|█▋        | 1030/6100 [1:33:14<7:37:31,  5.41s/it]
 17%|█▋        | 1031/6100 [1:33:20<7:37:13,  5.41s/it]
 17%|█▋        | 1032/6100 [1:33:25<7:37:08,  5.41s/it]
 17%|█▋        | 1033/6100 [1:33:30<7:37:17,  5.41s/it]
 17%|█▋        | 1034/6100 [1:33:36<7:36:58,  5.41s/it]
 17%|█▋        | 1035/6100 [1:33:41<7:37:10,  5.42s/it]
                                                       
{'loss': 0.1467, 'learning_rate': 0.0001, 'epoch': 1.7}

 17%|█▋        | 1035/6100 [1:33:41<7:37:10,  5.42s/it]
 17%|█▋        | 1036/6100 [1:33:47<7:37:08,  5.42s/it]
 17%|█▋        | 1037/6100 [1:33:52<7:37:26,  5.42s/it]
 17%|█▋        | 1038/6100 [1:33:57<7:37:00,  5.42s/it]
 17%|█▋        | 1039/6100 [1:34:03<7:36:57,  5.42s/it]
 17%|█▋        | 1040/6100 [1:34:08<7:37:20,  5.42s/it]
                                                       
{'loss': 0.1448, 'learning_rate': 0.0001, 'epoch': 1.7}

 17%|█▋        | 1040/6100 [1:34:08<7:37:20,  5.42s/it]
 17%|█▋        | 1041/6100 [1:34:14<7:36:46,  5.42s/it]
 17%|█▋        | 1042/6100 [1:34:19<7:36:38,  5.42s/it]
 17%|█▋        | 1043/6100 [1:34:25<7:37:22,  5.43s/it]
 17%|█▋        | 1044/6100 [1:34:30<7:36:52,  5.42s/it]
 17%|█▋        | 1045/6100 [1:34:35<7:36:09,  5.41s/it]
                                                       
{'loss': 0.148, 'learning_rate': 0.0001, 'epoch': 1.71}

 17%|█▋        | 1045/6100 [1:34:35<7:36:09,  5.41s/it]
 17%|█▋        | 1046/6100 [1:34:41<7:36:10,  5.42s/it]
 17%|█▋        | 1047/6100 [1:34:46<7:36:15,  5.42s/it]
 17%|█▋        | 1048/6100 [1:34:52<7:35:46,  5.41s/it]
 17%|█▋        | 1049/6100 [1:34:57<7:35:42,  5.41s/it]
 17%|█▋        | 1050/6100 [1:35:02<7:35:29,  5.41s/it]
                                                       
{'loss': 0.1414, 'learning_rate': 0.0001, 'epoch': 1.72}

 17%|█▋        | 1050/6100 [1:35:02<7:35:29,  5.41s/it]Saving model checkpoint to ./results/checkpoint-1050
Configuration saved in ./results/checkpoint-1050/config.json
Model weights saved in ./results/checkpoint-1050/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-1050/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-1050/special_tokens_map.json
[2023-02-21 21:27:18,616] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step1050 is begin to save!
[2023-02-21 21:27:18,619] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-1050/global_step1050/mp_rank_00_model_states.pt
[2023-02-21 21:27:18,619] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-1050/global_step1050/mp_rank_00_model_states.pt...
[2023-02-21 21:27:19,437] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-1050/global_step1050/mp_rank_00_model_states.pt.
[2023-02-21 21:27:19,439] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-1050/global_step1050/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-21 21:27:19,615] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-1050/global_step1050/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-21 21:27:19,616] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-1050/global_step1050/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-21 21:27:19,616] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1050 is ready now!
Deleting older checkpoint [results/checkpoint-900] due to args.save_total_limit

 17%|█▋        | 1051/6100 [1:35:10<8:29:14,  6.05s/it]
 17%|█▋        | 1052/6100 [1:35:15<8:12:47,  5.86s/it]
 17%|█▋        | 1053/6100 [1:35:21<8:01:19,  5.72s/it]
 17%|█▋        | 1054/6100 [1:35:26<7:53:12,  5.63s/it]
 17%|█▋        | 1055/6100 [1:35:32<7:47:28,  5.56s/it]
                                                       
{'loss': 0.1518, 'learning_rate': 0.0001, 'epoch': 1.73}

 17%|█▋        | 1055/6100 [1:35:32<7:47:28,  5.56s/it]
 17%|█▋        | 1056/6100 [1:35:37<7:44:48,  5.53s/it]
 17%|█▋        | 1057/6100 [1:35:42<7:41:54,  5.50s/it]
 17%|█▋        | 1058/6100 [1:35:48<7:39:41,  5.47s/it]
 17%|█▋        | 1059/6100 [1:35:53<7:38:23,  5.46s/it]
 17%|█▋        | 1060/6100 [1:35:59<7:37:04,  5.44s/it]
                                                       
{'loss': 0.1479, 'learning_rate': 0.0001, 'epoch': 1.74}

 17%|█▋        | 1060/6100 [1:35:59<7:37:04,  5.44s/it]
 17%|█▋        | 1061/6100 [1:36:04<7:36:26,  5.43s/it]
 17%|█▋        | 1062/6100 [1:36:10<7:35:39,  5.43s/it]
 17%|█▋        | 1063/6100 [1:36:15<7:35:16,  5.42s/it]
 17%|█▋        | 1064/6100 [1:36:20<7:35:01,  5.42s/it]
 17%|█▋        | 1065/6100 [1:36:26<7:34:34,  5.42s/it]
                                                       
{'loss': 0.145, 'learning_rate': 0.0001, 'epoch': 1.75}

 17%|█▋        | 1065/6100 [1:36:26<7:34:34,  5.42s/it]
 17%|█▋        | 1066/6100 [1:36:31<7:34:32,  5.42s/it]
 17%|█▋        | 1067/6100 [1:36:37<7:37:57,  5.46s/it]
 18%|█▊        | 1068/6100 [1:36:42<7:36:43,  5.45s/it]
 18%|█▊        | 1069/6100 [1:36:48<7:35:47,  5.44s/it]
 18%|█▊        | 1070/6100 [1:36:53<7:35:31,  5.43s/it]
                                                       
{'loss': 0.1543, 'learning_rate': 0.0001, 'epoch': 1.75}

 18%|█▊        | 1070/6100 [1:36:53<7:35:31,  5.43s/it]
 18%|█▊        | 1071/6100 [1:36:58<7:35:06,  5.43s/it]
 18%|█▊        | 1072/6100 [1:37:04<7:34:32,  5.42s/it]
 18%|█▊        | 1073/6100 [1:37:09<7:34:18,  5.42s/it]
 18%|█▊        | 1074/6100 [1:37:15<7:33:51,  5.42s/it]
 18%|█▊        | 1075/6100 [1:37:20<7:33:48,  5.42s/it]
                                                       
{'loss': 0.1516, 'learning_rate': 0.0001, 'epoch': 1.76}

 18%|█▊        | 1075/6100 [1:37:20<7:33:48,  5.42s/it]
 18%|█▊        | 1076/6100 [1:37:26<7:33:48,  5.42s/it]
 18%|█▊        | 1077/6100 [1:37:31<7:33:50,  5.42s/it]
 18%|█▊        | 1078/6100 [1:37:36<7:33:25,  5.42s/it]
 18%|█▊        | 1079/6100 [1:37:42<7:33:19,  5.42s/it]
 18%|█▊        | 1080/6100 [1:37:47<7:33:20,  5.42s/it]
                                                       
{'loss': 0.1441, 'learning_rate': 0.0001, 'epoch': 1.77}

 18%|█▊        | 1080/6100 [1:37:47<7:33:20,  5.42s/it]
 18%|█▊        | 1081/6100 [1:37:53<7:33:25,  5.42s/it]
 18%|█▊        | 1082/6100 [1:37:58<7:33:34,  5.42s/it]
 18%|█▊        | 1083/6100 [1:38:03<7:33:05,  5.42s/it]
 18%|█▊        | 1084/6100 [1:38:09<7:32:50,  5.42s/it]
 18%|█▊        | 1085/6100 [1:38:14<7:32:53,  5.42s/it]
                                                       
{'loss': 0.1353, 'learning_rate': 0.0001, 'epoch': 1.78}

 18%|█▊        | 1085/6100 [1:38:14<7:32:53,  5.42s/it]
 18%|█▊        | 1086/6100 [1:38:20<7:33:01,  5.42s/it]
 18%|█▊        | 1087/6100 [1:38:25<7:33:04,  5.42s/it]
 18%|█▊        | 1088/6100 [1:38:31<7:32:46,  5.42s/it]
 18%|█▊        | 1089/6100 [1:38:36<7:32:54,  5.42s/it]
 18%|█▊        | 1090/6100 [1:38:41<7:32:29,  5.42s/it]
                                                       
{'loss': 0.1465, 'learning_rate': 0.0001, 'epoch': 1.79}

 18%|█▊        | 1090/6100 [1:38:41<7:32:29,  5.42s/it]
 18%|█▊        | 1091/6100 [1:38:47<7:32:36,  5.42s/it]
 18%|█▊        | 1092/6100 [1:38:52<7:32:06,  5.42s/it]
 18%|█▊        | 1093/6100 [1:38:58<7:32:08,  5.42s/it]
 18%|█▊        | 1094/6100 [1:39:03<7:31:58,  5.42s/it]
 18%|█▊        | 1095/6100 [1:39:09<7:34:17,  5.45s/it]
                                                       
{'loss': 0.1461, 'learning_rate': 0.0001, 'epoch': 1.79}

 18%|█▊        | 1095/6100 [1:39:09<7:34:17,  5.45s/it]
 18%|█▊        | 1096/6100 [1:39:14<7:33:15,  5.43s/it]
 18%|█▊        | 1097/6100 [1:39:19<7:32:55,  5.43s/it]
 18%|█▊        | 1098/6100 [1:39:25<7:32:45,  5.43s/it]
 18%|█▊        | 1099/6100 [1:39:30<7:32:36,  5.43s/it]
 18%|█▊        | 1100/6100 [1:39:36<7:32:12,  5.43s/it]
                                                       
{'loss': 0.1431, 'learning_rate': 0.0001, 'epoch': 1.8}

 18%|█▊        | 1100/6100 [1:39:36<7:32:12,  5.43s/it]
 18%|█▊        | 1101/6100 [1:39:41<7:31:52,  5.42s/it]
 18%|█▊        | 1102/6100 [1:39:47<7:31:28,  5.42s/it]
 18%|█▊        | 1103/6100 [1:39:52<7:31:38,  5.42s/it]
 18%|█▊        | 1104/6100 [1:39:57<7:31:25,  5.42s/it]
 18%|█▊        | 1105/6100 [1:40:03<7:30:55,  5.42s/it]
                                                       
{'loss': 0.15, 'learning_rate': 0.0001, 'epoch': 1.81}

 18%|█▊        | 1105/6100 [1:40:03<7:30:55,  5.42s/it]
 18%|█▊        | 1106/6100 [1:40:08<7:31:01,  5.42s/it]
 18%|█▊        | 1107/6100 [1:40:14<7:30:34,  5.41s/it]
 18%|█▊        | 1108/6100 [1:40:19<7:30:18,  5.41s/it]
 18%|█▊        | 1109/6100 [1:40:24<7:30:13,  5.41s/it]
 18%|█▊        | 1110/6100 [1:40:30<7:30:01,  5.41s/it]
                                                       
{'loss': 0.1487, 'learning_rate': 0.0001, 'epoch': 1.82}

 18%|█▊        | 1110/6100 [1:40:30<7:30:01,  5.41s/it]
 18%|█▊        | 1111/6100 [1:40:35<7:30:16,  5.42s/it]
 18%|█▊        | 1112/6100 [1:40:41<7:29:56,  5.41s/it]
 18%|█▊        | 1113/6100 [1:40:46<7:30:26,  5.42s/it]
 18%|█▊        | 1114/6100 [1:40:52<7:30:12,  5.42s/it]
 18%|█▊        | 1115/6100 [1:40:57<7:29:50,  5.41s/it]
                                                       
{'loss': 0.1432, 'learning_rate': 0.0001, 'epoch': 1.83}

 18%|█▊        | 1115/6100 [1:40:57<7:29:50,  5.41s/it]
 18%|█▊        | 1116/6100 [1:41:02<7:29:57,  5.42s/it]
 18%|█▊        | 1117/6100 [1:41:08<7:29:57,  5.42s/it]
 18%|█▊        | 1118/6100 [1:41:13<7:29:47,  5.42s/it]
 18%|█▊        | 1119/6100 [1:41:19<7:29:34,  5.42s/it]
 18%|█▊        | 1120/6100 [1:41:24<7:29:40,  5.42s/it]
                                                       
{'loss': 0.137, 'learning_rate': 0.0001, 'epoch': 1.84}

 18%|█▊        | 1120/6100 [1:41:24<7:29:40,  5.42s/it]
 18%|█▊        | 1121/6100 [1:41:29<7:29:20,  5.41s/it]
 18%|█▊        | 1122/6100 [1:41:35<7:29:04,  5.41s/it]
 18%|█▊        | 1123/6100 [1:41:40<7:29:01,  5.41s/it]
 18%|█▊        | 1124/6100 [1:41:46<7:28:52,  5.41s/it]
 18%|█▊        | 1125/6100 [1:41:51<7:28:50,  5.41s/it]
                                                       
{'loss': 0.1441, 'learning_rate': 0.0001, 'epoch': 1.84}

 18%|█▊        | 1125/6100 [1:41:51<7:28:50,  5.41s/it]
 18%|█▊        | 1126/6100 [1:41:56<7:29:15,  5.42s/it]
 18%|█▊        | 1127/6100 [1:42:02<7:29:11,  5.42s/it]
 18%|█▊        | 1128/6100 [1:42:07<7:28:58,  5.42s/it]
 19%|█▊        | 1129/6100 [1:42:13<7:28:56,  5.42s/it]
 19%|█▊        | 1130/6100 [1:42:18<7:29:30,  5.43s/it]
                                                       
{'loss': 0.1339, 'learning_rate': 0.0001, 'epoch': 1.85}

 19%|█▊        | 1130/6100 [1:42:18<7:29:30,  5.43s/it]
 19%|█▊        | 1131/6100 [1:42:24<7:29:08,  5.42s/it]
 19%|█▊        | 1132/6100 [1:42:29<7:28:53,  5.42s/it]
 19%|█▊        | 1133/6100 [1:42:34<7:28:36,  5.42s/it]
 19%|█▊        | 1134/6100 [1:42:40<7:28:22,  5.42s/it]
 19%|█▊        | 1135/6100 [1:42:45<7:28:34,  5.42s/it]
                                                       
{'loss': 0.1379, 'learning_rate': 0.0001, 'epoch': 1.86}

 19%|█▊        | 1135/6100 [1:42:45<7:28:34,  5.42s/it]
 19%|█▊        | 1136/6100 [1:42:51<7:28:18,  5.42s/it]
 19%|█▊        | 1137/6100 [1:42:56<7:28:26,  5.42s/it]
 19%|█▊        | 1138/6100 [1:43:02<7:28:36,  5.42s/it]
 19%|█▊        | 1139/6100 [1:43:07<7:28:09,  5.42s/it]
 19%|█▊        | 1140/6100 [1:43:12<7:27:48,  5.42s/it]
                                                       
{'loss': 0.1392, 'learning_rate': 0.0001, 'epoch': 1.87}

 19%|█▊        | 1140/6100 [1:43:12<7:27:48,  5.42s/it]
 19%|█▊        | 1141/6100 [1:43:18<7:27:45,  5.42s/it]
 19%|█▊        | 1142/6100 [1:43:23<7:27:36,  5.42s/it]
 19%|█▊        | 1143/6100 [1:43:29<7:27:44,  5.42s/it]
 19%|█▉        | 1144/6100 [1:43:34<7:28:05,  5.42s/it]
 19%|█▉        | 1145/6100 [1:43:40<7:28:04,  5.43s/it]
                                                       
{'loss': 0.1376, 'learning_rate': 0.0001, 'epoch': 1.88}

 19%|█▉        | 1145/6100 [1:43:40<7:28:04,  5.43s/it]
 19%|█▉        | 1146/6100 [1:43:45<7:27:47,  5.42s/it]
 19%|█▉        | 1147/6100 [1:43:50<7:27:40,  5.42s/it]
 19%|█▉        | 1148/6100 [1:43:56<7:27:14,  5.42s/it]
 19%|█▉        | 1149/6100 [1:44:01<7:27:54,  5.43s/it]
 19%|█▉        | 1150/6100 [1:44:07<7:27:44,  5.43s/it]
                                                       
{'loss': 0.1489, 'learning_rate': 0.0001, 'epoch': 1.88}

 19%|█▉        | 1150/6100 [1:44:07<7:27:44,  5.43s/it]
 19%|█▉        | 1151/6100 [1:44:12<7:27:26,  5.42s/it]
 19%|█▉        | 1152/6100 [1:44:17<7:26:53,  5.42s/it]
 19%|█▉        | 1153/6100 [1:44:23<7:26:33,  5.42s/it]
 19%|█▉        | 1154/6100 [1:44:28<7:26:32,  5.42s/it]
 19%|█▉        | 1155/6100 [1:44:34<7:26:27,  5.42s/it]
                                                       
{'loss': 0.1398, 'learning_rate': 0.0001, 'epoch': 1.89}

 19%|█▉        | 1155/6100 [1:44:34<7:26:27,  5.42s/it]
 19%|█▉        | 1156/6100 [1:44:39<7:26:13,  5.42s/it]
 19%|█▉        | 1157/6100 [1:44:45<7:26:07,  5.42s/it]
 19%|█▉        | 1158/6100 [1:44:50<7:26:06,  5.42s/it]
 19%|█▉        | 1159/6100 [1:44:55<7:26:21,  5.42s/it]
 19%|█▉        | 1160/6100 [1:45:01<7:26:28,  5.42s/it]
                                                       
{'loss': 0.1383, 'learning_rate': 0.0001, 'epoch': 1.9}

 19%|█▉        | 1160/6100 [1:45:01<7:26:28,  5.42s/it]
 19%|█▉        | 1161/6100 [1:45:06<7:26:56,  5.43s/it]
 19%|█▉        | 1162/6100 [1:45:12<7:26:50,  5.43s/it]
 19%|█▉        | 1163/6100 [1:45:17<7:26:37,  5.43s/it]
 19%|█▉        | 1164/6100 [1:45:23<7:26:10,  5.42s/it]
 19%|█▉        | 1165/6100 [1:45:28<7:25:43,  5.42s/it]
                                                       
{'loss': 0.137, 'learning_rate': 0.0001, 'epoch': 1.91}

 19%|█▉        | 1165/6100 [1:45:28<7:25:43,  5.42s/it]
 19%|█▉        | 1166/6100 [1:45:33<7:25:18,  5.42s/it]
 19%|█▉        | 1167/6100 [1:45:39<7:25:26,  5.42s/it]
 19%|█▉        | 1168/6100 [1:45:44<7:25:14,  5.42s/it]
 19%|█▉        | 1169/6100 [1:45:50<7:25:11,  5.42s/it]
 19%|█▉        | 1170/6100 [1:45:55<7:25:12,  5.42s/it]
                                                       
{'loss': 0.129, 'learning_rate': 0.0001, 'epoch': 1.92}

 19%|█▉        | 1170/6100 [1:45:55<7:25:12,  5.42s/it]
 19%|█▉        | 1171/6100 [1:46:00<7:25:39,  5.43s/it]
 19%|█▉        | 1172/6100 [1:46:06<7:25:24,  5.42s/it]
 19%|█▉        | 1173/6100 [1:46:11<7:25:28,  5.42s/it]
 19%|█▉        | 1174/6100 [1:46:17<7:25:07,  5.42s/it]
 19%|█▉        | 1175/6100 [1:46:22<7:24:58,  5.42s/it]
                                                       
{'loss': 0.1413, 'learning_rate': 0.0001, 'epoch': 1.93}

 19%|█▉        | 1175/6100 [1:46:22<7:24:58,  5.42s/it]
 19%|█▉        | 1176/6100 [1:46:28<7:25:01,  5.42s/it]
 19%|█▉        | 1177/6100 [1:46:34<7:43:02,  5.64s/it]
 19%|█▉        | 1178/6100 [1:46:39<7:37:30,  5.58s/it]
 19%|█▉        | 1179/6100 [1:46:45<7:33:36,  5.53s/it]
 19%|█▉        | 1180/6100 [1:46:50<7:30:56,  5.50s/it]
                                                       
{'loss': 0.1381, 'learning_rate': 0.0001, 'epoch': 1.93}

 19%|█▉        | 1180/6100 [1:46:50<7:30:56,  5.50s/it]
 19%|█▉        | 1181/6100 [1:46:55<7:29:27,  5.48s/it]
 19%|█▉        | 1182/6100 [1:47:01<7:27:47,  5.46s/it]
 19%|█▉        | 1183/6100 [1:47:06<7:26:55,  5.45s/it]
 19%|█▉        | 1184/6100 [1:47:12<7:25:57,  5.44s/it]
 19%|█▉        | 1185/6100 [1:47:17<7:25:12,  5.43s/it]
                                                       
{'loss': 0.1372, 'learning_rate': 0.0001, 'epoch': 1.94}

 19%|█▉        | 1185/6100 [1:47:17<7:25:12,  5.43s/it]
 19%|█▉        | 1186/6100 [1:47:23<7:24:28,  5.43s/it]
 19%|█▉        | 1187/6100 [1:47:28<7:24:30,  5.43s/it]
 19%|█▉        | 1188/6100 [1:47:33<7:23:49,  5.42s/it]
 19%|█▉        | 1189/6100 [1:47:39<7:23:41,  5.42s/it]
 20%|█▉        | 1190/6100 [1:47:44<7:23:37,  5.42s/it]
                                                       
{'loss': 0.1362, 'learning_rate': 0.0001, 'epoch': 1.95}

 20%|█▉        | 1190/6100 [1:47:44<7:23:37,  5.42s/it]
 20%|█▉        | 1191/6100 [1:47:50<7:24:05,  5.43s/it]
 20%|█▉        | 1192/6100 [1:47:55<7:24:00,  5.43s/it]
 20%|█▉        | 1193/6100 [1:48:00<7:23:40,  5.43s/it]
 20%|█▉        | 1194/6100 [1:48:06<7:23:32,  5.42s/it]
 20%|█▉        | 1195/6100 [1:48:11<7:22:59,  5.42s/it]
                                                       
{'loss': 0.1394, 'learning_rate': 0.0001, 'epoch': 1.96}

 20%|█▉        | 1195/6100 [1:48:11<7:22:59,  5.42s/it]
 20%|█▉        | 1196/6100 [1:48:17<7:22:54,  5.42s/it]
 20%|█▉        | 1197/6100 [1:48:22<7:22:47,  5.42s/it]
 20%|█▉        | 1198/6100 [1:48:28<7:22:27,  5.42s/it]
 20%|█▉        | 1199/6100 [1:48:33<7:22:39,  5.42s/it]
 20%|█▉        | 1200/6100 [1:48:38<7:22:19,  5.42s/it]
                                                       
{'loss': 0.1271, 'learning_rate': 0.0001, 'epoch': 1.97}

 20%|█▉        | 1200/6100 [1:48:38<7:22:19,  5.42s/it]Saving model checkpoint to ./results/checkpoint-1200
Configuration saved in ./results/checkpoint-1200/config.json
Model weights saved in ./results/checkpoint-1200/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-1200/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-1200/special_tokens_map.json
[2023-02-21 21:40:54,564] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step1200 is begin to save!
[2023-02-21 21:40:54,567] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-1200/global_step1200/mp_rank_00_model_states.pt
[2023-02-21 21:40:54,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-1200/global_step1200/mp_rank_00_model_states.pt...
[2023-02-21 21:40:55,382] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-1200/global_step1200/mp_rank_00_model_states.pt.
[2023-02-21 21:40:55,384] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-1200/global_step1200/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-21 21:40:55,560] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-1200/global_step1200/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-21 21:40:55,560] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-1200/global_step1200/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-21 21:40:55,560] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1200 is ready now!
Deleting older checkpoint [results/checkpoint-1050] due to args.save_total_limit

 20%|█▉        | 1201/6100 [1:48:46<8:14:42,  6.06s/it]
 20%|█▉        | 1202/6100 [1:48:51<7:58:39,  5.86s/it]
 20%|█▉        | 1203/6100 [1:48:57<7:47:35,  5.73s/it]
 20%|█▉        | 1204/6100 [1:49:02<7:39:52,  5.64s/it]
 20%|█▉        | 1205/6100 [1:49:08<7:34:15,  5.57s/it]
                                                       
{'loss': 0.1454, 'learning_rate': 0.0001, 'epoch': 1.97}

 20%|█▉        | 1205/6100 [1:49:08<7:34:15,  5.57s/it]
 20%|█▉        | 1206/6100 [1:49:13<7:30:11,  5.52s/it]
 20%|█▉        | 1207/6100 [1:49:18<7:27:37,  5.49s/it]
 20%|█▉        | 1208/6100 [1:49:24<7:25:41,  5.47s/it]
 20%|█▉        | 1209/6100 [1:49:29<7:24:20,  5.45s/it]
 20%|█▉        | 1210/6100 [1:49:35<7:23:26,  5.44s/it]
                                                       
{'loss': 0.1376, 'learning_rate': 0.0001, 'epoch': 1.98}

 20%|█▉        | 1210/6100 [1:49:35<7:23:26,  5.44s/it]
 20%|█▉        | 1211/6100 [1:49:40<7:22:32,  5.43s/it]
 20%|█▉        | 1212/6100 [1:49:46<7:22:15,  5.43s/it]
 20%|█▉        | 1213/6100 [1:49:51<7:22:08,  5.43s/it]
 20%|█▉        | 1214/6100 [1:49:56<7:21:44,  5.42s/it]
 20%|█▉        | 1215/6100 [1:50:02<7:21:26,  5.42s/it]
                                                       
{'loss': 0.1362, 'learning_rate': 0.0001, 'epoch': 1.99}

 20%|█▉        | 1215/6100 [1:50:02<7:21:26,  5.42s/it]
 20%|█▉        | 1216/6100 [1:50:07<7:21:47,  5.43s/it]
 20%|█▉        | 1217/6100 [1:50:13<7:21:29,  5.42s/it]
 20%|█▉        | 1218/6100 [1:50:18<7:21:06,  5.42s/it]
 20%|█▉        | 1219/6100 [1:50:23<7:20:56,  5.42s/it]
 20%|██        | 1220/6100 [1:50:29<7:20:54,  5.42s/it]
                                                       
{'loss': 0.146, 'learning_rate': 0.0001, 'epoch': 2.0}

 20%|██        | 1220/6100 [1:50:29<7:20:54,  5.42s/it]
 20%|██        | 1221/6100 [1:50:37<8:31:32,  6.29s/it]
 20%|██        | 1222/6100 [1:50:43<8:10:18,  6.03s/it]
 20%|██        | 1223/6100 [1:50:48<7:55:05,  5.84s/it]
 20%|██        | 1224/6100 [1:50:53<7:44:29,  5.72s/it]
 20%|██        | 1225/6100 [1:50:59<7:37:03,  5.63s/it]
                                                       
{'loss': 0.1394, 'learning_rate': 0.0001, 'epoch': 2.01}

 20%|██        | 1225/6100 [1:50:59<7:37:03,  5.63s/it]
 20%|██        | 1226/6100 [1:51:04<7:32:10,  5.57s/it]
 20%|██        | 1227/6100 [1:51:10<7:28:28,  5.52s/it]
 20%|██        | 1228/6100 [1:51:16<7:37:19,  5.63s/it]
 20%|██        | 1229/6100 [1:51:21<7:31:59,  5.57s/it]
 20%|██        | 1230/6100 [1:51:26<7:27:52,  5.52s/it]
                                                       
{'loss': 0.1284, 'learning_rate': 0.0001, 'epoch': 2.02}

 20%|██        | 1230/6100 [1:51:26<7:27:52,  5.52s/it]
 20%|██        | 1231/6100 [1:51:32<7:25:25,  5.49s/it]
 20%|██        | 1232/6100 [1:51:37<7:23:29,  5.47s/it]
 20%|██        | 1233/6100 [1:51:43<7:22:02,  5.45s/it]
 20%|██        | 1234/6100 [1:51:48<7:21:23,  5.44s/it]
 20%|██        | 1235/6100 [1:51:54<7:20:33,  5.43s/it]
                                                       
{'loss': 0.1269, 'learning_rate': 0.0001, 'epoch': 2.02}

 20%|██        | 1235/6100 [1:51:54<7:20:33,  5.43s/it]
 20%|██        | 1236/6100 [1:51:59<7:20:11,  5.43s/it]
 20%|██        | 1237/6100 [1:52:04<7:19:54,  5.43s/it]
 20%|██        | 1238/6100 [1:52:10<7:19:47,  5.43s/it]
 20%|██        | 1239/6100 [1:52:15<7:19:51,  5.43s/it]
 20%|██        | 1240/6100 [1:52:21<7:19:24,  5.42s/it]
                                                       
{'loss': 0.1232, 'learning_rate': 0.0001, 'epoch': 2.03}

 20%|██        | 1240/6100 [1:52:21<7:19:24,  5.42s/it]
 20%|██        | 1241/6100 [1:52:26<7:18:44,  5.42s/it]
 20%|██        | 1242/6100 [1:52:31<7:18:35,  5.42s/it]
 20%|██        | 1243/6100 [1:52:37<7:18:22,  5.42s/it]
 20%|██        | 1244/6100 [1:52:42<7:18:08,  5.41s/it]
 20%|██        | 1245/6100 [1:52:48<7:18:25,  5.42s/it]
                                                       
{'loss': 0.1327, 'learning_rate': 0.0001, 'epoch': 2.04}

 20%|██        | 1245/6100 [1:52:48<7:18:25,  5.42s/it]
 20%|██        | 1246/6100 [1:52:53<7:18:24,  5.42s/it]
 20%|██        | 1247/6100 [1:52:59<7:18:26,  5.42s/it]
 20%|██        | 1248/6100 [1:53:04<7:18:18,  5.42s/it]
 20%|██        | 1249/6100 [1:53:09<7:18:03,  5.42s/it]
 20%|██        | 1250/6100 [1:53:15<7:18:05,  5.42s/it]
                                                       
{'loss': 0.1264, 'learning_rate': 0.0001, 'epoch': 2.05}

 20%|██        | 1250/6100 [1:53:15<7:18:05,  5.42s/it]
 21%|██        | 1251/6100 [1:53:20<7:18:19,  5.42s/it]
 21%|██        | 1252/6100 [1:53:26<7:18:34,  5.43s/it]
 21%|██        | 1253/6100 [1:53:31<7:18:03,  5.42s/it]
 21%|██        | 1254/6100 [1:53:36<7:17:49,  5.42s/it]
 21%|██        | 1255/6100 [1:53:42<7:17:35,  5.42s/it]
                                                       
{'loss': 0.1233, 'learning_rate': 0.0001, 'epoch': 2.06}

 21%|██        | 1255/6100 [1:53:42<7:17:35,  5.42s/it]
 21%|██        | 1256/6100 [1:53:47<7:17:25,  5.42s/it]
 21%|██        | 1257/6100 [1:53:53<7:17:51,  5.42s/it]
 21%|██        | 1258/6100 [1:53:58<7:17:27,  5.42s/it]
 21%|██        | 1259/6100 [1:54:04<7:17:40,  5.42s/it]
 21%|██        | 1260/6100 [1:54:09<7:17:27,  5.42s/it]
                                                       
{'loss': 0.1216, 'learning_rate': 0.0001, 'epoch': 2.07}

 21%|██        | 1260/6100 [1:54:09<7:17:27,  5.42s/it]
 21%|██        | 1261/6100 [1:54:14<7:17:19,  5.42s/it]
 21%|██        | 1262/6100 [1:54:20<7:17:13,  5.42s/it]
 21%|██        | 1263/6100 [1:54:25<7:16:58,  5.42s/it]
 21%|██        | 1264/6100 [1:54:31<7:16:33,  5.42s/it]
 21%|██        | 1265/6100 [1:54:36<7:16:31,  5.42s/it]
                                                       
{'loss': 0.1241, 'learning_rate': 0.0001, 'epoch': 2.07}

 21%|██        | 1265/6100 [1:54:36<7:16:31,  5.42s/it]
 21%|██        | 1266/6100 [1:54:42<7:16:31,  5.42s/it]
 21%|██        | 1267/6100 [1:54:47<7:16:44,  5.42s/it]
 21%|██        | 1268/6100 [1:54:52<7:16:36,  5.42s/it]
 21%|██        | 1269/6100 [1:54:58<7:16:11,  5.42s/it]
 21%|██        | 1270/6100 [1:55:03<7:16:13,  5.42s/it]
                                                       
{'loss': 0.1283, 'learning_rate': 0.0001, 'epoch': 2.08}

 21%|██        | 1270/6100 [1:55:03<7:16:13,  5.42s/it]
 21%|██        | 1271/6100 [1:55:09<7:16:01,  5.42s/it]
 21%|██        | 1272/6100 [1:55:14<7:16:06,  5.42s/it]
 21%|██        | 1273/6100 [1:55:19<7:16:13,  5.42s/it]
 21%|██        | 1274/6100 [1:55:25<7:16:19,  5.42s/it]
 21%|██        | 1275/6100 [1:55:30<7:16:11,  5.42s/it]
                                                       
{'loss': 0.1318, 'learning_rate': 0.0001, 'epoch': 2.09}

 21%|██        | 1275/6100 [1:55:30<7:16:11,  5.42s/it]
 21%|██        | 1276/6100 [1:55:36<7:15:46,  5.42s/it]
 21%|██        | 1277/6100 [1:55:41<7:15:30,  5.42s/it]
 21%|██        | 1278/6100 [1:55:47<7:15:17,  5.42s/it]
 21%|██        | 1279/6100 [1:55:52<7:15:04,  5.41s/it]
 21%|██        | 1280/6100 [1:55:57<7:14:54,  5.41s/it]
                                                       
{'loss': 0.1255, 'learning_rate': 0.0001, 'epoch': 2.1}

 21%|██        | 1280/6100 [1:55:57<7:14:54,  5.41s/it]
 21%|██        | 1281/6100 [1:56:03<7:15:22,  5.42s/it]
 21%|██        | 1282/6100 [1:56:08<7:15:12,  5.42s/it]
 21%|██        | 1283/6100 [1:56:14<7:15:06,  5.42s/it]
 21%|██        | 1284/6100 [1:56:19<7:14:47,  5.42s/it]
 21%|██        | 1285/6100 [1:56:24<7:14:21,  5.41s/it]
                                                       
{'loss': 0.127, 'learning_rate': 0.0001, 'epoch': 2.11}

 21%|██        | 1285/6100 [1:56:24<7:14:21,  5.41s/it]
 21%|██        | 1286/6100 [1:56:30<7:14:25,  5.41s/it]
 21%|██        | 1287/6100 [1:56:35<7:14:20,  5.41s/it]
 21%|██        | 1288/6100 [1:56:41<7:14:10,  5.41s/it]
 21%|██        | 1289/6100 [1:56:46<7:14:08,  5.41s/it]
 21%|██        | 1290/6100 [1:56:52<7:13:46,  5.41s/it]
                                                       
{'loss': 0.122, 'learning_rate': 0.0001, 'epoch': 2.11}

 21%|██        | 1290/6100 [1:56:52<7:13:46,  5.41s/it]
 21%|██        | 1291/6100 [1:56:57<7:14:03,  5.42s/it]
 21%|██        | 1292/6100 [1:57:02<7:14:10,  5.42s/it]
 21%|██        | 1293/6100 [1:57:08<7:14:21,  5.42s/it]
 21%|██        | 1294/6100 [1:57:13<7:13:58,  5.42s/it]
 21%|██        | 1295/6100 [1:57:19<7:14:09,  5.42s/it]
                                                       
{'loss': 0.1178, 'learning_rate': 0.0001, 'epoch': 2.12}

 21%|██        | 1295/6100 [1:57:19<7:14:09,  5.42s/it]
 21%|██        | 1296/6100 [1:57:24<7:14:02,  5.42s/it]
 21%|██▏       | 1297/6100 [1:57:30<7:14:00,  5.42s/it]
 21%|██▏       | 1298/6100 [1:57:35<7:14:02,  5.42s/it]
 21%|██▏       | 1299/6100 [1:57:40<7:13:52,  5.42s/it]
 21%|██▏       | 1300/6100 [1:57:46<7:13:46,  5.42s/it]
                                                       
{'loss': 0.1196, 'learning_rate': 0.0001, 'epoch': 2.13}

 21%|██▏       | 1300/6100 [1:57:46<7:13:46,  5.42s/it]
 21%|██▏       | 1301/6100 [1:57:51<7:13:39,  5.42s/it]
 21%|██▏       | 1302/6100 [1:57:57<7:14:18,  5.43s/it]
 21%|██▏       | 1303/6100 [1:58:02<7:14:16,  5.43s/it]
 21%|██▏       | 1304/6100 [1:58:08<7:14:04,  5.43s/it]
 21%|██▏       | 1305/6100 [1:58:13<7:13:50,  5.43s/it]
                                                       
{'loss': 0.1266, 'learning_rate': 0.0001, 'epoch': 2.14}

 21%|██▏       | 1305/6100 [1:58:13<7:13:50,  5.43s/it]
 21%|██▏       | 1306/6100 [1:58:18<7:13:18,  5.42s/it]
 21%|██▏       | 1307/6100 [1:58:24<7:13:06,  5.42s/it]
 21%|██▏       | 1308/6100 [1:58:29<7:12:59,  5.42s/it]
 21%|██▏       | 1309/6100 [1:58:35<7:12:49,  5.42s/it]
 21%|██▏       | 1310/6100 [1:58:40<7:12:34,  5.42s/it]
                                                       
{'loss': 0.1193, 'learning_rate': 0.0001, 'epoch': 2.15}

 21%|██▏       | 1310/6100 [1:58:40<7:12:34,  5.42s/it]
 21%|██▏       | 1311/6100 [1:58:45<7:12:21,  5.42s/it]
 22%|██▏       | 1312/6100 [1:58:51<7:12:16,  5.42s/it]
 22%|██▏       | 1313/6100 [1:58:56<7:14:08,  5.44s/it]
 22%|██▏       | 1314/6100 [1:59:02<7:13:19,  5.43s/it]
 22%|██▏       | 1315/6100 [1:59:07<7:12:21,  5.42s/it]
                                                       
{'loss': 0.1129, 'learning_rate': 0.0001, 'epoch': 2.16}

 22%|██▏       | 1315/6100 [1:59:07<7:12:21,  5.42s/it]
 22%|██▏       | 1316/6100 [1:59:13<7:12:10,  5.42s/it]
 22%|██▏       | 1317/6100 [1:59:18<7:12:07,  5.42s/it]
 22%|██▏       | 1318/6100 [1:59:23<7:11:50,  5.42s/it]
 22%|██▏       | 1319/6100 [1:59:29<7:11:39,  5.42s/it]
 22%|██▏       | 1320/6100 [1:59:34<7:11:30,  5.42s/it]
                                                       
{'loss': 0.1165, 'learning_rate': 0.0001, 'epoch': 2.16}

 22%|██▏       | 1320/6100 [1:59:34<7:11:30,  5.42s/it]
 22%|██▏       | 1321/6100 [1:59:40<7:11:42,  5.42s/it]
 22%|██▏       | 1322/6100 [1:59:45<7:11:55,  5.42s/it]
 22%|██▏       | 1323/6100 [1:59:50<7:11:23,  5.42s/it]
 22%|██▏       | 1324/6100 [1:59:56<7:10:57,  5.41s/it]
 22%|██▏       | 1325/6100 [2:00:01<7:10:55,  5.41s/it]
                                                       
{'loss': 0.1234, 'learning_rate': 0.0001, 'epoch': 2.17}

 22%|██▏       | 1325/6100 [2:00:01<7:10:55,  5.41s/it]
 22%|██▏       | 1326/6100 [2:00:07<7:10:43,  5.41s/it]
 22%|██▏       | 1327/6100 [2:00:12<7:10:37,  5.41s/it]
 22%|██▏       | 1328/6100 [2:00:18<7:10:40,  5.42s/it]
 22%|██▏       | 1329/6100 [2:00:23<7:10:37,  5.42s/it]
 22%|██▏       | 1330/6100 [2:00:28<7:10:27,  5.41s/it]
                                                       
{'loss': 0.1241, 'learning_rate': 0.0001, 'epoch': 2.18}

 22%|██▏       | 1330/6100 [2:00:28<7:10:27,  5.41s/it]
 22%|██▏       | 1331/6100 [2:00:34<7:10:21,  5.41s/it]
 22%|██▏       | 1332/6100 [2:00:39<7:10:15,  5.41s/it]
 22%|██▏       | 1333/6100 [2:00:45<7:10:00,  5.41s/it]
 22%|██▏       | 1334/6100 [2:00:50<7:09:50,  5.41s/it]
 22%|██▏       | 1335/6100 [2:00:55<7:09:51,  5.41s/it]
                                                       
{'loss': 0.1249, 'learning_rate': 0.0001, 'epoch': 2.19}

 22%|██▏       | 1335/6100 [2:00:55<7:09:51,  5.41s/it]
 22%|██▏       | 1336/6100 [2:01:01<7:15:47,  5.49s/it]
 22%|██▏       | 1337/6100 [2:01:07<7:13:54,  5.47s/it]
 22%|██▏       | 1338/6100 [2:01:12<7:12:48,  5.45s/it]
 22%|██▏       | 1339/6100 [2:01:17<7:11:51,  5.44s/it]
 22%|██▏       | 1340/6100 [2:01:23<7:11:05,  5.43s/it]
                                                       
{'loss': 0.1289, 'learning_rate': 0.0001, 'epoch': 2.2}

 22%|██▏       | 1340/6100 [2:01:23<7:11:05,  5.43s/it]
 22%|██▏       | 1341/6100 [2:01:28<7:10:51,  5.43s/it]
 22%|██▏       | 1342/6100 [2:01:34<7:10:42,  5.43s/it]
 22%|██▏       | 1343/6100 [2:01:39<7:10:25,  5.43s/it]
 22%|██▏       | 1344/6100 [2:01:44<7:09:55,  5.42s/it]
 22%|██▏       | 1345/6100 [2:01:50<7:09:38,  5.42s/it]
                                                       
{'loss': 0.1181, 'learning_rate': 0.0001, 'epoch': 2.2}

 22%|██▏       | 1345/6100 [2:01:50<7:09:38,  5.42s/it]
 22%|██▏       | 1346/6100 [2:01:55<7:09:41,  5.42s/it]
 22%|██▏       | 1347/6100 [2:02:01<7:09:31,  5.42s/it]
 22%|██▏       | 1348/6100 [2:02:06<7:09:16,  5.42s/it]
 22%|██▏       | 1349/6100 [2:02:12<7:09:09,  5.42s/it]
 22%|██▏       | 1350/6100 [2:02:17<7:08:46,  5.42s/it]
                                                       
{'loss': 0.1159, 'learning_rate': 0.0001, 'epoch': 2.21}

 22%|██▏       | 1350/6100 [2:02:17<7:08:46,  5.42s/it]Saving model checkpoint to ./results/checkpoint-1350
Configuration saved in ./results/checkpoint-1350/config.json
Model weights saved in ./results/checkpoint-1350/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-1350/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-1350/special_tokens_map.json
[2023-02-21 21:54:33,148] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step1351 is begin to save!
[2023-02-21 21:54:33,151] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-1350/global_step1351/mp_rank_00_model_states.pt
[2023-02-21 21:54:33,151] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-1350/global_step1351/mp_rank_00_model_states.pt...
[2023-02-21 21:54:33,976] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-1350/global_step1351/mp_rank_00_model_states.pt.
[2023-02-21 21:54:33,978] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-1350/global_step1351/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-21 21:54:34,154] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-1350/global_step1351/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-21 21:54:34,155] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-1350/global_step1351/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-21 21:54:34,155] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1351 is ready now!
Deleting older checkpoint [results/checkpoint-1200] due to args.save_total_limit

 22%|██▏       | 1351/6100 [2:02:25<7:59:12,  6.05s/it]
 22%|██▏       | 1352/6100 [2:02:30<7:43:51,  5.86s/it]
 22%|██▏       | 1353/6100 [2:02:35<7:33:17,  5.73s/it]
 22%|██▏       | 1354/6100 [2:02:41<7:25:30,  5.63s/it]
 22%|██▏       | 1355/6100 [2:02:46<7:20:09,  5.57s/it]
                                                       
{'loss': 0.1228, 'learning_rate': 0.0001, 'epoch': 2.22}

 22%|██▏       | 1355/6100 [2:02:46<7:20:09,  5.57s/it]
 22%|██▏       | 1356/6100 [2:02:52<7:16:25,  5.52s/it]
 22%|██▏       | 1357/6100 [2:02:57<7:14:15,  5.49s/it]
 22%|██▏       | 1358/6100 [2:03:02<7:12:12,  5.47s/it]
 22%|██▏       | 1359/6100 [2:03:08<7:10:59,  5.45s/it]
 22%|██▏       | 1360/6100 [2:03:13<7:10:00,  5.44s/it]
                                                       
{'loss': 0.1326, 'learning_rate': 0.0001, 'epoch': 2.23}

 22%|██▏       | 1360/6100 [2:03:13<7:10:00,  5.44s/it]
 22%|██▏       | 1361/6100 [2:03:19<7:11:27,  5.46s/it]
 22%|██▏       | 1362/6100 [2:03:24<7:10:01,  5.45s/it]
 22%|██▏       | 1363/6100 [2:03:30<7:09:23,  5.44s/it]
 22%|██▏       | 1364/6100 [2:03:35<7:08:54,  5.43s/it]
 22%|██▏       | 1365/6100 [2:03:40<7:08:28,  5.43s/it]
                                                       
{'loss': 0.1299, 'learning_rate': 0.0001, 'epoch': 2.24}

 22%|██▏       | 1365/6100 [2:03:40<7:08:28,  5.43s/it]
 22%|██▏       | 1366/6100 [2:03:46<7:08:21,  5.43s/it]
 22%|██▏       | 1367/6100 [2:03:51<7:07:36,  5.42s/it]
 22%|██▏       | 1368/6100 [2:03:57<7:07:06,  5.42s/it]
 22%|██▏       | 1369/6100 [2:04:02<7:07:23,  5.42s/it]
 22%|██▏       | 1370/6100 [2:04:08<7:07:06,  5.42s/it]
                                                       
{'loss': 0.1161, 'learning_rate': 0.0001, 'epoch': 2.25}

 22%|██▏       | 1370/6100 [2:04:08<7:07:06,  5.42s/it]
 22%|██▏       | 1371/6100 [2:04:13<7:07:16,  5.42s/it]
 22%|██▏       | 1372/6100 [2:04:18<7:06:52,  5.42s/it]
 23%|██▎       | 1373/6100 [2:04:24<7:06:51,  5.42s/it]
 23%|██▎       | 1374/6100 [2:04:29<7:06:48,  5.42s/it]
 23%|██▎       | 1375/6100 [2:04:35<7:06:52,  5.42s/it]
                                                       
{'loss': 0.1225, 'learning_rate': 0.0001, 'epoch': 2.25}

 23%|██▎       | 1375/6100 [2:04:35<7:06:52,  5.42s/it]
 23%|██▎       | 1376/6100 [2:04:40<7:06:41,  5.42s/it]
 23%|██▎       | 1377/6100 [2:04:45<7:06:46,  5.42s/it]
 23%|██▎       | 1378/6100 [2:04:51<7:06:22,  5.42s/it]
 23%|██▎       | 1379/6100 [2:04:56<7:06:09,  5.42s/it]
 23%|██▎       | 1380/6100 [2:05:02<7:06:10,  5.42s/it]
                                                       
{'loss': 0.1177, 'learning_rate': 0.0001, 'epoch': 2.26}

 23%|██▎       | 1380/6100 [2:05:02<7:06:10,  5.42s/it]
 23%|██▎       | 1381/6100 [2:05:07<7:06:10,  5.42s/it]
 23%|██▎       | 1382/6100 [2:05:13<7:05:45,  5.41s/it]
 23%|██▎       | 1383/6100 [2:05:18<7:05:34,  5.41s/it]
 23%|██▎       | 1384/6100 [2:05:23<7:05:35,  5.41s/it]
 23%|██▎       | 1385/6100 [2:05:29<7:05:24,  5.41s/it]
                                                       
{'loss': 0.1189, 'learning_rate': 0.0001, 'epoch': 2.27}

 23%|██▎       | 1385/6100 [2:05:29<7:05:24,  5.41s/it]
 23%|██▎       | 1386/6100 [2:05:34<7:05:48,  5.42s/it]
 23%|██▎       | 1387/6100 [2:05:40<7:05:42,  5.42s/it]
 23%|██▎       | 1388/6100 [2:05:45<7:05:15,  5.42s/it]
 23%|██▎       | 1389/6100 [2:05:50<7:05:10,  5.42s/it]
 23%|██▎       | 1390/6100 [2:05:56<7:05:10,  5.42s/it]
                                                       
{'loss': 0.1143, 'learning_rate': 0.0001, 'epoch': 2.28}

 23%|██▎       | 1390/6100 [2:05:56<7:05:10,  5.42s/it]
 23%|██▎       | 1391/6100 [2:06:01<7:05:30,  5.42s/it]
 23%|██▎       | 1392/6100 [2:06:07<7:05:13,  5.42s/it]
 23%|██▎       | 1393/6100 [2:06:12<7:05:11,  5.42s/it]
 23%|██▎       | 1394/6100 [2:06:18<7:05:05,  5.42s/it]
 23%|██▎       | 1395/6100 [2:06:23<7:05:00,  5.42s/it]
                                                       
{'loss': 0.1136, 'learning_rate': 0.0001, 'epoch': 2.29}

 23%|██▎       | 1395/6100 [2:06:23<7:05:00,  5.42s/it]
 23%|██▎       | 1396/6100 [2:06:28<7:04:41,  5.42s/it]
 23%|██▎       | 1397/6100 [2:06:34<7:04:50,  5.42s/it]
 23%|██▎       | 1398/6100 [2:06:39<7:04:36,  5.42s/it]
 23%|██▎       | 1399/6100 [2:06:45<7:08:09,  5.46s/it]
 23%|██▎       | 1400/6100 [2:06:50<7:06:48,  5.45s/it]
                                                       
{'loss': 0.1126, 'learning_rate': 0.0001, 'epoch': 2.29}

 23%|██▎       | 1400/6100 [2:06:50<7:06:48,  5.45s/it]
 23%|██▎       | 1401/6100 [2:06:56<7:06:17,  5.44s/it]
 23%|██▎       | 1402/6100 [2:07:01<7:05:23,  5.43s/it]
 23%|██▎       | 1403/6100 [2:07:06<7:04:56,  5.43s/it]
 23%|██▎       | 1404/6100 [2:07:12<7:04:46,  5.43s/it]
 23%|██▎       | 1405/6100 [2:07:17<7:04:22,  5.42s/it]
                                                       
{'loss': 0.1192, 'learning_rate': 0.0001, 'epoch': 2.3}

 23%|██▎       | 1405/6100 [2:07:17<7:04:22,  5.42s/it]
 23%|██▎       | 1406/6100 [2:07:23<7:04:14,  5.42s/it]
 23%|██▎       | 1407/6100 [2:07:28<7:04:30,  5.43s/it]
 23%|██▎       | 1408/6100 [2:07:34<7:03:47,  5.42s/it]
 23%|██▎       | 1409/6100 [2:07:39<7:03:35,  5.42s/it]
 23%|██▎       | 1410/6100 [2:07:44<7:03:37,  5.42s/it]
                                                       
{'loss': 0.1234, 'learning_rate': 0.0001, 'epoch': 2.31}

 23%|██▎       | 1410/6100 [2:07:44<7:03:37,  5.42s/it]
 23%|██▎       | 1411/6100 [2:07:50<7:03:24,  5.42s/it]
 23%|██▎       | 1412/6100 [2:07:55<7:03:14,  5.42s/it]
 23%|██▎       | 1413/6100 [2:08:01<7:03:04,  5.42s/it]
 23%|██▎       | 1414/6100 [2:08:06<7:03:06,  5.42s/it]
 23%|██▎       | 1415/6100 [2:08:11<7:02:51,  5.42s/it]
                                                       
{'loss': 0.1242, 'learning_rate': 0.0001, 'epoch': 2.32}

 23%|██▎       | 1415/6100 [2:08:11<7:02:51,  5.42s/it]
 23%|██▎       | 1416/6100 [2:08:17<7:02:56,  5.42s/it]
 23%|██▎       | 1417/6100 [2:08:22<7:02:50,  5.42s/it]
 23%|██▎       | 1418/6100 [2:08:28<7:02:52,  5.42s/it]
 23%|██▎       | 1419/6100 [2:08:33<7:02:54,  5.42s/it]
 23%|██▎       | 1420/6100 [2:08:39<7:02:41,  5.42s/it]
                                                       
{'loss': 0.1202, 'learning_rate': 0.0001, 'epoch': 2.33}

 23%|██▎       | 1420/6100 [2:08:39<7:02:41,  5.42s/it]
 23%|██▎       | 1421/6100 [2:08:44<7:02:37,  5.42s/it]
 23%|██▎       | 1422/6100 [2:08:49<7:02:40,  5.42s/it]
 23%|██▎       | 1423/6100 [2:08:55<7:02:23,  5.42s/it]
 23%|██▎       | 1424/6100 [2:09:00<7:02:28,  5.42s/it]
 23%|██▎       | 1425/6100 [2:09:06<7:02:18,  5.42s/it]
                                                       
{'loss': 0.1157, 'learning_rate': 0.0001, 'epoch': 2.34}

 23%|██▎       | 1425/6100 [2:09:06<7:02:18,  5.42s/it]
 23%|██▎       | 1426/6100 [2:09:11<7:02:12,  5.42s/it]
 23%|██▎       | 1427/6100 [2:09:17<7:01:57,  5.42s/it]
 23%|██▎       | 1428/6100 [2:09:22<7:01:45,  5.42s/it]
 23%|██▎       | 1429/6100 [2:09:27<7:01:42,  5.42s/it]
 23%|██▎       | 1430/6100 [2:09:33<7:01:56,  5.42s/it]
                                                       
{'loss': 0.1114, 'learning_rate': 0.0001, 'epoch': 2.34}

 23%|██▎       | 1430/6100 [2:09:33<7:01:56,  5.42s/it]
 23%|██▎       | 1431/6100 [2:09:38<7:01:49,  5.42s/it]
 23%|██▎       | 1432/6100 [2:09:44<7:01:37,  5.42s/it]
 23%|██▎       | 1433/6100 [2:09:49<7:01:23,  5.42s/it]
 24%|██▎       | 1434/6100 [2:09:54<7:01:30,  5.42s/it]
 24%|██▎       | 1435/6100 [2:10:00<7:01:22,  5.42s/it]
                                                       
{'loss': 0.1083, 'learning_rate': 0.0001, 'epoch': 2.35}

 24%|██▎       | 1435/6100 [2:10:00<7:01:22,  5.42s/it]
 24%|██▎       | 1436/6100 [2:10:05<7:01:41,  5.42s/it]
 24%|██▎       | 1437/6100 [2:10:11<7:02:01,  5.43s/it]
 24%|██▎       | 1438/6100 [2:10:16<7:02:10,  5.43s/it]
 24%|██▎       | 1439/6100 [2:10:22<7:01:38,  5.43s/it]
 24%|██▎       | 1440/6100 [2:10:27<7:01:17,  5.42s/it]
                                                       
{'loss': 0.1126, 'learning_rate': 0.0001, 'epoch': 2.36}

 24%|██▎       | 1440/6100 [2:10:27<7:01:17,  5.42s/it]
 24%|██▎       | 1441/6100 [2:10:32<7:00:55,  5.42s/it]
 24%|██▎       | 1442/6100 [2:10:38<7:00:39,  5.42s/it]
 24%|██▎       | 1443/6100 [2:10:43<7:00:45,  5.42s/it]
 24%|██▎       | 1444/6100 [2:10:49<7:00:17,  5.42s/it]
 24%|██▎       | 1445/6100 [2:10:54<7:00:03,  5.41s/it]
                                                       
{'loss': 0.1152, 'learning_rate': 0.0001, 'epoch': 2.37}

 24%|██▎       | 1445/6100 [2:10:54<7:00:03,  5.41s/it]
 24%|██▎       | 1446/6100 [2:11:00<7:00:08,  5.42s/it]
 24%|██▎       | 1447/6100 [2:11:05<7:00:12,  5.42s/it]
 24%|██▎       | 1448/6100 [2:11:10<7:00:09,  5.42s/it]
 24%|██▍       | 1449/6100 [2:11:16<7:00:16,  5.42s/it]
 24%|██▍       | 1450/6100 [2:11:21<7:00:02,  5.42s/it]
                                                       
{'loss': 0.1146, 'learning_rate': 0.0001, 'epoch': 2.38}

 24%|██▍       | 1450/6100 [2:11:21<7:00:02,  5.42s/it]
 24%|██▍       | 1451/6100 [2:11:27<6:59:58,  5.42s/it]
 24%|██▍       | 1452/6100 [2:11:32<6:59:36,  5.42s/it]
 24%|██▍       | 1453/6100 [2:11:37<6:59:19,  5.41s/it]
 24%|██▍       | 1454/6100 [2:11:43<6:59:10,  5.41s/it]
 24%|██▍       | 1455/6100 [2:11:48<6:59:25,  5.42s/it]
                                                       
{'loss': 0.1135, 'learning_rate': 0.0001, 'epoch': 2.38}

 24%|██▍       | 1455/6100 [2:11:48<6:59:25,  5.42s/it]
 24%|██▍       | 1456/6100 [2:11:54<6:59:30,  5.42s/it]
 24%|██▍       | 1457/6100 [2:11:59<6:59:22,  5.42s/it]
 24%|██▍       | 1458/6100 [2:12:05<6:59:01,  5.42s/it]
 24%|██▍       | 1459/6100 [2:12:10<6:59:19,  5.42s/it]
 24%|██▍       | 1460/6100 [2:12:15<6:58:51,  5.42s/it]
                                                       
{'loss': 0.1156, 'learning_rate': 0.0001, 'epoch': 2.39}

 24%|██▍       | 1460/6100 [2:12:15<6:58:51,  5.42s/it]
 24%|██▍       | 1461/6100 [2:12:21<6:58:54,  5.42s/it]
 24%|██▍       | 1462/6100 [2:12:26<6:58:38,  5.42s/it]
 24%|██▍       | 1463/6100 [2:12:32<6:58:21,  5.41s/it]
 24%|██▍       | 1464/6100 [2:12:37<6:58:43,  5.42s/it]
 24%|██▍       | 1465/6100 [2:12:42<6:58:25,  5.42s/it]
                                                       
{'loss': 0.1143, 'learning_rate': 0.0001, 'epoch': 2.4}

 24%|██▍       | 1465/6100 [2:12:42<6:58:25,  5.42s/it]
 24%|██▍       | 1466/6100 [2:12:48<6:58:15,  5.42s/it]
 24%|██▍       | 1467/6100 [2:12:53<6:58:12,  5.42s/it]
 24%|██▍       | 1468/6100 [2:12:59<6:58:03,  5.42s/it]
 24%|██▍       | 1469/6100 [2:13:04<6:57:53,  5.41s/it]
 24%|██▍       | 1470/6100 [2:13:10<6:57:45,  5.41s/it]
                                                       
{'loss': 0.122, 'learning_rate': 0.0001, 'epoch': 2.41}

 24%|██▍       | 1470/6100 [2:13:10<6:57:45,  5.41s/it]
 24%|██▍       | 1471/6100 [2:13:15<6:57:55,  5.42s/it]
 24%|██▍       | 1472/6100 [2:13:20<6:57:36,  5.41s/it]
 24%|██▍       | 1473/6100 [2:13:26<6:57:44,  5.42s/it]
 24%|██▍       | 1474/6100 [2:13:31<6:57:44,  5.42s/it]
 24%|██▍       | 1475/6100 [2:13:37<6:57:29,  5.42s/it]
                                                       
{'loss': 0.1097, 'learning_rate': 0.0001, 'epoch': 2.42}

 24%|██▍       | 1475/6100 [2:13:37<6:57:29,  5.42s/it]
 24%|██▍       | 1476/6100 [2:13:42<6:57:32,  5.42s/it]
 24%|██▍       | 1477/6100 [2:13:47<6:57:30,  5.42s/it]
 24%|██▍       | 1478/6100 [2:13:53<6:57:04,  5.41s/it]
 24%|██▍       | 1479/6100 [2:13:58<6:56:51,  5.41s/it]
 24%|██▍       | 1480/6100 [2:14:04<6:56:47,  5.41s/it]
                                                       
{'loss': 0.1151, 'learning_rate': 0.0001, 'epoch': 2.43}

 24%|██▍       | 1480/6100 [2:14:04<6:56:47,  5.41s/it]
 24%|██▍       | 1481/6100 [2:14:09<6:57:20,  5.42s/it]
 24%|██▍       | 1482/6100 [2:14:15<6:57:21,  5.42s/it]
 24%|██▍       | 1483/6100 [2:14:20<6:57:06,  5.42s/it]
 24%|██▍       | 1484/6100 [2:14:25<6:56:54,  5.42s/it]
 24%|██▍       | 1485/6100 [2:14:31<6:56:29,  5.41s/it]
                                                       
{'loss': 0.1118, 'learning_rate': 0.0001, 'epoch': 2.43}

 24%|██▍       | 1485/6100 [2:14:31<6:56:29,  5.41s/it]
 24%|██▍       | 1486/6100 [2:14:36<6:56:14,  5.41s/it]
 24%|██▍       | 1487/6100 [2:14:42<6:56:01,  5.41s/it]
 24%|██▍       | 1488/6100 [2:14:47<6:56:11,  5.41s/it]
 24%|██▍       | 1489/6100 [2:14:52<6:56:13,  5.42s/it]
 24%|██▍       | 1490/6100 [2:14:58<6:55:57,  5.41s/it]
                                                       
{'loss': 0.1138, 'learning_rate': 0.0001, 'epoch': 2.44}

 24%|██▍       | 1490/6100 [2:14:58<6:55:57,  5.41s/it]
 24%|██▍       | 1491/6100 [2:15:03<6:55:47,  5.41s/it]
 24%|██▍       | 1492/6100 [2:15:09<6:55:43,  5.41s/it]
 24%|██▍       | 1493/6100 [2:15:14<6:55:18,  5.41s/it]
 24%|██▍       | 1494/6100 [2:15:19<6:55:05,  5.41s/it]
 25%|██▍       | 1495/6100 [2:15:25<6:55:27,  5.41s/it]
                                                       
{'loss': 0.1157, 'learning_rate': 0.0001, 'epoch': 2.45}

 25%|██▍       | 1495/6100 [2:15:25<6:55:27,  5.41s/it]
 25%|██▍       | 1496/6100 [2:15:30<6:56:35,  5.43s/it]
 25%|██▍       | 1497/6100 [2:15:36<6:56:20,  5.43s/it]
 25%|██▍       | 1498/6100 [2:15:41<6:56:06,  5.43s/it]
 25%|██▍       | 1499/6100 [2:15:47<6:55:57,  5.42s/it]
 25%|██▍       | 1500/6100 [2:15:52<6:55:38,  5.42s/it]
                                                       
{'loss': 0.1137, 'learning_rate': 0.0001, 'epoch': 2.46}

 25%|██▍       | 1500/6100 [2:15:52<6:55:38,  5.42s/it]Saving model checkpoint to ./results/checkpoint-1500
Configuration saved in ./results/checkpoint-1500/config.json
Model weights saved in ./results/checkpoint-1500/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-1500/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-1500/special_tokens_map.json
[2023-02-21 22:08:08,230] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step1501 is begin to save!
[2023-02-21 22:08:08,233] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-1500/global_step1501/mp_rank_00_model_states.pt
[2023-02-21 22:08:08,233] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-1500/global_step1501/mp_rank_00_model_states.pt...
[2023-02-21 22:08:09,102] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-1500/global_step1501/mp_rank_00_model_states.pt.
[2023-02-21 22:08:09,104] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-1500/global_step1501/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-21 22:08:09,279] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-1500/global_step1501/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-21 22:08:09,280] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-1500/global_step1501/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-21 22:08:09,280] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1501 is ready now!
Deleting older checkpoint [results/checkpoint-1350] due to args.save_total_limit

 25%|██▍       | 1501/6100 [2:16:00<7:46:08,  6.08s/it]
 25%|██▍       | 1502/6100 [2:16:05<7:30:58,  5.88s/it]
 25%|██▍       | 1503/6100 [2:16:10<7:19:52,  5.74s/it]
 25%|██▍       | 1504/6100 [2:16:16<7:11:54,  5.64s/it]
 25%|██▍       | 1505/6100 [2:16:21<7:07:09,  5.58s/it]
                                                       
{'loss': 0.1135, 'learning_rate': 0.0001, 'epoch': 2.47}

 25%|██▍       | 1505/6100 [2:16:21<7:07:09,  5.58s/it]
 25%|██▍       | 1506/6100 [2:16:27<7:03:07,  5.53s/it]
 25%|██▍       | 1507/6100 [2:16:32<7:00:47,  5.50s/it]
 25%|██▍       | 1508/6100 [2:16:38<6:58:29,  5.47s/it]
 25%|██▍       | 1509/6100 [2:16:43<6:57:11,  5.45s/it]
 25%|██▍       | 1510/6100 [2:16:48<6:56:35,  5.45s/it]
                                                       
{'loss': 0.1157, 'learning_rate': 0.0001, 'epoch': 2.48}

 25%|██▍       | 1510/6100 [2:16:48<6:56:35,  5.45s/it]
 25%|██▍       | 1511/6100 [2:16:54<6:55:49,  5.44s/it]
 25%|██▍       | 1512/6100 [2:16:59<6:55:04,  5.43s/it]
 25%|██▍       | 1513/6100 [2:17:05<6:54:32,  5.42s/it]
 25%|██▍       | 1514/6100 [2:17:10<6:54:11,  5.42s/it]
 25%|██▍       | 1515/6100 [2:17:15<6:53:53,  5.42s/it]
                                                       
{'loss': 0.1196, 'learning_rate': 0.0001, 'epoch': 2.48}

 25%|██▍       | 1515/6100 [2:17:15<6:53:53,  5.42s/it]
 25%|██▍       | 1516/6100 [2:17:21<6:53:40,  5.41s/it]
 25%|██▍       | 1517/6100 [2:17:26<6:53:34,  5.41s/it]
 25%|██▍       | 1518/6100 [2:17:32<6:53:35,  5.42s/it]
 25%|██▍       | 1519/6100 [2:17:37<6:53:34,  5.42s/it]
 25%|██▍       | 1520/6100 [2:17:43<6:54:31,  5.43s/it]
                                                       
{'loss': 0.113, 'learning_rate': 0.0001, 'epoch': 2.49}

 25%|██▍       | 1520/6100 [2:17:43<6:54:31,  5.43s/it]
 25%|██▍       | 1521/6100 [2:17:48<6:54:07,  5.43s/it]
 25%|██▍       | 1522/6100 [2:17:53<6:53:52,  5.42s/it]
 25%|██▍       | 1523/6100 [2:17:59<6:53:42,  5.42s/it]
 25%|██▍       | 1524/6100 [2:18:04<6:53:29,  5.42s/it]
 25%|██▌       | 1525/6100 [2:18:10<6:52:54,  5.42s/it]
                                                       
{'loss': 0.1124, 'learning_rate': 0.0001, 'epoch': 2.5}

 25%|██▌       | 1525/6100 [2:18:10<6:52:54,  5.42s/it]
 25%|██▌       | 1526/6100 [2:18:15<6:52:36,  5.41s/it]
 25%|██▌       | 1527/6100 [2:18:20<6:52:34,  5.41s/it]
 25%|██▌       | 1528/6100 [2:18:26<6:52:28,  5.41s/it]
 25%|██▌       | 1529/6100 [2:18:31<6:52:13,  5.41s/it]
 25%|██▌       | 1530/6100 [2:18:37<6:52:04,  5.41s/it]
                                                       
{'loss': 0.1119, 'learning_rate': 0.0001, 'epoch': 2.51}

 25%|██▌       | 1530/6100 [2:18:37<6:52:04,  5.41s/it]
 25%|██▌       | 1531/6100 [2:18:42<6:52:04,  5.41s/it]
 25%|██▌       | 1532/6100 [2:18:48<6:52:13,  5.41s/it]
 25%|██▌       | 1533/6100 [2:18:53<6:52:07,  5.41s/it]
 25%|██▌       | 1534/6100 [2:18:58<6:52:17,  5.42s/it]
 25%|██▌       | 1535/6100 [2:19:04<6:54:27,  5.45s/it]
                                                       
{'loss': 0.1103, 'learning_rate': 0.0001, 'epoch': 2.52}

 25%|██▌       | 1535/6100 [2:19:04<6:54:27,  5.45s/it]
 25%|██▌       | 1536/6100 [2:19:09<6:53:39,  5.44s/it]
 25%|██▌       | 1537/6100 [2:19:15<6:53:02,  5.43s/it]
 25%|██▌       | 1538/6100 [2:19:20<6:52:48,  5.43s/it]
 25%|██▌       | 1539/6100 [2:19:26<6:52:14,  5.42s/it]
 25%|██▌       | 1540/6100 [2:19:31<6:51:53,  5.42s/it]
                                                       
{'loss': 0.1081, 'learning_rate': 0.0001, 'epoch': 2.52}

 25%|██▌       | 1540/6100 [2:19:31<6:51:53,  5.42s/it]
 25%|██▌       | 1541/6100 [2:19:36<6:51:48,  5.42s/it]
 25%|██▌       | 1542/6100 [2:19:42<6:51:49,  5.42s/it]
 25%|██▌       | 1543/6100 [2:19:47<6:51:51,  5.42s/it]
 25%|██▌       | 1544/6100 [2:19:53<6:51:52,  5.42s/it]
 25%|██▌       | 1545/6100 [2:19:58<6:51:28,  5.42s/it]
                                                       
{'loss': 0.1081, 'learning_rate': 0.0001, 'epoch': 2.53}

 25%|██▌       | 1545/6100 [2:19:58<6:51:28,  5.42s/it]
 25%|██▌       | 1546/6100 [2:20:04<6:51:29,  5.42s/it]
 25%|██▌       | 1547/6100 [2:20:09<6:51:25,  5.42s/it]
 25%|██▌       | 1548/6100 [2:20:14<6:51:09,  5.42s/it]
 25%|██▌       | 1549/6100 [2:20:20<6:51:01,  5.42s/it]
 25%|██▌       | 1550/6100 [2:20:25<6:50:52,  5.42s/it]
                                                       
{'loss': 0.1139, 'learning_rate': 0.0001, 'epoch': 2.54}

 25%|██▌       | 1550/6100 [2:20:25<6:50:52,  5.42s/it]
 25%|██▌       | 1551/6100 [2:20:31<6:51:00,  5.42s/it]
 25%|██▌       | 1552/6100 [2:20:36<6:50:39,  5.42s/it]
 25%|██▌       | 1553/6100 [2:20:41<6:50:47,  5.42s/it]
 25%|██▌       | 1554/6100 [2:20:47<6:50:25,  5.42s/it]
 25%|██▌       | 1555/6100 [2:20:52<6:50:36,  5.42s/it]
                                                       
{'loss': 0.1094, 'learning_rate': 0.0001, 'epoch': 2.55}

 25%|██▌       | 1555/6100 [2:20:52<6:50:36,  5.42s/it]
 26%|██▌       | 1556/6100 [2:20:58<6:50:18,  5.42s/it]
 26%|██▌       | 1557/6100 [2:21:03<6:50:01,  5.42s/it]
 26%|██▌       | 1558/6100 [2:21:09<6:49:54,  5.41s/it]
 26%|██▌       | 1559/6100 [2:21:14<6:50:01,  5.42s/it]
 26%|██▌       | 1560/6100 [2:21:19<6:49:54,  5.42s/it]
                                                       
{'loss': 0.1148, 'learning_rate': 0.0001, 'epoch': 2.56}

 26%|██▌       | 1560/6100 [2:21:19<6:49:54,  5.42s/it]
 26%|██▌       | 1561/6100 [2:21:25<6:49:41,  5.42s/it]
 26%|██▌       | 1562/6100 [2:21:30<6:49:47,  5.42s/it]
 26%|██▌       | 1563/6100 [2:21:36<6:49:39,  5.42s/it]
 26%|██▌       | 1564/6100 [2:21:41<6:49:48,  5.42s/it]
 26%|██▌       | 1565/6100 [2:21:46<6:49:42,  5.42s/it]
                                                       
{'loss': 0.1136, 'learning_rate': 0.0001, 'epoch': 2.57}

 26%|██▌       | 1565/6100 [2:21:46<6:49:42,  5.42s/it]
 26%|██▌       | 1566/6100 [2:21:52<6:49:45,  5.42s/it]
 26%|██▌       | 1567/6100 [2:21:57<6:49:19,  5.42s/it]
 26%|██▌       | 1568/6100 [2:22:03<6:49:12,  5.42s/it]
 26%|██▌       | 1569/6100 [2:22:08<6:49:11,  5.42s/it]
 26%|██▌       | 1570/6100 [2:22:14<6:49:09,  5.42s/it]
                                                       
{'loss': 0.1058, 'learning_rate': 0.0001, 'epoch': 2.57}

 26%|██▌       | 1570/6100 [2:22:14<6:49:09,  5.42s/it]
 26%|██▌       | 1571/6100 [2:22:19<6:49:03,  5.42s/it]
 26%|██▌       | 1572/6100 [2:22:24<6:49:07,  5.42s/it]
 26%|██▌       | 1573/6100 [2:22:30<6:48:50,  5.42s/it]
 26%|██▌       | 1574/6100 [2:22:35<6:48:31,  5.42s/it]
 26%|██▌       | 1575/6100 [2:22:41<6:48:15,  5.41s/it]
                                                       
{'loss': 0.1134, 'learning_rate': 0.0001, 'epoch': 2.58}

 26%|██▌       | 1575/6100 [2:22:41<6:48:15,  5.41s/it]
 26%|██▌       | 1576/6100 [2:22:46<6:47:56,  5.41s/it]
 26%|██▌       | 1577/6100 [2:22:51<6:47:59,  5.41s/it]
 26%|██▌       | 1578/6100 [2:22:57<6:48:14,  5.42s/it]
 26%|██▌       | 1579/6100 [2:23:02<6:48:07,  5.42s/it]
 26%|██▌       | 1580/6100 [2:23:08<6:47:47,  5.41s/it]
                                                       
{'loss': 0.1113, 'learning_rate': 0.0001, 'epoch': 2.59}

 26%|██▌       | 1580/6100 [2:23:08<6:47:47,  5.41s/it]
 26%|██▌       | 1581/6100 [2:23:13<6:47:41,  5.41s/it]
 26%|██▌       | 1582/6100 [2:23:19<6:47:25,  5.41s/it]
 26%|██▌       | 1583/6100 [2:23:24<6:47:47,  5.42s/it]
 26%|██▌       | 1584/6100 [2:23:30<6:52:53,  5.49s/it]
 26%|██▌       | 1585/6100 [2:23:35<6:51:14,  5.47s/it]
                                                       
{'loss': 0.1059, 'learning_rate': 0.0001, 'epoch': 2.6}

 26%|██▌       | 1585/6100 [2:23:35<6:51:14,  5.47s/it]
 26%|██▌       | 1586/6100 [2:23:40<6:49:55,  5.45s/it]
 26%|██▌       | 1587/6100 [2:23:46<6:49:01,  5.44s/it]
 26%|██▌       | 1588/6100 [2:23:51<6:48:23,  5.43s/it]
 26%|██▌       | 1589/6100 [2:23:57<6:47:41,  5.42s/it]
 26%|██▌       | 1590/6100 [2:24:02<6:47:17,  5.42s/it]
                                                       
{'loss': 0.1078, 'learning_rate': 0.0001, 'epoch': 2.61}

 26%|██▌       | 1590/6100 [2:24:02<6:47:17,  5.42s/it]
 26%|██▌       | 1591/6100 [2:24:07<6:47:34,  5.42s/it]
 26%|██▌       | 1592/6100 [2:24:13<6:47:28,  5.42s/it]
 26%|██▌       | 1593/6100 [2:24:18<6:46:55,  5.42s/it]
 26%|██▌       | 1594/6100 [2:24:24<6:47:12,  5.42s/it]
 26%|██▌       | 1595/6100 [2:24:29<6:46:49,  5.42s/it]
                                                       
{'loss': 0.1049, 'learning_rate': 0.0001, 'epoch': 2.61}

 26%|██▌       | 1595/6100 [2:24:29<6:46:49,  5.42s/it]
 26%|██▌       | 1596/6100 [2:24:35<6:46:09,  5.41s/it]
 26%|██▌       | 1597/6100 [2:24:40<6:46:30,  5.42s/it]
 26%|██▌       | 1598/6100 [2:24:45<6:46:34,  5.42s/it]
 26%|██▌       | 1599/6100 [2:24:51<6:46:46,  5.42s/it]
 26%|██▌       | 1600/6100 [2:24:56<6:47:04,  5.43s/it]
                                                       
{'loss': 0.1105, 'learning_rate': 0.0001, 'epoch': 2.62}

 26%|██▌       | 1600/6100 [2:24:56<6:47:04,  5.43s/it]
 26%|██▌       | 1601/6100 [2:25:02<6:47:02,  5.43s/it]
 26%|██▋       | 1602/6100 [2:25:07<6:46:41,  5.43s/it]
 26%|██▋       | 1603/6100 [2:25:13<6:46:43,  5.43s/it]
 26%|██▋       | 1604/6100 [2:25:18<6:46:18,  5.42s/it]
 26%|██▋       | 1605/6100 [2:25:23<6:45:54,  5.42s/it]
                                                       
{'loss': 0.1222, 'learning_rate': 0.0001, 'epoch': 2.63}

 26%|██▋       | 1605/6100 [2:25:23<6:45:54,  5.42s/it]
 26%|██▋       | 1606/6100 [2:25:29<6:45:53,  5.42s/it]
 26%|██▋       | 1607/6100 [2:25:34<6:45:52,  5.42s/it]
 26%|██▋       | 1608/6100 [2:25:40<6:45:25,  5.42s/it]
 26%|██▋       | 1609/6100 [2:25:45<6:45:26,  5.42s/it]
 26%|██▋       | 1610/6100 [2:25:50<6:45:27,  5.42s/it]
                                                       
{'loss': 0.1086, 'learning_rate': 0.0001, 'epoch': 2.64}

 26%|██▋       | 1610/6100 [2:25:50<6:45:27,  5.42s/it]
 26%|██▋       | 1611/6100 [2:25:56<6:45:37,  5.42s/it]
 26%|██▋       | 1612/6100 [2:26:01<6:45:21,  5.42s/it]
 26%|██▋       | 1613/6100 [2:26:07<6:45:07,  5.42s/it]
 26%|██▋       | 1614/6100 [2:26:12<6:44:56,  5.42s/it]
 26%|██▋       | 1615/6100 [2:26:18<6:45:11,  5.42s/it]
                                                       
{'loss': 0.1065, 'learning_rate': 0.0001, 'epoch': 2.65}

 26%|██▋       | 1615/6100 [2:26:18<6:45:11,  5.42s/it]
 26%|██▋       | 1616/6100 [2:26:23<6:45:04,  5.42s/it]
 27%|██▋       | 1617/6100 [2:26:28<6:45:10,  5.42s/it]
 27%|██▋       | 1618/6100 [2:26:34<6:45:06,  5.42s/it]
 27%|██▋       | 1619/6100 [2:26:39<6:45:26,  5.43s/it]
 27%|██▋       | 1620/6100 [2:26:45<6:45:03,  5.42s/it]
                                                       
{'loss': 0.1164, 'learning_rate': 0.0001, 'epoch': 2.66}

 27%|██▋       | 1620/6100 [2:26:45<6:45:03,  5.42s/it]
 27%|██▋       | 1621/6100 [2:26:50<6:45:08,  5.43s/it]
 27%|██▋       | 1622/6100 [2:26:56<6:44:35,  5.42s/it]
 27%|██▋       | 1623/6100 [2:27:01<6:44:18,  5.42s/it]
 27%|██▋       | 1624/6100 [2:27:06<6:43:57,  5.41s/it]
 27%|██▋       | 1625/6100 [2:27:12<6:43:49,  5.41s/it]
                                                       
{'loss': 0.1072, 'learning_rate': 0.0001, 'epoch': 2.66}

 27%|██▋       | 1625/6100 [2:27:12<6:43:49,  5.41s/it]
 27%|██▋       | 1626/6100 [2:27:17<6:43:49,  5.42s/it]
 27%|██▋       | 1627/6100 [2:27:23<6:43:57,  5.42s/it]
 27%|██▋       | 1628/6100 [2:27:28<6:43:47,  5.42s/it]
 27%|██▋       | 1629/6100 [2:27:33<6:43:41,  5.42s/it]
 27%|██▋       | 1630/6100 [2:27:39<6:56:39,  5.59s/it]
                                                       
{'loss': 0.1117, 'learning_rate': 0.0001, 'epoch': 2.67}

 27%|██▋       | 1630/6100 [2:27:39<6:56:39,  5.59s/it]
 27%|██▋       | 1631/6100 [2:27:45<6:53:59,  5.56s/it]
 27%|██▋       | 1632/6100 [2:27:50<6:50:34,  5.51s/it]
 27%|██▋       | 1633/6100 [2:27:56<6:48:56,  5.49s/it]
 27%|██▋       | 1634/6100 [2:28:01<6:47:06,  5.47s/it]
 27%|██▋       | 1635/6100 [2:28:07<6:45:35,  5.45s/it]
                                                       
{'loss': 0.1204, 'learning_rate': 0.0001, 'epoch': 2.68}

 27%|██▋       | 1635/6100 [2:28:07<6:45:35,  5.45s/it]
 27%|██▋       | 1636/6100 [2:28:12<6:44:43,  5.44s/it]
 27%|██▋       | 1637/6100 [2:28:17<6:44:06,  5.43s/it]
 27%|██▋       | 1638/6100 [2:28:23<6:43:28,  5.43s/it]
 27%|██▋       | 1639/6100 [2:28:28<6:43:21,  5.43s/it]
 27%|██▋       | 1640/6100 [2:28:34<6:43:00,  5.42s/it]
                                                       
{'loss': 0.1123, 'learning_rate': 0.0001, 'epoch': 2.69}

 27%|██▋       | 1640/6100 [2:28:34<6:43:00,  5.42s/it]
 27%|██▋       | 1641/6100 [2:28:39<6:42:37,  5.42s/it]
 27%|██▋       | 1642/6100 [2:28:44<6:42:15,  5.41s/it]
 27%|██▋       | 1643/6100 [2:28:50<6:42:09,  5.41s/it]
 27%|██▋       | 1644/6100 [2:28:55<6:42:00,  5.41s/it]
 27%|██▋       | 1645/6100 [2:29:01<6:41:44,  5.41s/it]
                                                       
{'loss': 0.1087, 'learning_rate': 0.0001, 'epoch': 2.7}

 27%|██▋       | 1645/6100 [2:29:01<6:41:44,  5.41s/it]
 27%|██▋       | 1646/6100 [2:29:06<6:41:39,  5.41s/it]
 27%|██▋       | 1647/6100 [2:29:12<6:41:40,  5.41s/it]
 27%|██▋       | 1648/6100 [2:29:17<6:41:45,  5.41s/it]
 27%|██▋       | 1649/6100 [2:29:22<6:41:36,  5.41s/it]
 27%|██▋       | 1650/6100 [2:29:28<6:41:22,  5.41s/it]
                                                       
{'loss': 0.107, 'learning_rate': 0.0001, 'epoch': 2.7}

 27%|██▋       | 1650/6100 [2:29:28<6:41:22,  5.41s/it]Saving model checkpoint to ./results/checkpoint-1650
Configuration saved in ./results/checkpoint-1650/config.json
Model weights saved in ./results/checkpoint-1650/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-1650/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-1650/special_tokens_map.json
[2023-02-21 22:21:43,979] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step1651 is begin to save!
[2023-02-21 22:21:43,982] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-1650/global_step1651/mp_rank_00_model_states.pt
[2023-02-21 22:21:43,982] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-1650/global_step1651/mp_rank_00_model_states.pt...
[2023-02-21 22:21:44,805] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-1650/global_step1651/mp_rank_00_model_states.pt.
[2023-02-21 22:21:44,807] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-1650/global_step1651/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-21 22:21:45,019] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-1650/global_step1651/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-21 22:21:45,020] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-1650/global_step1651/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-21 22:21:45,020] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1651 is ready now!
Deleting older checkpoint [results/checkpoint-1500] due to args.save_total_limit

 27%|██▋       | 1651/6100 [2:29:35<7:29:59,  6.07s/it]
 27%|██▋       | 1652/6100 [2:29:41<7:14:59,  5.87s/it]
 27%|██▋       | 1653/6100 [2:29:46<7:04:17,  5.72s/it]
 27%|██▋       | 1654/6100 [2:29:52<6:57:34,  5.64s/it]
 27%|██▋       | 1655/6100 [2:29:57<6:52:19,  5.57s/it]
                                                       
{'loss': 0.1031, 'learning_rate': 0.0001, 'epoch': 2.71}

 27%|██▋       | 1655/6100 [2:29:57<6:52:19,  5.57s/it]
 27%|██▋       | 1656/6100 [2:30:02<6:48:52,  5.52s/it]
 27%|██▋       | 1657/6100 [2:30:08<6:46:38,  5.49s/it]
 27%|██▋       | 1658/6100 [2:30:13<6:44:41,  5.47s/it]
 27%|██▋       | 1659/6100 [2:30:19<6:43:13,  5.45s/it]
 27%|██▋       | 1660/6100 [2:30:24<6:42:40,  5.44s/it]
                                                       
{'loss': 0.1142, 'learning_rate': 0.0001, 'epoch': 2.72}

 27%|██▋       | 1660/6100 [2:30:24<6:42:40,  5.44s/it]
 27%|██▋       | 1661/6100 [2:30:29<6:41:38,  5.43s/it]
 27%|██▋       | 1662/6100 [2:30:35<6:40:56,  5.42s/it]
 27%|██▋       | 1663/6100 [2:30:40<6:40:54,  5.42s/it]
 27%|██▋       | 1664/6100 [2:30:46<6:40:41,  5.42s/it]
 27%|██▋       | 1665/6100 [2:30:51<6:40:32,  5.42s/it]
                                                       
{'loss': 0.1081, 'learning_rate': 0.0001, 'epoch': 2.73}

 27%|██▋       | 1665/6100 [2:30:51<6:40:32,  5.42s/it]
 27%|██▋       | 1666/6100 [2:30:57<6:40:25,  5.42s/it]
 27%|██▋       | 1667/6100 [2:31:02<6:40:02,  5.41s/it]
 27%|██▋       | 1668/6100 [2:31:07<6:39:36,  5.41s/it]
 27%|██▋       | 1669/6100 [2:31:13<6:39:32,  5.41s/it]
 27%|██▋       | 1670/6100 [2:31:18<6:39:43,  5.41s/it]
                                                       
{'loss': 0.1025, 'learning_rate': 0.0001, 'epoch': 2.74}

 27%|██▋       | 1670/6100 [2:31:18<6:39:43,  5.41s/it]
 27%|██▋       | 1671/6100 [2:31:24<6:39:22,  5.41s/it]
 27%|██▋       | 1672/6100 [2:31:29<6:39:22,  5.41s/it]
 27%|██▋       | 1673/6100 [2:31:34<6:39:41,  5.42s/it]
 27%|██▋       | 1674/6100 [2:31:40<6:39:44,  5.42s/it]
 27%|██▋       | 1675/6100 [2:31:45<6:39:57,  5.42s/it]
                                                       
{'loss': 0.1079, 'learning_rate': 0.0001, 'epoch': 2.75}

 27%|██▋       | 1675/6100 [2:31:45<6:39:57,  5.42s/it]
 27%|██▋       | 1676/6100 [2:31:51<6:40:06,  5.43s/it]
 27%|██▋       | 1677/6100 [2:31:56<6:39:38,  5.42s/it]
 28%|██▊       | 1678/6100 [2:32:02<6:39:14,  5.42s/it]
 28%|██▊       | 1679/6100 [2:32:07<6:39:15,  5.42s/it]
 28%|██▊       | 1680/6100 [2:32:12<6:38:52,  5.41s/it]
                                                       
{'loss': 0.1072, 'learning_rate': 0.0001, 'epoch': 2.75}

 28%|██▊       | 1680/6100 [2:32:12<6:38:52,  5.41s/it]
 28%|██▊       | 1681/6100 [2:32:18<6:38:59,  5.42s/it]
 28%|██▊       | 1682/6100 [2:32:23<6:38:49,  5.42s/it]
 28%|██▊       | 1683/6100 [2:32:29<6:38:38,  5.42s/it]
 28%|██▊       | 1684/6100 [2:32:34<6:38:48,  5.42s/it]
 28%|██▊       | 1685/6100 [2:32:39<6:38:30,  5.42s/it]
                                                       
{'loss': 0.1077, 'learning_rate': 0.0001, 'epoch': 2.76}

 28%|██▊       | 1685/6100 [2:32:39<6:38:30,  5.42s/it]
 28%|██▊       | 1686/6100 [2:32:45<6:38:40,  5.42s/it]
 28%|██▊       | 1687/6100 [2:32:50<6:38:40,  5.42s/it]
 28%|██▊       | 1688/6100 [2:32:56<6:38:17,  5.42s/it]
 28%|██▊       | 1689/6100 [2:33:01<6:38:09,  5.42s/it]
 28%|██▊       | 1690/6100 [2:33:07<6:37:56,  5.41s/it]
                                                       
{'loss': 0.105, 'learning_rate': 0.0001, 'epoch': 2.77}

 28%|██▊       | 1690/6100 [2:33:07<6:37:56,  5.41s/it]
 28%|██▊       | 1691/6100 [2:33:12<6:37:55,  5.42s/it]
 28%|██▊       | 1692/6100 [2:33:17<6:37:50,  5.42s/it]
 28%|██▊       | 1693/6100 [2:33:23<6:37:50,  5.42s/it]
 28%|██▊       | 1694/6100 [2:33:28<6:37:38,  5.41s/it]
 28%|██▊       | 1695/6100 [2:33:34<6:37:17,  5.41s/it]
                                                       
{'loss': 0.1033, 'learning_rate': 0.0001, 'epoch': 2.78}

 28%|██▊       | 1695/6100 [2:33:34<6:37:17,  5.41s/it]
 28%|██▊       | 1696/6100 [2:33:39<6:37:01,  5.41s/it]
 28%|██▊       | 1697/6100 [2:33:44<6:36:53,  5.41s/it]
 28%|██▊       | 1698/6100 [2:33:50<6:37:14,  5.41s/it]
 28%|██▊       | 1699/6100 [2:33:55<6:37:13,  5.42s/it]
 28%|██▊       | 1700/6100 [2:34:01<6:37:03,  5.41s/it]
                                                       
{'loss': 0.1032, 'learning_rate': 0.0001, 'epoch': 2.79}

 28%|██▊       | 1700/6100 [2:34:01<6:37:03,  5.41s/it]
 28%|██▊       | 1701/6100 [2:34:06<6:36:40,  5.41s/it]
 28%|██▊       | 1702/6100 [2:34:11<6:36:29,  5.41s/it]
 28%|██▊       | 1703/6100 [2:34:17<6:36:10,  5.41s/it]
 28%|██▊       | 1704/6100 [2:34:22<6:36:05,  5.41s/it]
 28%|██▊       | 1705/6100 [2:34:28<6:36:03,  5.41s/it]
                                                       
{'loss': 0.1091, 'learning_rate': 0.0001, 'epoch': 2.79}

 28%|██▊       | 1705/6100 [2:34:28<6:36:03,  5.41s/it]
 28%|██▊       | 1706/6100 [2:34:33<6:36:19,  5.41s/it]
 28%|██▊       | 1707/6100 [2:34:39<6:36:08,  5.41s/it]
 28%|██▊       | 1708/6100 [2:34:44<6:36:17,  5.41s/it]
 28%|██▊       | 1709/6100 [2:34:49<6:36:12,  5.41s/it]
 28%|██▊       | 1710/6100 [2:34:55<6:36:02,  5.41s/it]
                                                       
{'loss': 0.1141, 'learning_rate': 0.0001, 'epoch': 2.8}

 28%|██▊       | 1710/6100 [2:34:55<6:36:02,  5.41s/it]
 28%|██▊       | 1711/6100 [2:35:00<6:35:56,  5.41s/it]
 28%|██▊       | 1712/6100 [2:35:06<6:35:48,  5.41s/it]
 28%|██▊       | 1713/6100 [2:35:11<6:35:43,  5.41s/it]
 28%|██▊       | 1714/6100 [2:35:16<6:35:46,  5.41s/it]
 28%|██▊       | 1715/6100 [2:35:22<6:35:39,  5.41s/it]
                                                       
{'loss': 0.1049, 'learning_rate': 0.0001, 'epoch': 2.81}

 28%|██▊       | 1715/6100 [2:35:22<6:35:39,  5.41s/it]
 28%|██▊       | 1716/6100 [2:35:27<6:35:58,  5.42s/it]
 28%|██▊       | 1717/6100 [2:35:33<6:35:42,  5.42s/it]
 28%|██▊       | 1718/6100 [2:35:38<6:35:27,  5.41s/it]
 28%|██▊       | 1719/6100 [2:35:44<6:35:27,  5.42s/it]
 28%|██▊       | 1720/6100 [2:35:49<6:35:06,  5.41s/it]
                                                       
{'loss': 0.1057, 'learning_rate': 0.0001, 'epoch': 2.82}

 28%|██▊       | 1720/6100 [2:35:49<6:35:06,  5.41s/it]
 28%|██▊       | 1721/6100 [2:35:54<6:35:01,  5.41s/it]
 28%|██▊       | 1722/6100 [2:36:00<6:35:16,  5.42s/it]
 28%|██▊       | 1723/6100 [2:36:05<6:35:34,  5.42s/it]
 28%|██▊       | 1724/6100 [2:36:11<6:34:59,  5.42s/it]
 28%|██▊       | 1725/6100 [2:36:16<6:34:40,  5.41s/it]
                                                       
{'loss': 0.1067, 'learning_rate': 0.0001, 'epoch': 2.83}

 28%|██▊       | 1725/6100 [2:36:16<6:34:40,  5.41s/it]
 28%|██▊       | 1726/6100 [2:36:21<6:34:32,  5.41s/it]
 28%|██▊       | 1727/6100 [2:36:27<6:34:21,  5.41s/it]
 28%|██▊       | 1728/6100 [2:36:32<6:34:00,  5.41s/it]
 28%|██▊       | 1729/6100 [2:36:38<6:33:53,  5.41s/it]
 28%|██▊       | 1730/6100 [2:36:43<6:33:55,  5.41s/it]
                                                       
{'loss': 0.11, 'learning_rate': 0.0001, 'epoch': 2.84}

 28%|██▊       | 1730/6100 [2:36:43<6:33:55,  5.41s/it]
 28%|██▊       | 1731/6100 [2:36:48<6:33:39,  5.41s/it]
 28%|██▊       | 1732/6100 [2:36:54<6:33:52,  5.41s/it]
 28%|██▊       | 1733/6100 [2:36:59<6:33:32,  5.41s/it]
 28%|██▊       | 1734/6100 [2:37:05<6:33:30,  5.41s/it]
 28%|██▊       | 1735/6100 [2:37:10<6:33:24,  5.41s/it]
                                                       
{'loss': 0.103, 'learning_rate': 0.0001, 'epoch': 2.84}

 28%|██▊       | 1735/6100 [2:37:10<6:33:24,  5.41s/it]
 28%|██▊       | 1736/6100 [2:37:16<6:33:27,  5.41s/it]
 28%|██▊       | 1737/6100 [2:37:21<6:33:21,  5.41s/it]
 28%|██▊       | 1738/6100 [2:37:26<6:36:44,  5.46s/it]
 29%|██▊       | 1739/6100 [2:37:32<6:35:52,  5.45s/it]
 29%|██▊       | 1740/6100 [2:37:37<6:34:51,  5.43s/it]
                                                       
{'loss': 0.1093, 'learning_rate': 0.0001, 'epoch': 2.85}

 29%|██▊       | 1740/6100 [2:37:37<6:34:51,  5.43s/it]
 29%|██▊       | 1741/6100 [2:37:43<6:35:16,  5.44s/it]
 29%|██▊       | 1742/6100 [2:37:48<6:34:27,  5.43s/it]
 29%|██▊       | 1743/6100 [2:37:54<6:34:05,  5.43s/it]
 29%|██▊       | 1744/6100 [2:37:59<6:33:37,  5.42s/it]
 29%|██▊       | 1745/6100 [2:38:04<6:33:10,  5.42s/it]
                                                       
{'loss': 0.101, 'learning_rate': 0.0001, 'epoch': 2.86}

 29%|██▊       | 1745/6100 [2:38:04<6:33:10,  5.42s/it]
 29%|██▊       | 1746/6100 [2:38:10<6:32:46,  5.41s/it]
 29%|██▊       | 1747/6100 [2:38:15<6:32:32,  5.41s/it]
 29%|██▊       | 1748/6100 [2:38:21<6:32:44,  5.41s/it]
 29%|██▊       | 1749/6100 [2:38:26<6:32:49,  5.42s/it]
 29%|██▊       | 1750/6100 [2:38:31<6:32:27,  5.41s/it]
                                                       
{'loss': 0.1013, 'learning_rate': 0.0001, 'epoch': 2.87}

 29%|██▊       | 1750/6100 [2:38:31<6:32:27,  5.41s/it]
 29%|██▊       | 1751/6100 [2:38:37<6:32:42,  5.42s/it]
 29%|██▊       | 1752/6100 [2:38:42<6:32:26,  5.42s/it]
 29%|██▊       | 1753/6100 [2:38:48<6:32:12,  5.41s/it]
 29%|██▉       | 1754/6100 [2:38:53<6:32:16,  5.42s/it]
 29%|██▉       | 1755/6100 [2:38:59<6:31:56,  5.41s/it]
                                                       
{'loss': 0.097, 'learning_rate': 0.0001, 'epoch': 2.88}

 29%|██▉       | 1755/6100 [2:38:59<6:31:56,  5.41s/it]
 29%|██▉       | 1756/6100 [2:39:04<6:31:53,  5.41s/it]
 29%|██▉       | 1757/6100 [2:39:09<6:31:57,  5.42s/it]
 29%|██▉       | 1758/6100 [2:39:15<6:31:45,  5.41s/it]
 29%|██▉       | 1759/6100 [2:39:20<6:31:18,  5.41s/it]
 29%|██▉       | 1760/6100 [2:39:26<6:31:23,  5.41s/it]
                                                       
{'loss': 0.0933, 'learning_rate': 0.0001, 'epoch': 2.88}

 29%|██▉       | 1760/6100 [2:39:26<6:31:23,  5.41s/it]
 29%|██▉       | 1761/6100 [2:39:31<6:31:20,  5.41s/it]
 29%|██▉       | 1762/6100 [2:39:36<6:31:42,  5.42s/it]
 29%|██▉       | 1763/6100 [2:39:42<6:31:48,  5.42s/it]
 29%|██▉       | 1764/6100 [2:39:47<6:31:40,  5.42s/it]
 29%|██▉       | 1765/6100 [2:39:53<6:31:23,  5.42s/it]
                                                       
{'loss': 0.1049, 'learning_rate': 0.0001, 'epoch': 2.89}

 29%|██▉       | 1765/6100 [2:39:53<6:31:23,  5.42s/it]
 29%|██▉       | 1766/6100 [2:39:58<6:31:20,  5.42s/it]
 29%|██▉       | 1767/6100 [2:40:04<6:31:36,  5.42s/it]
 29%|██▉       | 1768/6100 [2:40:09<6:31:11,  5.42s/it]
 29%|██▉       | 1769/6100 [2:40:14<6:31:20,  5.42s/it]
 29%|██▉       | 1770/6100 [2:40:20<6:33:08,  5.45s/it]
                                                       
{'loss': 0.1075, 'learning_rate': 0.0001, 'epoch': 2.9}

 29%|██▉       | 1770/6100 [2:40:20<6:33:08,  5.45s/it]
 29%|██▉       | 1771/6100 [2:40:25<6:32:20,  5.44s/it]
 29%|██▉       | 1772/6100 [2:40:31<6:31:47,  5.43s/it]
 29%|██▉       | 1773/6100 [2:40:36<6:31:20,  5.43s/it]
 29%|██▉       | 1774/6100 [2:40:42<6:30:50,  5.42s/it]
 29%|██▉       | 1775/6100 [2:40:47<6:30:47,  5.42s/it]
                                                       
{'loss': 0.1065, 'learning_rate': 0.0001, 'epoch': 2.91}

 29%|██▉       | 1775/6100 [2:40:47<6:30:47,  5.42s/it]
 29%|██▉       | 1776/6100 [2:40:52<6:30:32,  5.42s/it]
 29%|██▉       | 1777/6100 [2:40:58<6:30:23,  5.42s/it]
 29%|██▉       | 1778/6100 [2:41:03<6:29:57,  5.41s/it]
 29%|██▉       | 1779/6100 [2:41:09<6:29:57,  5.41s/it]
 29%|██▉       | 1780/6100 [2:41:14<6:30:00,  5.42s/it]
                                                       
{'loss': 0.099, 'learning_rate': 0.0001, 'epoch': 2.92}

 29%|██▉       | 1780/6100 [2:41:14<6:30:00,  5.42s/it]
 29%|██▉       | 1781/6100 [2:41:19<6:29:44,  5.41s/it]
 29%|██▉       | 1782/6100 [2:41:25<6:29:31,  5.41s/it]
 29%|██▉       | 1783/6100 [2:41:30<6:29:19,  5.41s/it]
 29%|██▉       | 1784/6100 [2:41:36<6:29:22,  5.41s/it]
 29%|██▉       | 1785/6100 [2:41:41<6:29:28,  5.42s/it]
                                                       
{'loss': 0.0974, 'learning_rate': 0.0001, 'epoch': 2.93}

 29%|██▉       | 1785/6100 [2:41:41<6:29:28,  5.42s/it]
 29%|██▉       | 1786/6100 [2:41:47<6:29:21,  5.42s/it]
 29%|██▉       | 1787/6100 [2:41:52<6:29:01,  5.41s/it]
 29%|██▉       | 1788/6100 [2:41:57<6:28:55,  5.41s/it]
 29%|██▉       | 1789/6100 [2:42:03<6:28:49,  5.41s/it]
 29%|██▉       | 1790/6100 [2:42:08<6:28:45,  5.41s/it]
                                                       
{'loss': 0.0988, 'learning_rate': 0.0001, 'epoch': 2.93}

 29%|██▉       | 1790/6100 [2:42:08<6:28:45,  5.41s/it]
 29%|██▉       | 1791/6100 [2:42:14<6:29:16,  5.42s/it]
 29%|██▉       | 1792/6100 [2:42:19<6:29:13,  5.42s/it]
 29%|██▉       | 1793/6100 [2:42:24<6:29:17,  5.42s/it]
 29%|██▉       | 1794/6100 [2:42:30<6:28:53,  5.42s/it]
 29%|██▉       | 1795/6100 [2:42:35<6:28:39,  5.42s/it]
                                                       
{'loss': 0.1004, 'learning_rate': 0.0001, 'epoch': 2.94}

 29%|██▉       | 1795/6100 [2:42:35<6:28:39,  5.42s/it]
 29%|██▉       | 1796/6100 [2:42:41<6:28:48,  5.42s/it]
 29%|██▉       | 1797/6100 [2:42:46<6:28:15,  5.41s/it]
 29%|██▉       | 1798/6100 [2:42:52<6:28:04,  5.41s/it]
 29%|██▉       | 1799/6100 [2:42:57<6:28:05,  5.41s/it]
 30%|██▉       | 1800/6100 [2:43:03<6:43:07,  5.62s/it]
                                                       
{'loss': 0.0993, 'learning_rate': 0.0001, 'epoch': 2.95}

 30%|██▉       | 1800/6100 [2:43:03<6:43:07,  5.62s/it]Saving model checkpoint to ./results/checkpoint-1800
Configuration saved in ./results/checkpoint-1800/config.json
Model weights saved in ./results/checkpoint-1800/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-1800/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-1800/special_tokens_map.json
[2023-02-21 22:35:19,224] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step1801 is begin to save!
[2023-02-21 22:35:19,227] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-1800/global_step1801/mp_rank_00_model_states.pt
[2023-02-21 22:35:19,227] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-1800/global_step1801/mp_rank_00_model_states.pt...
[2023-02-21 22:35:20,044] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-1800/global_step1801/mp_rank_00_model_states.pt.
[2023-02-21 22:35:20,046] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-1800/global_step1801/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-21 22:35:20,222] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-1800/global_step1801/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-21 22:35:20,222] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-1800/global_step1801/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-21 22:35:20,222] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1801 is ready now!
Deleting older checkpoint [results/checkpoint-1650] due to args.save_total_limit

 30%|██▉       | 1801/6100 [2:43:11<7:24:26,  6.20s/it]
 30%|██▉       | 1802/6100 [2:43:16<7:07:15,  5.96s/it]
 30%|██▉       | 1803/6100 [2:43:21<6:55:10,  5.80s/it]
 30%|██▉       | 1804/6100 [2:43:27<6:46:53,  5.68s/it]
 30%|██▉       | 1805/6100 [2:43:32<6:40:46,  5.60s/it]
                                                       
{'loss': 0.0979, 'learning_rate': 0.0001, 'epoch': 2.96}

 30%|██▉       | 1805/6100 [2:43:32<6:40:46,  5.60s/it]
 30%|██▉       | 1806/6100 [2:43:38<6:36:27,  5.54s/it]
 30%|██▉       | 1807/6100 [2:43:43<6:33:34,  5.50s/it]
 30%|██▉       | 1808/6100 [2:43:48<6:31:37,  5.47s/it]
 30%|██▉       | 1809/6100 [2:43:54<6:30:17,  5.46s/it]
 30%|██▉       | 1810/6100 [2:43:59<6:29:13,  5.44s/it]
                                                       
{'loss': 0.1046, 'learning_rate': 0.0001, 'epoch': 2.97}

 30%|██▉       | 1810/6100 [2:43:59<6:29:13,  5.44s/it]
 30%|██▉       | 1811/6100 [2:44:05<6:28:43,  5.44s/it]
 30%|██▉       | 1812/6100 [2:44:10<6:28:07,  5.43s/it]
 30%|██▉       | 1813/6100 [2:44:16<6:27:27,  5.42s/it]
 30%|██▉       | 1814/6100 [2:44:21<6:26:58,  5.42s/it]
 30%|██▉       | 1815/6100 [2:44:26<6:27:00,  5.42s/it]
                                                       
{'loss': 0.1034, 'learning_rate': 0.0001, 'epoch': 2.97}

 30%|██▉       | 1815/6100 [2:44:26<6:27:00,  5.42s/it]
 30%|██▉       | 1816/6100 [2:44:32<6:26:52,  5.42s/it]
 30%|██▉       | 1817/6100 [2:44:37<6:26:26,  5.41s/it]
 30%|██▉       | 1818/6100 [2:44:43<6:26:15,  5.41s/it]
 30%|██▉       | 1819/6100 [2:44:48<6:26:03,  5.41s/it]
 30%|██▉       | 1820/6100 [2:44:53<6:25:56,  5.41s/it]
                                                       
{'loss': 0.0966, 'learning_rate': 0.0001, 'epoch': 2.98}

 30%|██▉       | 1820/6100 [2:44:53<6:25:56,  5.41s/it]
 30%|██▉       | 1821/6100 [2:44:59<6:25:53,  5.41s/it]
 30%|██▉       | 1822/6100 [2:45:04<6:25:43,  5.41s/it]
 30%|██▉       | 1823/6100 [2:45:10<6:25:43,  5.41s/it]
 30%|██▉       | 1824/6100 [2:45:15<6:25:26,  5.41s/it]
 30%|██▉       | 1825/6100 [2:45:20<6:25:22,  5.41s/it]
                                                       
{'loss': 0.1023, 'learning_rate': 0.0001, 'epoch': 2.99}

 30%|██▉       | 1825/6100 [2:45:20<6:25:22,  5.41s/it]
 30%|██▉       | 1826/6100 [2:45:26<6:25:18,  5.41s/it]
 30%|██▉       | 1827/6100 [2:45:31<6:25:15,  5.41s/it]
 30%|██▉       | 1828/6100 [2:45:37<6:25:12,  5.41s/it]
 30%|██▉       | 1829/6100 [2:45:42<6:25:19,  5.41s/it]
 30%|███       | 1830/6100 [2:45:48<6:25:05,  5.41s/it]
                                                       
{'loss': 0.1062, 'learning_rate': 0.0001, 'epoch': 3.0}

 30%|███       | 1830/6100 [2:45:48<6:25:05,  5.41s/it]
 30%|███       | 1831/6100 [2:45:56<7:20:37,  6.19s/it]
 30%|███       | 1832/6100 [2:46:01<7:03:54,  5.96s/it]
 30%|███       | 1833/6100 [2:46:06<6:52:04,  5.79s/it]
 30%|███       | 1834/6100 [2:46:12<6:43:53,  5.68s/it]
 30%|███       | 1835/6100 [2:46:17<6:37:53,  5.60s/it]
                                                       
{'loss': 0.1079, 'learning_rate': 0.0001, 'epoch': 3.01}

 30%|███       | 1835/6100 [2:46:17<6:37:53,  5.60s/it]
 30%|███       | 1836/6100 [2:46:23<6:34:12,  5.55s/it]
 30%|███       | 1837/6100 [2:46:28<6:31:24,  5.51s/it]
 30%|███       | 1838/6100 [2:46:33<6:29:19,  5.48s/it]
 30%|███       | 1839/6100 [2:46:39<6:27:43,  5.46s/it]
 30%|███       | 1840/6100 [2:46:44<6:26:41,  5.45s/it]
                                                       
{'loss': 0.0967, 'learning_rate': 0.0001, 'epoch': 3.02}

 30%|███       | 1840/6100 [2:46:44<6:26:41,  5.45s/it]
 30%|███       | 1841/6100 [2:46:50<6:26:04,  5.44s/it]
 30%|███       | 1842/6100 [2:46:55<6:25:06,  5.43s/it]
 30%|███       | 1843/6100 [2:47:00<6:24:34,  5.42s/it]
 30%|███       | 1844/6100 [2:47:06<6:24:06,  5.41s/it]
 30%|███       | 1845/6100 [2:47:11<6:24:07,  5.42s/it]
                                                       
{'loss': 0.0921, 'learning_rate': 0.0001, 'epoch': 3.02}

 30%|███       | 1845/6100 [2:47:11<6:24:07,  5.42s/it]
 30%|███       | 1846/6100 [2:47:17<6:24:26,  5.42s/it]
 30%|███       | 1847/6100 [2:47:22<6:24:23,  5.42s/it]
 30%|███       | 1848/6100 [2:47:28<6:24:02,  5.42s/it]
 30%|███       | 1849/6100 [2:47:33<6:24:16,  5.42s/it]
 30%|███       | 1850/6100 [2:47:38<6:23:53,  5.42s/it]
                                                       
{'loss': 0.1009, 'learning_rate': 0.0001, 'epoch': 3.03}

 30%|███       | 1850/6100 [2:47:38<6:23:53,  5.42s/it]
 30%|███       | 1851/6100 [2:47:44<6:23:37,  5.42s/it]
 30%|███       | 1852/6100 [2:47:49<6:23:30,  5.42s/it]
 30%|███       | 1853/6100 [2:47:55<6:23:19,  5.42s/it]
 30%|███       | 1854/6100 [2:48:00<6:23:17,  5.42s/it]
 30%|███       | 1855/6100 [2:48:05<6:23:08,  5.42s/it]
                                                       
{'loss': 0.095, 'learning_rate': 0.0001, 'epoch': 3.04}

 30%|███       | 1855/6100 [2:48:05<6:23:08,  5.42s/it]
 30%|███       | 1856/6100 [2:48:11<6:23:26,  5.42s/it]
 30%|███       | 1857/6100 [2:48:16<6:23:49,  5.43s/it]
 30%|███       | 1858/6100 [2:48:22<6:23:13,  5.42s/it]
 30%|███       | 1859/6100 [2:48:27<6:23:10,  5.42s/it]
 30%|███       | 1860/6100 [2:48:33<6:23:22,  5.43s/it]
                                                       
{'loss': 0.1015, 'learning_rate': 0.0001, 'epoch': 3.05}

 30%|███       | 1860/6100 [2:48:33<6:23:22,  5.43s/it]
 31%|███       | 1861/6100 [2:48:38<6:23:06,  5.42s/it]
 31%|███       | 1862/6100 [2:48:43<6:22:56,  5.42s/it]
 31%|███       | 1863/6100 [2:48:49<6:23:04,  5.42s/it]
 31%|███       | 1864/6100 [2:48:54<6:23:21,  5.43s/it]
 31%|███       | 1865/6100 [2:49:00<6:22:47,  5.42s/it]
                                                       
{'loss': 0.0975, 'learning_rate': 0.0001, 'epoch': 3.06}

 31%|███       | 1865/6100 [2:49:00<6:22:47,  5.42s/it]
 31%|███       | 1866/6100 [2:49:05<6:22:40,  5.42s/it]
 31%|███       | 1867/6100 [2:49:11<6:22:26,  5.42s/it]
 31%|███       | 1868/6100 [2:49:16<6:22:08,  5.42s/it]
 31%|███       | 1869/6100 [2:49:21<6:21:59,  5.42s/it]
 31%|███       | 1870/6100 [2:49:27<6:21:39,  5.41s/it]
                                                       
{'loss': 0.0928, 'learning_rate': 0.0001, 'epoch': 3.07}

 31%|███       | 1870/6100 [2:49:27<6:21:39,  5.41s/it]
 31%|███       | 1871/6100 [2:49:32<6:21:37,  5.41s/it]
 31%|███       | 1872/6100 [2:49:38<6:21:40,  5.42s/it]
 31%|███       | 1873/6100 [2:49:43<6:21:26,  5.41s/it]
 31%|███       | 1874/6100 [2:49:48<6:21:13,  5.41s/it]
 31%|███       | 1875/6100 [2:49:54<6:20:54,  5.41s/it]
                                                       
{'loss': 0.0971, 'learning_rate': 0.0001, 'epoch': 3.07}

 31%|███       | 1875/6100 [2:49:54<6:20:54,  5.41s/it]
 31%|███       | 1876/6100 [2:49:59<6:21:18,  5.42s/it]
 31%|███       | 1877/6100 [2:50:05<6:21:12,  5.42s/it]
 31%|███       | 1878/6100 [2:50:10<6:21:09,  5.42s/it]
 31%|███       | 1879/6100 [2:50:16<6:20:58,  5.42s/it]
 31%|███       | 1880/6100 [2:50:21<6:20:44,  5.41s/it]
                                                       
{'loss': 0.0957, 'learning_rate': 0.0001, 'epoch': 3.08}

 31%|███       | 1880/6100 [2:50:21<6:20:44,  5.41s/it]
 31%|███       | 1881/6100 [2:50:26<6:21:04,  5.42s/it]
 31%|███       | 1882/6100 [2:50:32<6:20:58,  5.42s/it]
 31%|███       | 1883/6100 [2:50:37<6:20:49,  5.42s/it]
 31%|███       | 1884/6100 [2:50:43<6:20:41,  5.42s/it]
 31%|███       | 1885/6100 [2:50:48<6:20:25,  5.42s/it]
                                                       
{'loss': 0.0987, 'learning_rate': 0.0001, 'epoch': 3.09}

 31%|███       | 1885/6100 [2:50:48<6:20:25,  5.42s/it]
 31%|███       | 1886/6100 [2:50:53<6:20:13,  5.41s/it]
 31%|███       | 1887/6100 [2:50:59<6:20:06,  5.41s/it]
 31%|███       | 1888/6100 [2:51:04<6:19:53,  5.41s/it]
 31%|███       | 1889/6100 [2:51:10<6:19:34,  5.41s/it]
 31%|███       | 1890/6100 [2:51:15<6:19:30,  5.41s/it]
                                                       
{'loss': 0.0957, 'learning_rate': 0.0001, 'epoch': 3.1}

 31%|███       | 1890/6100 [2:51:15<6:19:30,  5.41s/it]
 31%|███       | 1891/6100 [2:51:20<6:19:15,  5.41s/it]
 31%|███       | 1892/6100 [2:51:26<6:19:20,  5.41s/it]
 31%|███       | 1893/6100 [2:51:31<6:19:36,  5.41s/it]
 31%|███       | 1894/6100 [2:51:37<6:19:30,  5.41s/it]
 31%|███       | 1895/6100 [2:51:42<6:19:25,  5.41s/it]
                                                       
{'loss': 0.0962, 'learning_rate': 0.0001, 'epoch': 3.11}

 31%|███       | 1895/6100 [2:51:42<6:19:25,  5.41s/it]
 31%|███       | 1896/6100 [2:51:48<6:19:12,  5.41s/it]
 31%|███       | 1897/6100 [2:51:53<6:19:39,  5.42s/it]
 31%|███       | 1898/6100 [2:51:58<6:19:43,  5.42s/it]
 31%|███       | 1899/6100 [2:52:04<6:19:37,  5.42s/it]
 31%|███       | 1900/6100 [2:52:09<6:19:11,  5.42s/it]
                                                       
{'loss': 0.0996, 'learning_rate': 0.0001, 'epoch': 3.11}

 31%|███       | 1900/6100 [2:52:09<6:19:11,  5.42s/it]
 31%|███       | 1901/6100 [2:52:15<6:19:11,  5.42s/it]
 31%|███       | 1902/6100 [2:52:20<6:19:00,  5.42s/it]
 31%|███       | 1903/6100 [2:52:25<6:18:39,  5.41s/it]
 31%|███       | 1904/6100 [2:52:31<6:18:28,  5.41s/it]
 31%|███       | 1905/6100 [2:52:36<6:18:10,  5.41s/it]
                                                       
{'loss': 0.0914, 'learning_rate': 0.0001, 'epoch': 3.12}

 31%|███       | 1905/6100 [2:52:36<6:18:10,  5.41s/it]
 31%|███       | 1906/6100 [2:52:42<6:18:16,  5.41s/it]
 31%|███▏      | 1907/6100 [2:52:47<6:18:11,  5.41s/it]
 31%|███▏      | 1908/6100 [2:52:53<6:18:40,  5.42s/it]
 31%|███▏      | 1909/6100 [2:52:58<6:18:13,  5.41s/it]
 31%|███▏      | 1910/6100 [2:53:03<6:18:10,  5.42s/it]
                                                       
{'loss': 0.0942, 'learning_rate': 0.0001, 'epoch': 3.13}

 31%|███▏      | 1910/6100 [2:53:03<6:18:10,  5.42s/it]
 31%|███▏      | 1911/6100 [2:53:09<6:17:51,  5.41s/it]
 31%|███▏      | 1912/6100 [2:53:14<6:17:49,  5.41s/it]
 31%|███▏      | 1913/6100 [2:53:20<6:17:39,  5.41s/it]
 31%|███▏      | 1914/6100 [2:53:25<6:17:31,  5.41s/it]
 31%|███▏      | 1915/6100 [2:53:30<6:17:21,  5.41s/it]
                                                       
{'loss': 0.0948, 'learning_rate': 0.0001, 'epoch': 3.14}

 31%|███▏      | 1915/6100 [2:53:30<6:17:21,  5.41s/it]
 31%|███▏      | 1916/6100 [2:53:36<6:17:28,  5.41s/it]
 31%|███▏      | 1917/6100 [2:53:41<6:17:18,  5.41s/it]
 31%|███▏      | 1918/6100 [2:53:47<6:17:08,  5.41s/it]
 31%|███▏      | 1919/6100 [2:53:52<6:17:00,  5.41s/it]
 31%|███▏      | 1920/6100 [2:53:58<6:17:30,  5.42s/it]
                                                       
{'loss': 0.0923, 'learning_rate': 0.0001, 'epoch': 3.15}

 31%|███▏      | 1920/6100 [2:53:58<6:17:30,  5.42s/it]
 31%|███▏      | 1921/6100 [2:54:03<6:17:14,  5.42s/it]
 32%|███▏      | 1922/6100 [2:54:08<6:17:11,  5.42s/it]
 32%|███▏      | 1923/6100 [2:54:14<6:16:55,  5.41s/it]
 32%|███▏      | 1924/6100 [2:54:19<6:17:30,  5.42s/it]
 32%|███▏      | 1925/6100 [2:54:25<6:17:18,  5.42s/it]
                                                       
{'loss': 0.0963, 'learning_rate': 0.0001, 'epoch': 3.16}

 32%|███▏      | 1925/6100 [2:54:25<6:17:18,  5.42s/it]
 32%|███▏      | 1926/6100 [2:54:30<6:17:06,  5.42s/it]
 32%|███▏      | 1927/6100 [2:54:35<6:17:07,  5.42s/it]
 32%|███▏      | 1928/6100 [2:54:41<6:16:39,  5.42s/it]
 32%|███▏      | 1929/6100 [2:54:46<6:16:28,  5.42s/it]
 32%|███▏      | 1930/6100 [2:54:52<6:16:02,  5.41s/it]
                                                       
{'loss': 0.0985, 'learning_rate': 0.0001, 'epoch': 3.16}

 32%|███▏      | 1930/6100 [2:54:52<6:16:02,  5.41s/it]
 32%|███▏      | 1931/6100 [2:54:57<6:16:29,  5.42s/it]
 32%|███▏      | 1932/6100 [2:55:03<6:16:13,  5.42s/it]
 32%|███▏      | 1933/6100 [2:55:08<6:15:55,  5.41s/it]
 32%|███▏      | 1934/6100 [2:55:13<6:15:32,  5.41s/it]
 32%|███▏      | 1935/6100 [2:55:19<6:15:14,  5.41s/it]
                                                       
{'loss': 0.0941, 'learning_rate': 0.0001, 'epoch': 3.17}

 32%|███▏      | 1935/6100 [2:55:19<6:15:14,  5.41s/it]
 32%|███▏      | 1936/6100 [2:55:24<6:15:18,  5.41s/it]
 32%|███▏      | 1937/6100 [2:55:30<6:15:57,  5.42s/it]
 32%|███▏      | 1938/6100 [2:55:35<6:15:41,  5.42s/it]
 32%|███▏      | 1939/6100 [2:55:40<6:15:28,  5.41s/it]
 32%|███▏      | 1940/6100 [2:55:46<6:15:19,  5.41s/it]
                                                       
{'loss': 0.0908, 'learning_rate': 0.0001, 'epoch': 3.18}

 32%|███▏      | 1940/6100 [2:55:46<6:15:19,  5.41s/it]
 32%|███▏      | 1941/6100 [2:55:51<6:15:20,  5.41s/it]
 32%|███▏      | 1942/6100 [2:55:57<6:14:58,  5.41s/it]
 32%|███▏      | 1943/6100 [2:56:02<6:14:54,  5.41s/it]
 32%|███▏      | 1944/6100 [2:56:07<6:14:46,  5.41s/it]
 32%|███▏      | 1945/6100 [2:56:13<6:14:29,  5.41s/it]
                                                       
{'loss': 0.0945, 'learning_rate': 0.0001, 'epoch': 3.19}

 32%|███▏      | 1945/6100 [2:56:13<6:14:29,  5.41s/it]
 32%|███▏      | 1946/6100 [2:56:18<6:14:43,  5.41s/it]
 32%|███▏      | 1947/6100 [2:56:24<6:14:29,  5.41s/it]
 32%|███▏      | 1948/6100 [2:56:29<6:14:26,  5.41s/it]
 32%|███▏      | 1949/6100 [2:56:35<6:14:22,  5.41s/it]
 32%|███▏      | 1950/6100 [2:56:40<6:13:59,  5.41s/it]
                                                       
{'loss': 0.0922, 'learning_rate': 0.0001, 'epoch': 3.2}

 32%|███▏      | 1950/6100 [2:56:40<6:13:59,  5.41s/it]Saving model checkpoint to ./results/checkpoint-1950
Configuration saved in ./results/checkpoint-1950/config.json
Model weights saved in ./results/checkpoint-1950/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-1950/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-1950/special_tokens_map.json
[2023-02-21 22:48:56,111] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step1951 is begin to save!
[2023-02-21 22:48:56,115] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-1950/global_step1951/mp_rank_00_model_states.pt
[2023-02-21 22:48:56,115] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-1950/global_step1951/mp_rank_00_model_states.pt...
[2023-02-21 22:48:56,937] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-1950/global_step1951/mp_rank_00_model_states.pt.
[2023-02-21 22:48:56,939] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-1950/global_step1951/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-21 22:48:57,154] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-1950/global_step1951/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-21 22:48:57,155] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-1950/global_step1951/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-21 22:48:57,155] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1951 is ready now!
Deleting older checkpoint [results/checkpoint-1800] due to args.save_total_limit

 32%|███▏      | 1951/6100 [2:56:48<6:59:08,  6.06s/it]
 32%|███▏      | 1952/6100 [2:56:53<6:45:23,  5.86s/it]
 32%|███▏      | 1953/6100 [2:56:58<6:35:41,  5.72s/it]
 32%|███▏      | 1954/6100 [2:57:04<6:29:09,  5.63s/it]
 32%|███▏      | 1955/6100 [2:57:09<6:24:35,  5.57s/it]
                                                       
{'loss': 0.0943, 'learning_rate': 0.0001, 'epoch': 3.2}

 32%|███▏      | 1955/6100 [2:57:09<6:24:35,  5.57s/it]
 32%|███▏      | 1956/6100 [2:57:15<6:21:26,  5.52s/it]
 32%|███▏      | 1957/6100 [2:57:20<6:18:40,  5.48s/it]
 32%|███▏      | 1958/6100 [2:57:25<6:16:59,  5.46s/it]
 32%|███▏      | 1959/6100 [2:57:31<6:15:40,  5.44s/it]
 32%|███▏      | 1960/6100 [2:57:36<6:14:54,  5.43s/it]
                                                       
{'loss': 0.0925, 'learning_rate': 0.0001, 'epoch': 3.21}

 32%|███▏      | 1960/6100 [2:57:36<6:14:54,  5.43s/it]
 32%|███▏      | 1961/6100 [2:57:42<6:14:26,  5.43s/it]
 32%|███▏      | 1962/6100 [2:57:47<6:13:49,  5.42s/it]
 32%|███▏      | 1963/6100 [2:57:52<6:13:27,  5.42s/it]
 32%|███▏      | 1964/6100 [2:57:58<6:13:07,  5.41s/it]
 32%|███▏      | 1965/6100 [2:58:03<6:13:16,  5.42s/it]
                                                       
{'loss': 0.0936, 'learning_rate': 0.0001, 'epoch': 3.22}

 32%|███▏      | 1965/6100 [2:58:03<6:13:16,  5.42s/it]
 32%|███▏      | 1966/6100 [2:58:09<6:13:18,  5.42s/it]
 32%|███▏      | 1967/6100 [2:58:14<6:12:51,  5.41s/it]
 32%|███▏      | 1968/6100 [2:58:19<6:12:27,  5.41s/it]
 32%|███▏      | 1969/6100 [2:58:25<6:17:30,  5.48s/it]
 32%|███▏      | 1970/6100 [2:58:31<6:15:39,  5.46s/it]
                                                       
{'loss': 0.096, 'learning_rate': 0.0001, 'epoch': 3.23}

 32%|███▏      | 1970/6100 [2:58:31<6:15:39,  5.46s/it]
 32%|███▏      | 1971/6100 [2:58:36<6:14:42,  5.44s/it]
 32%|███▏      | 1972/6100 [2:58:41<6:13:51,  5.43s/it]
 32%|███▏      | 1973/6100 [2:58:47<6:13:08,  5.42s/it]
 32%|███▏      | 1974/6100 [2:58:52<6:12:42,  5.42s/it]
 32%|███▏      | 1975/6100 [2:58:58<6:12:21,  5.42s/it]
                                                       
{'loss': 0.0905, 'learning_rate': 0.0001, 'epoch': 3.24}

 32%|███▏      | 1975/6100 [2:58:58<6:12:21,  5.42s/it]
 32%|███▏      | 1976/6100 [2:59:03<6:12:00,  5.41s/it]
 32%|███▏      | 1977/6100 [2:59:08<6:12:25,  5.42s/it]
 32%|███▏      | 1978/6100 [2:59:14<6:11:57,  5.41s/it]
 32%|███▏      | 1979/6100 [2:59:19<6:11:51,  5.41s/it]
 32%|███▏      | 1980/6100 [2:59:25<6:11:51,  5.42s/it]
                                                       
{'loss': 0.095, 'learning_rate': 0.0001, 'epoch': 3.25}

 32%|███▏      | 1980/6100 [2:59:25<6:11:51,  5.42s/it]
 32%|███▏      | 1981/6100 [2:59:30<6:11:48,  5.42s/it]
 32%|███▏      | 1982/6100 [2:59:35<6:11:37,  5.41s/it]
 33%|███▎      | 1983/6100 [2:59:41<6:11:22,  5.41s/it]
 33%|███▎      | 1984/6100 [2:59:46<6:11:00,  5.41s/it]
 33%|███▎      | 1985/6100 [2:59:52<6:10:57,  5.41s/it]
                                                       
{'loss': 0.0997, 'learning_rate': 0.0001, 'epoch': 3.25}

 33%|███▎      | 1985/6100 [2:59:52<6:10:57,  5.41s/it]
 33%|███▎      | 1986/6100 [2:59:57<6:11:03,  5.41s/it]
 33%|███▎      | 1987/6100 [3:00:03<6:11:57,  5.43s/it]
 33%|███▎      | 1988/6100 [3:00:08<6:11:37,  5.42s/it]
 33%|███▎      | 1989/6100 [3:00:13<6:11:30,  5.42s/it]
 33%|███▎      | 1990/6100 [3:00:19<6:11:04,  5.42s/it]
                                                       
{'loss': 0.0929, 'learning_rate': 0.0001, 'epoch': 3.26}

 33%|███▎      | 1990/6100 [3:00:19<6:11:04,  5.42s/it]
 33%|███▎      | 1991/6100 [3:00:24<6:10:49,  5.41s/it]
 33%|███▎      | 1992/6100 [3:00:30<6:10:51,  5.42s/it]
 33%|███▎      | 1993/6100 [3:00:35<6:10:46,  5.42s/it]
 33%|███▎      | 1994/6100 [3:00:40<6:10:28,  5.41s/it]
 33%|███▎      | 1995/6100 [3:00:46<6:10:43,  5.42s/it]
                                                       
{'loss': 0.091, 'learning_rate': 0.0001, 'epoch': 3.27}

 33%|███▎      | 1995/6100 [3:00:46<6:10:43,  5.42s/it]
 33%|███▎      | 1996/6100 [3:00:51<6:10:37,  5.42s/it]
 33%|███▎      | 1997/6100 [3:00:57<6:10:20,  5.42s/it]
 33%|███▎      | 1998/6100 [3:01:02<6:10:24,  5.42s/it][2023-02-21 22:53:20,140] [INFO] [logging.py:75:log_dist] [Rank 0] step=2000, skipped=0, lr=[0.0001], mom=[[0.9, 0.999]]
[2023-02-21 22:53:20,205] [INFO] [timer.py:198:stop] epoch=0/micro_step=8000/global_step=2000, RunningAvgSamplesPerSec=35.86774677569224, CurrSamplesPerSec=35.90280932691704, MemAllocated=0.66GB, MaxMemAllocated=36.72GB

 33%|███▎      | 1999/6100 [3:01:08<6:10:14,  5.42s/it]
 33%|███▎      | 2000/6100 [3:01:13<6:10:03,  5.42s/it]
                                                       
{'loss': 0.0935, 'learning_rate': 0.0001, 'epoch': 3.28}

 33%|███▎      | 2000/6100 [3:01:13<6:10:03,  5.42s/it]
 33%|███▎      | 2001/6100 [3:01:18<6:09:56,  5.42s/it]
 33%|███▎      | 2002/6100 [3:01:24<6:09:36,  5.41s/it]
 33%|███▎      | 2003/6100 [3:01:29<6:09:30,  5.41s/it]
 33%|███▎      | 2004/6100 [3:01:35<6:09:37,  5.41s/it]
 33%|███▎      | 2005/6100 [3:01:40<6:09:25,  5.41s/it]
                                                       
{'loss': 0.0892, 'learning_rate': 0.0001, 'epoch': 3.29}

 33%|███▎      | 2005/6100 [3:01:40<6:09:25,  5.41s/it]
 33%|███▎      | 2006/6100 [3:01:45<6:09:38,  5.42s/it]
 33%|███▎      | 2007/6100 [3:01:51<6:09:58,  5.42s/it]
 33%|███▎      | 2008/6100 [3:01:56<6:09:21,  5.42s/it]
 33%|███▎      | 2009/6100 [3:02:02<6:09:01,  5.41s/it]
 33%|███▎      | 2010/6100 [3:02:07<6:09:00,  5.41s/it]
                                                       
{'loss': 0.0949, 'learning_rate': 0.0001, 'epoch': 3.29}

 33%|███▎      | 2010/6100 [3:02:07<6:09:00,  5.41s/it]
 33%|███▎      | 2011/6100 [3:02:13<6:09:05,  5.42s/it]
 33%|███▎      | 2012/6100 [3:02:18<6:09:01,  5.42s/it]
 33%|███▎      | 2013/6100 [3:02:24<6:12:43,  5.47s/it]
 33%|███▎      | 2014/6100 [3:02:29<6:11:16,  5.45s/it]
 33%|███▎      | 2015/6100 [3:02:34<6:10:38,  5.44s/it]
                                                       
{'loss': 0.0967, 'learning_rate': 0.0001, 'epoch': 3.3}

 33%|███▎      | 2015/6100 [3:02:34<6:10:38,  5.44s/it]
 33%|███▎      | 2016/6100 [3:02:40<6:09:53,  5.43s/it]
 33%|███▎      | 2017/6100 [3:02:45<6:09:23,  5.43s/it]
 33%|███▎      | 2018/6100 [3:02:51<6:09:01,  5.42s/it]
 33%|███▎      | 2019/6100 [3:02:56<6:08:38,  5.42s/it]
 33%|███▎      | 2020/6100 [3:03:01<6:08:20,  5.42s/it]
                                                       
{'loss': 0.093, 'learning_rate': 0.0001, 'epoch': 3.31}

 33%|███▎      | 2020/6100 [3:03:01<6:08:20,  5.42s/it]
 33%|███▎      | 2021/6100 [3:03:07<6:08:02,  5.41s/it]
 33%|███▎      | 2022/6100 [3:03:12<6:07:49,  5.41s/it]
 33%|███▎      | 2023/6100 [3:03:18<6:08:00,  5.42s/it]
 33%|███▎      | 2024/6100 [3:03:23<6:07:59,  5.42s/it]
 33%|███▎      | 2025/6100 [3:03:29<6:09:46,  5.44s/it]
                                                       
{'loss': 0.095, 'learning_rate': 0.0001, 'epoch': 3.32}

 33%|███▎      | 2025/6100 [3:03:29<6:09:46,  5.44s/it]
 33%|███▎      | 2026/6100 [3:03:34<6:09:07,  5.44s/it]
 33%|███▎      | 2027/6100 [3:03:39<6:08:29,  5.43s/it]
 33%|███▎      | 2028/6100 [3:03:45<6:08:12,  5.43s/it]
 33%|███▎      | 2029/6100 [3:03:50<6:07:37,  5.42s/it]
 33%|███▎      | 2030/6100 [3:03:56<6:07:29,  5.42s/it]
                                                       
{'loss': 0.0918, 'learning_rate': 0.0001, 'epoch': 3.33}

 33%|███▎      | 2030/6100 [3:03:56<6:07:29,  5.42s/it]
 33%|███▎      | 2031/6100 [3:04:01<6:07:06,  5.41s/it]
 33%|███▎      | 2032/6100 [3:04:06<6:06:58,  5.41s/it]
 33%|███▎      | 2033/6100 [3:04:12<6:06:42,  5.41s/it]
 33%|███▎      | 2034/6100 [3:04:17<6:06:29,  5.41s/it]
 33%|███▎      | 2035/6100 [3:04:23<6:06:21,  5.41s/it]
                                                       
{'loss': 0.0904, 'learning_rate': 0.0001, 'epoch': 3.34}

 33%|███▎      | 2035/6100 [3:04:23<6:06:21,  5.41s/it]
 33%|███▎      | 2036/6100 [3:04:28<6:06:17,  5.41s/it]
 33%|███▎      | 2037/6100 [3:04:33<6:05:56,  5.40s/it]
 33%|███▎      | 2038/6100 [3:04:39<6:06:02,  5.41s/it]
 33%|███▎      | 2039/6100 [3:04:44<6:06:17,  5.41s/it]
 33%|███▎      | 2040/6100 [3:04:50<6:05:57,  5.41s/it]
                                                       
{'loss': 0.09, 'learning_rate': 0.0001, 'epoch': 3.34}

 33%|███▎      | 2040/6100 [3:04:50<6:05:57,  5.41s/it]
 33%|███▎      | 2041/6100 [3:04:55<6:06:09,  5.41s/it]
 33%|███▎      | 2042/6100 [3:05:01<6:06:24,  5.42s/it]
 33%|███▎      | 2043/6100 [3:05:06<6:06:17,  5.42s/it]
 34%|███▎      | 2044/6100 [3:05:11<6:06:06,  5.42s/it]
 34%|███▎      | 2045/6100 [3:05:17<6:05:59,  5.42s/it]
                                                       
{'loss': 0.0934, 'learning_rate': 0.0001, 'epoch': 3.35}

 34%|███▎      | 2045/6100 [3:05:17<6:05:59,  5.42s/it]
 34%|███▎      | 2046/6100 [3:05:22<6:05:56,  5.42s/it]
 34%|███▎      | 2047/6100 [3:05:28<6:05:35,  5.41s/it]
 34%|███▎      | 2048/6100 [3:05:33<6:05:16,  5.41s/it]
 34%|███▎      | 2049/6100 [3:05:38<6:05:22,  5.41s/it]
 34%|███▎      | 2050/6100 [3:05:44<6:05:08,  5.41s/it]
                                                       
{'loss': 0.0972, 'learning_rate': 0.0001, 'epoch': 3.36}

 34%|███▎      | 2050/6100 [3:05:44<6:05:08,  5.41s/it]
 34%|███▎      | 2051/6100 [3:05:49<6:04:52,  5.41s/it]
 34%|███▎      | 2052/6100 [3:05:55<6:04:56,  5.41s/it]
 34%|███▎      | 2053/6100 [3:06:00<6:04:53,  5.41s/it]
 34%|███▎      | 2054/6100 [3:06:06<6:04:56,  5.41s/it]
 34%|███▎      | 2055/6100 [3:06:11<6:04:49,  5.41s/it]
                                                       
{'loss': 0.0878, 'learning_rate': 0.0001, 'epoch': 3.37}

 34%|███▎      | 2055/6100 [3:06:11<6:04:49,  5.41s/it]
 34%|███▎      | 2056/6100 [3:06:16<6:04:41,  5.41s/it]
 34%|███▎      | 2057/6100 [3:06:22<6:04:45,  5.41s/it]
 34%|███▎      | 2058/6100 [3:06:27<6:04:40,  5.41s/it]
 34%|███▍      | 2059/6100 [3:06:33<6:04:28,  5.41s/it]
 34%|███▍      | 2060/6100 [3:06:38<6:04:33,  5.41s/it]
                                                       
{'loss': 0.0875, 'learning_rate': 0.0001, 'epoch': 3.38}

 34%|███▍      | 2060/6100 [3:06:38<6:04:33,  5.41s/it]
 34%|███▍      | 2061/6100 [3:06:43<6:04:42,  5.42s/it]
 34%|███▍      | 2062/6100 [3:06:49<6:04:46,  5.42s/it]
 34%|███▍      | 2063/6100 [3:06:54<6:04:32,  5.42s/it]
 34%|███▍      | 2064/6100 [3:07:00<6:04:29,  5.42s/it]
 34%|███▍      | 2065/6100 [3:07:05<6:04:23,  5.42s/it]
                                                       
{'loss': 0.0951, 'learning_rate': 0.0001, 'epoch': 3.38}

 34%|███▍      | 2065/6100 [3:07:05<6:04:23,  5.42s/it]
 34%|███▍      | 2066/6100 [3:07:11<6:04:18,  5.42s/it]
 34%|███▍      | 2067/6100 [3:07:16<6:04:06,  5.42s/it]
 34%|███▍      | 2068/6100 [3:07:21<6:03:54,  5.42s/it]
 34%|███▍      | 2069/6100 [3:07:27<6:04:01,  5.42s/it]
 34%|███▍      | 2070/6100 [3:07:32<6:03:46,  5.42s/it]
                                                       
{'loss': 0.0901, 'learning_rate': 0.0001, 'epoch': 3.39}

 34%|███▍      | 2070/6100 [3:07:32<6:03:46,  5.42s/it]
 34%|███▍      | 2071/6100 [3:07:38<6:03:45,  5.42s/it]
 34%|███▍      | 2072/6100 [3:07:43<6:03:36,  5.42s/it]
 34%|███▍      | 2073/6100 [3:07:48<6:03:26,  5.42s/it]
 34%|███▍      | 2074/6100 [3:07:54<6:03:20,  5.41s/it]
 34%|███▍      | 2075/6100 [3:07:59<6:03:15,  5.41s/it]
                                                       
{'loss': 0.095, 'learning_rate': 0.0001, 'epoch': 3.4}

 34%|███▍      | 2075/6100 [3:07:59<6:03:15,  5.41s/it]
 34%|███▍      | 2076/6100 [3:08:05<6:03:09,  5.41s/it]
 34%|███▍      | 2077/6100 [3:08:10<6:03:08,  5.42s/it]
 34%|███▍      | 2078/6100 [3:08:15<6:02:54,  5.41s/it]
 34%|███▍      | 2079/6100 [3:08:21<6:02:44,  5.41s/it]
 34%|███▍      | 2080/6100 [3:08:26<6:02:32,  5.41s/it]
                                                       
{'loss': 0.0894, 'learning_rate': 0.0001, 'epoch': 3.41}

 34%|███▍      | 2080/6100 [3:08:26<6:02:32,  5.41s/it]
 34%|███▍      | 2081/6100 [3:08:32<6:02:26,  5.41s/it]
 34%|███▍      | 2082/6100 [3:08:37<6:02:30,  5.41s/it]
 34%|███▍      | 2083/6100 [3:08:43<6:02:21,  5.41s/it]
 34%|███▍      | 2084/6100 [3:08:48<6:02:21,  5.41s/it]
 34%|███▍      | 2085/6100 [3:08:53<6:02:35,  5.42s/it]
                                                       
{'loss': 0.0956, 'learning_rate': 0.0001, 'epoch': 3.42}

 34%|███▍      | 2085/6100 [3:08:53<6:02:35,  5.42s/it]
 34%|███▍      | 2086/6100 [3:08:59<6:02:17,  5.42s/it]
 34%|███▍      | 2087/6100 [3:09:04<6:02:10,  5.42s/it]
 34%|███▍      | 2088/6100 [3:09:10<6:02:03,  5.41s/it]
 34%|███▍      | 2089/6100 [3:09:15<6:01:55,  5.41s/it]
 34%|███▍      | 2090/6100 [3:09:20<6:01:37,  5.41s/it]
                                                       
{'loss': 0.0942, 'learning_rate': 0.0001, 'epoch': 3.43}

 34%|███▍      | 2090/6100 [3:09:20<6:01:37,  5.41s/it]
 34%|███▍      | 2091/6100 [3:09:26<6:01:29,  5.41s/it]
 34%|███▍      | 2092/6100 [3:09:31<6:01:25,  5.41s/it]
 34%|███▍      | 2093/6100 [3:09:37<6:01:24,  5.41s/it]
 34%|███▍      | 2094/6100 [3:09:42<6:01:22,  5.41s/it]
 34%|███▍      | 2095/6100 [3:09:47<6:01:11,  5.41s/it]
                                                       
{'loss': 0.0902, 'learning_rate': 0.0001, 'epoch': 3.43}

 34%|███▍      | 2095/6100 [3:09:48<6:01:11,  5.41s/it]
 34%|███▍      | 2096/6100 [3:09:53<6:01:12,  5.41s/it]
 34%|███▍      | 2097/6100 [3:09:58<6:01:20,  5.42s/it]
 34%|███▍      | 2098/6100 [3:10:04<6:01:13,  5.42s/it]
 34%|███▍      | 2099/6100 [3:10:09<6:01:09,  5.42s/it]
 34%|███▍      | 2100/6100 [3:10:15<6:00:47,  5.41s/it]
                                                       
{'loss': 0.098, 'learning_rate': 0.0001, 'epoch': 3.44}

 34%|███▍      | 2100/6100 [3:10:15<6:00:47,  5.41s/it]Saving model checkpoint to ./results/checkpoint-2100
Configuration saved in ./results/checkpoint-2100/config.json
Model weights saved in ./results/checkpoint-2100/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-2100/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-2100/special_tokens_map.json
[2023-02-21 23:02:30,739] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step2101 is begin to save!
[2023-02-21 23:02:30,742] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-2100/global_step2101/mp_rank_00_model_states.pt
[2023-02-21 23:02:30,742] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-2100/global_step2101/mp_rank_00_model_states.pt...
[2023-02-21 23:02:31,558] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-2100/global_step2101/mp_rank_00_model_states.pt.
[2023-02-21 23:02:31,560] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-2100/global_step2101/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-21 23:02:31,771] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-2100/global_step2101/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-21 23:02:31,771] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-2100/global_step2101/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-21 23:02:31,771] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2101 is ready now!
Deleting older checkpoint [results/checkpoint-1950] due to args.save_total_limit

 34%|███▍      | 2101/6100 [3:10:22<6:46:40,  6.10s/it]
 34%|███▍      | 2102/6100 [3:10:28<6:32:37,  5.89s/it]
 34%|███▍      | 2103/6100 [3:10:33<6:22:46,  5.75s/it]
 34%|███▍      | 2104/6100 [3:10:38<6:15:45,  5.64s/it]
 35%|███▍      | 2105/6100 [3:10:44<6:10:57,  5.57s/it]
                                                       
{'loss': 0.093, 'learning_rate': 0.0001, 'epoch': 3.45}

 35%|███▍      | 2105/6100 [3:10:44<6:10:57,  5.57s/it]
 35%|███▍      | 2106/6100 [3:10:49<6:07:29,  5.52s/it]
 35%|███▍      | 2107/6100 [3:10:55<6:04:59,  5.48s/it]
 35%|███▍      | 2108/6100 [3:11:00<6:03:17,  5.46s/it]
 35%|███▍      | 2109/6100 [3:11:06<6:02:19,  5.45s/it]
 35%|███▍      | 2110/6100 [3:11:11<6:01:38,  5.44s/it]
                                                       
{'loss': 0.0911, 'learning_rate': 0.0001, 'epoch': 3.46}

 35%|███▍      | 2110/6100 [3:11:11<6:01:38,  5.44s/it]
 35%|███▍      | 2111/6100 [3:11:16<6:01:05,  5.43s/it]
 35%|███▍      | 2112/6100 [3:11:22<6:00:22,  5.42s/it]
 35%|███▍      | 2113/6100 [3:11:27<5:59:50,  5.42s/it]
 35%|███▍      | 2114/6100 [3:11:33<5:59:39,  5.41s/it]
 35%|███▍      | 2115/6100 [3:11:38<5:59:08,  5.41s/it]
                                                       
{'loss': 0.0865, 'learning_rate': 0.0001, 'epoch': 3.47}

 35%|███▍      | 2115/6100 [3:11:38<5:59:08,  5.41s/it]
 35%|███▍      | 2116/6100 [3:11:43<5:58:55,  5.41s/it]
 35%|███▍      | 2117/6100 [3:11:49<5:58:58,  5.41s/it]
 35%|███▍      | 2118/6100 [3:11:54<5:58:33,  5.40s/it]
 35%|███▍      | 2119/6100 [3:12:00<5:58:39,  5.41s/it]
 35%|███▍      | 2120/6100 [3:12:05<5:58:34,  5.41s/it]
                                                       
{'loss': 0.0913, 'learning_rate': 0.0001, 'epoch': 3.48}

 35%|███▍      | 2120/6100 [3:12:05<5:58:34,  5.41s/it]
 35%|███▍      | 2121/6100 [3:12:10<5:58:56,  5.41s/it]
 35%|███▍      | 2122/6100 [3:12:16<5:58:51,  5.41s/it]
 35%|███▍      | 2123/6100 [3:12:21<5:58:29,  5.41s/it]
 35%|███▍      | 2124/6100 [3:12:27<5:58:33,  5.41s/it]
 35%|███▍      | 2125/6100 [3:12:32<5:58:16,  5.41s/it]
                                                       
{'loss': 0.0891, 'learning_rate': 0.0001, 'epoch': 3.48}

 35%|███▍      | 2125/6100 [3:12:32<5:58:16,  5.41s/it]
 35%|███▍      | 2126/6100 [3:12:37<5:58:06,  5.41s/it]
 35%|███▍      | 2127/6100 [3:12:43<5:58:03,  5.41s/it]
 35%|███▍      | 2128/6100 [3:12:48<5:57:56,  5.41s/it]
 35%|███▍      | 2129/6100 [3:12:54<5:57:54,  5.41s/it]
 35%|███▍      | 2130/6100 [3:12:59<5:57:55,  5.41s/it]
                                                       
{'loss': 0.0947, 'learning_rate': 0.0001, 'epoch': 3.49}

 35%|███▍      | 2130/6100 [3:12:59<5:57:55,  5.41s/it]
 35%|███▍      | 2131/6100 [3:13:04<5:57:47,  5.41s/it]
 35%|███▍      | 2132/6100 [3:13:10<5:57:29,  5.41s/it]
 35%|███▍      | 2133/6100 [3:13:15<5:57:27,  5.41s/it]
 35%|███▍      | 2134/6100 [3:13:21<5:57:23,  5.41s/it]
 35%|███▌      | 2135/6100 [3:13:26<5:57:35,  5.41s/it]
                                                       
{'loss': 0.0881, 'learning_rate': 0.0001, 'epoch': 3.5}

 35%|███▌      | 2135/6100 [3:13:26<5:57:35,  5.41s/it]
 35%|███▌      | 2136/6100 [3:13:32<5:57:26,  5.41s/it]
 35%|███▌      | 2137/6100 [3:13:37<5:57:24,  5.41s/it]
 35%|███▌      | 2138/6100 [3:13:42<5:57:00,  5.41s/it]
 35%|███▌      | 2139/6100 [3:13:48<5:57:13,  5.41s/it]
 35%|███▌      | 2140/6100 [3:13:53<5:57:22,  5.41s/it]
                                                       
{'loss': 0.0844, 'learning_rate': 0.0001, 'epoch': 3.51}

 35%|███▌      | 2140/6100 [3:13:53<5:57:22,  5.41s/it]
 35%|███▌      | 2141/6100 [3:13:59<5:57:25,  5.42s/it]
 35%|███▌      | 2142/6100 [3:14:04<5:57:08,  5.41s/it]
 35%|███▌      | 2143/6100 [3:14:09<5:56:53,  5.41s/it]
 35%|███▌      | 2144/6100 [3:14:15<5:56:51,  5.41s/it]
 35%|███▌      | 2145/6100 [3:14:20<5:57:00,  5.42s/it]
                                                       
{'loss': 0.0855, 'learning_rate': 0.0001, 'epoch': 3.52}

 35%|███▌      | 2145/6100 [3:14:20<5:57:00,  5.42s/it]
 35%|███▌      | 2146/6100 [3:14:26<5:56:54,  5.42s/it]
 35%|███▌      | 2147/6100 [3:14:31<5:56:28,  5.41s/it]
 35%|███▌      | 2148/6100 [3:14:36<5:56:33,  5.41s/it]
 35%|███▌      | 2149/6100 [3:14:42<5:56:07,  5.41s/it]
 35%|███▌      | 2150/6100 [3:14:47<5:55:45,  5.40s/it]
                                                       
{'loss': 0.0913, 'learning_rate': 0.0001, 'epoch': 3.52}

 35%|███▌      | 2150/6100 [3:14:47<5:55:45,  5.40s/it]
 35%|███▌      | 2151/6100 [3:14:53<5:56:00,  5.41s/it]
 35%|███▌      | 2152/6100 [3:14:58<5:55:58,  5.41s/it]
 35%|███▌      | 2153/6100 [3:15:04<5:55:52,  5.41s/it]
 35%|███▌      | 2154/6100 [3:15:09<5:55:46,  5.41s/it]
 35%|███▌      | 2155/6100 [3:15:14<5:55:50,  5.41s/it]
                                                       
{'loss': 0.0885, 'learning_rate': 0.0001, 'epoch': 3.53}

 35%|███▌      | 2155/6100 [3:15:14<5:55:50,  5.41s/it]
 35%|███▌      | 2156/6100 [3:15:20<5:55:48,  5.41s/it]
 35%|███▌      | 2157/6100 [3:15:25<5:55:41,  5.41s/it]
 35%|███▌      | 2158/6100 [3:15:31<5:55:26,  5.41s/it]
 35%|███▌      | 2159/6100 [3:15:36<5:55:06,  5.41s/it]
 35%|███▌      | 2160/6100 [3:15:41<5:55:09,  5.41s/it]
                                                       
{'loss': 0.0886, 'learning_rate': 0.0001, 'epoch': 3.54}

 35%|███▌      | 2160/6100 [3:15:41<5:55:09,  5.41s/it]
 35%|███▌      | 2161/6100 [3:15:47<5:55:04,  5.41s/it]
 35%|███▌      | 2162/6100 [3:15:52<5:54:59,  5.41s/it]
 35%|███▌      | 2163/6100 [3:15:58<5:55:01,  5.41s/it]
 35%|███▌      | 2164/6100 [3:16:03<5:54:57,  5.41s/it]
 35%|███▌      | 2165/6100 [3:16:08<5:55:06,  5.41s/it]
                                                       
{'loss': 0.0941, 'learning_rate': 0.0001, 'epoch': 3.55}

 35%|███▌      | 2165/6100 [3:16:08<5:55:06,  5.41s/it]
 36%|███▌      | 2166/6100 [3:16:14<5:55:09,  5.42s/it]
 36%|███▌      | 2167/6100 [3:16:19<5:55:07,  5.42s/it]
 36%|███▌      | 2168/6100 [3:16:25<5:54:56,  5.42s/it]
 36%|███▌      | 2169/6100 [3:16:30<5:54:50,  5.42s/it]
 36%|███▌      | 2170/6100 [3:16:36<5:54:38,  5.41s/it]
                                                       
{'loss': 0.0932, 'learning_rate': 0.0001, 'epoch': 3.56}

 36%|███▌      | 2170/6100 [3:16:36<5:54:38,  5.41s/it]
 36%|███▌      | 2171/6100 [3:16:41<5:54:40,  5.42s/it]
 36%|███▌      | 2172/6100 [3:16:46<5:54:24,  5.41s/it]
 36%|███▌      | 2173/6100 [3:16:52<5:54:27,  5.42s/it]
 36%|███▌      | 2174/6100 [3:16:57<5:54:05,  5.41s/it]
 36%|███▌      | 2175/6100 [3:17:03<5:54:20,  5.42s/it]
                                                       
{'loss': 0.0917, 'learning_rate': 0.0001, 'epoch': 3.57}

 36%|███▌      | 2175/6100 [3:17:03<5:54:20,  5.42s/it]
 36%|███▌      | 2176/6100 [3:17:08<5:54:01,  5.41s/it]
 36%|███▌      | 2177/6100 [3:17:13<5:53:40,  5.41s/it]
 36%|███▌      | 2178/6100 [3:17:19<5:53:43,  5.41s/it]
 36%|███▌      | 2179/6100 [3:17:24<5:53:31,  5.41s/it]
 36%|███▌      | 2180/6100 [3:17:30<5:53:38,  5.41s/it]
                                                       
{'loss': 0.0935, 'learning_rate': 0.0001, 'epoch': 3.57}

 36%|███▌      | 2180/6100 [3:17:30<5:53:38,  5.41s/it]
 36%|███▌      | 2181/6100 [3:17:35<5:53:45,  5.42s/it]
 36%|███▌      | 2182/6100 [3:17:41<5:53:36,  5.42s/it]
 36%|███▌      | 2183/6100 [3:17:46<5:53:26,  5.41s/it]
 36%|███▌      | 2184/6100 [3:17:51<5:53:35,  5.42s/it]
 36%|███▌      | 2185/6100 [3:17:57<5:53:44,  5.42s/it]
                                                       
{'loss': 0.0927, 'learning_rate': 0.0001, 'epoch': 3.58}

 36%|███▌      | 2185/6100 [3:17:57<5:53:44,  5.42s/it]
 36%|███▌      | 2186/6100 [3:18:02<5:53:34,  5.42s/it]
 36%|███▌      | 2187/6100 [3:18:08<5:53:22,  5.42s/it]
 36%|███▌      | 2188/6100 [3:18:13<5:52:55,  5.41s/it]
 36%|███▌      | 2189/6100 [3:18:18<5:52:46,  5.41s/it]
 36%|███▌      | 2190/6100 [3:18:24<5:52:34,  5.41s/it]
                                                       
{'loss': 0.0878, 'learning_rate': 0.0001, 'epoch': 3.59}

 36%|███▌      | 2190/6100 [3:18:24<5:52:34,  5.41s/it]
 36%|███▌      | 2191/6100 [3:18:29<5:52:37,  5.41s/it]
 36%|███▌      | 2192/6100 [3:18:35<5:52:37,  5.41s/it]
 36%|███▌      | 2193/6100 [3:18:40<5:52:34,  5.41s/it]
 36%|███▌      | 2194/6100 [3:18:45<5:52:20,  5.41s/it]
 36%|███▌      | 2195/6100 [3:18:51<5:52:10,  5.41s/it]
                                                       
{'loss': 0.0934, 'learning_rate': 0.0001, 'epoch': 3.6}

 36%|███▌      | 2195/6100 [3:18:51<5:52:10,  5.41s/it]
 36%|███▌      | 2196/6100 [3:18:56<5:51:57,  5.41s/it]
 36%|███▌      | 2197/6100 [3:19:02<5:51:43,  5.41s/it]
 36%|███▌      | 2198/6100 [3:19:07<5:51:27,  5.40s/it]
 36%|███▌      | 2199/6100 [3:19:13<5:51:29,  5.41s/it]
 36%|███▌      | 2200/6100 [3:19:18<5:51:32,  5.41s/it]
                                                       
{'loss': 0.0886, 'learning_rate': 0.0001, 'epoch': 3.61}

 36%|███▌      | 2200/6100 [3:19:18<5:51:32,  5.41s/it]
 36%|███▌      | 2201/6100 [3:19:23<5:51:53,  5.42s/it]
 36%|███▌      | 2202/6100 [3:19:29<5:51:47,  5.41s/it]
 36%|███▌      | 2203/6100 [3:19:34<5:51:44,  5.42s/it]
 36%|███▌      | 2204/6100 [3:19:40<5:51:26,  5.41s/it]
 36%|███▌      | 2205/6100 [3:19:45<5:51:35,  5.42s/it]
                                                       
{'loss': 0.0895, 'learning_rate': 0.0001, 'epoch': 3.61}

 36%|███▌      | 2205/6100 [3:19:45<5:51:35,  5.42s/it]
 36%|███▌      | 2206/6100 [3:19:50<5:51:28,  5.42s/it]
 36%|███▌      | 2207/6100 [3:19:56<5:51:12,  5.41s/it]
 36%|███▌      | 2208/6100 [3:20:01<5:51:01,  5.41s/it]
 36%|███▌      | 2209/6100 [3:20:07<5:51:06,  5.41s/it]
 36%|███▌      | 2210/6100 [3:20:12<5:51:05,  5.42s/it]
                                                       
{'loss': 0.0864, 'learning_rate': 0.0001, 'epoch': 3.62}

 36%|███▌      | 2210/6100 [3:20:12<5:51:05,  5.42s/it]
 36%|███▌      | 2211/6100 [3:20:17<5:50:55,  5.41s/it]
 36%|███▋      | 2212/6100 [3:20:23<5:50:48,  5.41s/it]
 36%|███▋      | 2213/6100 [3:20:28<5:50:37,  5.41s/it]
 36%|███▋      | 2214/6100 [3:20:34<5:50:40,  5.41s/it]
 36%|███▋      | 2215/6100 [3:20:39<5:50:21,  5.41s/it]
                                                       
{'loss': 0.0884, 'learning_rate': 0.0001, 'epoch': 3.63}

 36%|███▋      | 2215/6100 [3:20:39<5:50:21,  5.41s/it]
 36%|███▋      | 2216/6100 [3:20:45<5:50:35,  5.42s/it]
 36%|███▋      | 2217/6100 [3:20:50<5:50:15,  5.41s/it]
 36%|███▋      | 2218/6100 [3:20:56<6:02:08,  5.60s/it]
 36%|███▋      | 2219/6100 [3:21:01<5:58:29,  5.54s/it]
 36%|███▋      | 2220/6100 [3:21:07<5:55:52,  5.50s/it]
                                                       
{'loss': 0.092, 'learning_rate': 0.0001, 'epoch': 3.64}

 36%|███▋      | 2220/6100 [3:21:07<5:55:52,  5.50s/it]
 36%|███▋      | 2221/6100 [3:21:12<5:54:01,  5.48s/it]
 36%|███▋      | 2222/6100 [3:21:18<5:52:20,  5.45s/it]
 36%|███▋      | 2223/6100 [3:21:23<5:51:15,  5.44s/it]
 36%|███▋      | 2224/6100 [3:21:28<5:50:48,  5.43s/it]
 36%|███▋      | 2225/6100 [3:21:34<5:50:26,  5.43s/it]
                                                       
{'loss': 0.0932, 'learning_rate': 0.0001, 'epoch': 3.65}

 36%|███▋      | 2225/6100 [3:21:34<5:50:26,  5.43s/it]
 36%|███▋      | 2226/6100 [3:21:39<5:50:19,  5.43s/it]
 37%|███▋      | 2227/6100 [3:21:45<5:50:17,  5.43s/it]
 37%|███▋      | 2228/6100 [3:21:50<5:49:58,  5.42s/it]
 37%|███▋      | 2229/6100 [3:21:56<5:49:56,  5.42s/it]
 37%|███▋      | 2230/6100 [3:22:01<5:50:06,  5.43s/it]
                                                       
{'loss': 0.0895, 'learning_rate': 0.0001, 'epoch': 3.66}

 37%|███▋      | 2230/6100 [3:22:01<5:50:06,  5.43s/it]
 37%|███▋      | 2231/6100 [3:22:06<5:49:58,  5.43s/it]
 37%|███▋      | 2232/6100 [3:22:12<5:49:44,  5.43s/it]
 37%|███▋      | 2233/6100 [3:22:17<5:49:27,  5.42s/it]
 37%|███▋      | 2234/6100 [3:22:23<5:49:42,  5.43s/it]
 37%|███▋      | 2235/6100 [3:22:28<5:49:12,  5.42s/it]
                                                       
{'loss': 0.0887, 'learning_rate': 0.0001, 'epoch': 3.66}

 37%|███▋      | 2235/6100 [3:22:28<5:49:12,  5.42s/it]
 37%|███▋      | 2236/6100 [3:22:34<5:48:58,  5.42s/it]
 37%|███▋      | 2237/6100 [3:22:39<5:48:37,  5.41s/it]
 37%|███▋      | 2238/6100 [3:22:44<5:48:19,  5.41s/it]
 37%|███▋      | 2239/6100 [3:22:50<5:48:25,  5.41s/it]
 37%|███▋      | 2240/6100 [3:22:55<5:48:40,  5.42s/it]
                                                       
{'loss': 0.0883, 'learning_rate': 0.0001, 'epoch': 3.67}

 37%|███▋      | 2240/6100 [3:22:55<5:48:40,  5.42s/it]
 37%|███▋      | 2241/6100 [3:23:01<5:48:33,  5.42s/it]
 37%|███▋      | 2242/6100 [3:23:06<5:48:16,  5.42s/it]
 37%|███▋      | 2243/6100 [3:23:11<5:48:09,  5.42s/it]
 37%|███▋      | 2244/6100 [3:23:17<5:47:56,  5.41s/it]
 37%|███▋      | 2245/6100 [3:23:22<5:47:40,  5.41s/it]
                                                       
{'loss': 0.0887, 'learning_rate': 0.0001, 'epoch': 3.68}

 37%|███▋      | 2245/6100 [3:23:22<5:47:40,  5.41s/it]
 37%|███▋      | 2246/6100 [3:23:28<5:47:30,  5.41s/it]
 37%|███▋      | 2247/6100 [3:23:33<5:47:18,  5.41s/it]
 37%|███▋      | 2248/6100 [3:23:38<5:47:27,  5.41s/it]
 37%|███▋      | 2249/6100 [3:23:44<5:47:21,  5.41s/it]
 37%|███▋      | 2250/6100 [3:23:49<5:47:19,  5.41s/it]
                                                       
{'loss': 0.0875, 'learning_rate': 0.0001, 'epoch': 3.69}

 37%|███▋      | 2250/6100 [3:23:49<5:47:19,  5.41s/it]Saving model checkpoint to ./results/checkpoint-2250
Configuration saved in ./results/checkpoint-2250/config.json
Model weights saved in ./results/checkpoint-2250/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-2250/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-2250/special_tokens_map.json
[2023-02-21 23:16:05,466] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step2251 is begin to save!
[2023-02-21 23:16:05,469] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-2250/global_step2251/mp_rank_00_model_states.pt
[2023-02-21 23:16:05,469] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-2250/global_step2251/mp_rank_00_model_states.pt...
[2023-02-21 23:16:06,289] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-2250/global_step2251/mp_rank_00_model_states.pt.
[2023-02-21 23:16:06,291] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-2250/global_step2251/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-21 23:16:06,467] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-2250/global_step2251/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-21 23:16:06,467] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-2250/global_step2251/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-21 23:16:06,467] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2251 is ready now!
Deleting older checkpoint [results/checkpoint-2100] due to args.save_total_limit

 37%|███▋      | 2251/6100 [3:23:57<6:28:37,  6.06s/it]
 37%|███▋      | 2252/6100 [3:24:02<6:16:01,  5.86s/it]
 37%|███▋      | 2253/6100 [3:24:08<6:07:15,  5.73s/it]
 37%|███▋      | 2254/6100 [3:24:13<6:01:14,  5.64s/it]
 37%|███▋      | 2255/6100 [3:24:18<5:56:35,  5.56s/it]
                                                       
{'loss': 0.093, 'learning_rate': 0.0001, 'epoch': 3.7}

 37%|███▋      | 2255/6100 [3:24:18<5:56:35,  5.56s/it]
 37%|███▋      | 2256/6100 [3:24:24<5:53:35,  5.52s/it]
 37%|███▋      | 2257/6100 [3:24:29<5:51:08,  5.48s/it]
 37%|███▋      | 2258/6100 [3:24:35<5:49:19,  5.46s/it]
 37%|███▋      | 2259/6100 [3:24:40<5:48:04,  5.44s/it]
 37%|███▋      | 2260/6100 [3:24:46<5:47:34,  5.43s/it]
                                                       
{'loss': 0.089, 'learning_rate': 0.0001, 'epoch': 3.7}

 37%|███▋      | 2260/6100 [3:24:46<5:47:34,  5.43s/it]
 37%|███▋      | 2261/6100 [3:24:51<5:47:03,  5.42s/it]
 37%|███▋      | 2262/6100 [3:24:56<5:46:26,  5.42s/it]
 37%|███▋      | 2263/6100 [3:25:02<5:46:25,  5.42s/it]
 37%|███▋      | 2264/6100 [3:25:07<5:46:09,  5.41s/it]
 37%|███▋      | 2265/6100 [3:25:13<5:46:05,  5.41s/it]
                                                       
{'loss': 0.0896, 'learning_rate': 0.0001, 'epoch': 3.71}

 37%|███▋      | 2265/6100 [3:25:13<5:46:05,  5.41s/it]
 37%|███▋      | 2266/6100 [3:25:18<5:46:21,  5.42s/it]
 37%|███▋      | 2267/6100 [3:25:23<5:45:46,  5.41s/it]
 37%|███▋      | 2268/6100 [3:25:29<5:45:27,  5.41s/it]
 37%|███▋      | 2269/6100 [3:25:34<5:45:21,  5.41s/it]
 37%|███▋      | 2270/6100 [3:25:40<5:45:14,  5.41s/it]
                                                       
{'loss': 0.0882, 'learning_rate': 0.0001, 'epoch': 3.72}

 37%|███▋      | 2270/6100 [3:25:40<5:45:14,  5.41s/it]
 37%|███▋      | 2271/6100 [3:25:45<5:44:56,  5.41s/it]
 37%|███▋      | 2272/6100 [3:25:50<5:44:48,  5.40s/it]
 37%|███▋      | 2273/6100 [3:25:56<5:44:54,  5.41s/it]
 37%|███▋      | 2274/6100 [3:26:01<5:44:43,  5.41s/it]
 37%|███▋      | 2275/6100 [3:26:07<5:44:26,  5.40s/it]
                                                       
{'loss': 0.0901, 'learning_rate': 0.0001, 'epoch': 3.73}

 37%|███▋      | 2275/6100 [3:26:07<5:44:26,  5.40s/it]
 37%|███▋      | 2276/6100 [3:26:12<5:44:26,  5.40s/it]
 37%|███▋      | 2277/6100 [3:26:17<5:44:35,  5.41s/it]
 37%|███▋      | 2278/6100 [3:26:23<5:44:33,  5.41s/it]
 37%|███▋      | 2279/6100 [3:26:28<5:46:28,  5.44s/it]
 37%|███▋      | 2280/6100 [3:26:34<5:45:47,  5.43s/it]
                                                       
{'loss': 0.0883, 'learning_rate': 0.0001, 'epoch': 3.74}

 37%|███▋      | 2280/6100 [3:26:34<5:45:47,  5.43s/it]
 37%|███▋      | 2281/6100 [3:26:39<5:45:20,  5.43s/it]
 37%|███▋      | 2282/6100 [3:26:45<5:44:47,  5.42s/it]
 37%|███▋      | 2283/6100 [3:26:50<5:44:31,  5.42s/it]
 37%|███▋      | 2284/6100 [3:26:55<5:44:38,  5.42s/it]
 37%|███▋      | 2285/6100 [3:27:01<5:44:26,  5.42s/it]
                                                       
{'loss': 0.0888, 'learning_rate': 0.0001, 'epoch': 3.75}

 37%|███▋      | 2285/6100 [3:27:01<5:44:26,  5.42s/it]
 37%|███▋      | 2286/6100 [3:27:06<5:44:24,  5.42s/it]
 37%|███▋      | 2287/6100 [3:27:12<5:44:07,  5.42s/it]
 38%|███▊      | 2288/6100 [3:27:17<5:44:08,  5.42s/it]
 38%|███▊      | 2289/6100 [3:27:22<5:44:01,  5.42s/it]
 38%|███▊      | 2290/6100 [3:27:28<5:44:16,  5.42s/it]
                                                       
{'loss': 0.0911, 'learning_rate': 0.0001, 'epoch': 3.75}

 38%|███▊      | 2290/6100 [3:27:28<5:44:16,  5.42s/it]
 38%|███▊      | 2291/6100 [3:27:33<5:45:54,  5.45s/it]
 38%|███▊      | 2292/6100 [3:27:39<5:45:08,  5.44s/it]
 38%|███▊      | 2293/6100 [3:27:44<5:44:44,  5.43s/it]
 38%|███▊      | 2294/6100 [3:27:50<5:44:21,  5.43s/it]
 38%|███▊      | 2295/6100 [3:27:55<5:44:08,  5.43s/it]
                                                       
{'loss': 0.085, 'learning_rate': 0.0001, 'epoch': 3.76}

 38%|███▊      | 2295/6100 [3:27:55<5:44:08,  5.43s/it]
 38%|███▊      | 2296/6100 [3:28:01<5:43:46,  5.42s/it]
 38%|███▊      | 2297/6100 [3:28:06<5:43:26,  5.42s/it]
 38%|███▊      | 2298/6100 [3:28:11<5:43:13,  5.42s/it]
 38%|███▊      | 2299/6100 [3:28:17<5:56:52,  5.63s/it]
 38%|███▊      | 2300/6100 [3:28:23<5:52:30,  5.57s/it]
                                                       
{'loss': 0.0905, 'learning_rate': 0.0001, 'epoch': 3.77}

 38%|███▊      | 2300/6100 [3:28:23<5:52:30,  5.57s/it]
 38%|███▊      | 2301/6100 [3:28:28<5:49:31,  5.52s/it]
 38%|███▊      | 2302/6100 [3:28:34<5:47:24,  5.49s/it]
 38%|███▊      | 2303/6100 [3:28:39<5:45:48,  5.46s/it]
 38%|███▊      | 2304/6100 [3:28:45<5:44:43,  5.45s/it]
 38%|███▊      | 2305/6100 [3:28:50<5:43:38,  5.43s/it]
                                                       
{'loss': 0.086, 'learning_rate': 0.0001, 'epoch': 3.78}

 38%|███▊      | 2305/6100 [3:28:50<5:43:38,  5.43s/it]
 38%|███▊      | 2306/6100 [3:28:55<5:43:19,  5.43s/it]
 38%|███▊      | 2307/6100 [3:29:01<5:42:47,  5.42s/it]
 38%|███▊      | 2308/6100 [3:29:06<5:42:40,  5.42s/it]
 38%|███▊      | 2309/6100 [3:29:12<5:42:42,  5.42s/it]
 38%|███▊      | 2310/6100 [3:29:17<5:42:17,  5.42s/it]
                                                       
{'loss': 0.0841, 'learning_rate': 0.0001, 'epoch': 3.79}

 38%|███▊      | 2310/6100 [3:29:17<5:42:17,  5.42s/it]
 38%|███▊      | 2311/6100 [3:29:22<5:41:59,  5.42s/it]
 38%|███▊      | 2312/6100 [3:29:28<5:41:48,  5.41s/it]
 38%|███▊      | 2313/6100 [3:29:33<5:41:36,  5.41s/it]
 38%|███▊      | 2314/6100 [3:29:39<5:41:50,  5.42s/it]
 38%|███▊      | 2315/6100 [3:29:45<5:51:06,  5.57s/it]
                                                       
{'loss': 0.0865, 'learning_rate': 0.0001, 'epoch': 3.79}

 38%|███▊      | 2315/6100 [3:29:45<5:51:06,  5.57s/it]
 38%|███▊      | 2316/6100 [3:29:50<5:48:06,  5.52s/it]
 38%|███▊      | 2317/6100 [3:29:55<5:45:49,  5.48s/it]
 38%|███▊      | 2318/6100 [3:30:01<5:44:12,  5.46s/it]
 38%|███▊      | 2319/6100 [3:30:06<5:42:56,  5.44s/it]
 38%|███▊      | 2320/6100 [3:30:12<5:42:33,  5.44s/it]
                                                       
{'loss': 0.0874, 'learning_rate': 0.0001, 'epoch': 3.8}

 38%|███▊      | 2320/6100 [3:30:12<5:42:33,  5.44s/it]
 38%|███▊      | 2321/6100 [3:30:17<5:42:05,  5.43s/it]
 38%|███▊      | 2322/6100 [3:30:22<5:41:34,  5.42s/it]
 38%|███▊      | 2323/6100 [3:30:28<5:41:11,  5.42s/it]
 38%|███▊      | 2324/6100 [3:30:33<5:40:52,  5.42s/it]
 38%|███▊      | 2325/6100 [3:30:39<5:40:34,  5.41s/it]
                                                       
{'loss': 0.0904, 'learning_rate': 0.0001, 'epoch': 3.81}

 38%|███▊      | 2325/6100 [3:30:39<5:40:34,  5.41s/it]
 38%|███▊      | 2326/6100 [3:30:44<5:40:21,  5.41s/it]
 38%|███▊      | 2327/6100 [3:30:50<5:40:15,  5.41s/it]
 38%|███▊      | 2328/6100 [3:30:55<5:40:07,  5.41s/it]
 38%|███▊      | 2329/6100 [3:31:00<5:40:16,  5.41s/it]
 38%|███▊      | 2330/6100 [3:31:06<5:40:04,  5.41s/it]
                                                       
{'loss': 0.0877, 'learning_rate': 0.0001, 'epoch': 3.82}

 38%|███▊      | 2330/6100 [3:31:06<5:40:04,  5.41s/it]
 38%|███▊      | 2331/6100 [3:31:11<5:39:53,  5.41s/it]
 38%|███▊      | 2332/6100 [3:31:17<5:39:43,  5.41s/it]
 38%|███▊      | 2333/6100 [3:31:22<5:39:33,  5.41s/it]
 38%|███▊      | 2334/6100 [3:31:27<5:39:28,  5.41s/it]
 38%|███▊      | 2335/6100 [3:31:33<5:39:31,  5.41s/it]
                                                       
{'loss': 0.0811, 'learning_rate': 0.0001, 'epoch': 3.83}

 38%|███▊      | 2335/6100 [3:31:33<5:39:31,  5.41s/it]
 38%|███▊      | 2336/6100 [3:31:38<5:39:34,  5.41s/it]
 38%|███▊      | 2337/6100 [3:31:44<5:39:16,  5.41s/it]
 38%|███▊      | 2338/6100 [3:31:49<5:39:09,  5.41s/it]
 38%|███▊      | 2339/6100 [3:31:54<5:40:04,  5.43s/it]
 38%|███▊      | 2340/6100 [3:32:00<5:39:33,  5.42s/it]
                                                       
{'loss': 0.0895, 'learning_rate': 0.0001, 'epoch': 3.84}

 38%|███▊      | 2340/6100 [3:32:00<5:39:33,  5.42s/it]
 38%|███▊      | 2341/6100 [3:32:05<5:39:11,  5.41s/it]
 38%|███▊      | 2342/6100 [3:32:11<5:39:38,  5.42s/it]
 38%|███▊      | 2343/6100 [3:32:16<5:39:20,  5.42s/it]
 38%|███▊      | 2344/6100 [3:32:22<5:39:30,  5.42s/it]
 38%|███▊      | 2345/6100 [3:32:27<5:39:01,  5.42s/it]
                                                       
{'loss': 0.0913, 'learning_rate': 0.0001, 'epoch': 3.84}

 38%|███▊      | 2345/6100 [3:32:27<5:39:01,  5.42s/it]
 38%|███▊      | 2346/6100 [3:32:32<5:38:45,  5.41s/it]
 38%|███▊      | 2347/6100 [3:32:38<5:38:45,  5.42s/it]
 38%|███▊      | 2348/6100 [3:32:43<5:38:40,  5.42s/it]
 39%|███▊      | 2349/6100 [3:32:49<5:38:28,  5.41s/it]
 39%|███▊      | 2350/6100 [3:32:54<5:38:22,  5.41s/it]
                                                       
{'loss': 0.091, 'learning_rate': 0.0001, 'epoch': 3.85}

 39%|███▊      | 2350/6100 [3:32:54<5:38:22,  5.41s/it]
 39%|███▊      | 2351/6100 [3:32:59<5:38:13,  5.41s/it]
 39%|███▊      | 2352/6100 [3:33:05<5:37:58,  5.41s/it]
 39%|███▊      | 2353/6100 [3:33:10<5:37:48,  5.41s/it]
 39%|███▊      | 2354/6100 [3:33:16<5:37:50,  5.41s/it]
 39%|███▊      | 2355/6100 [3:33:21<5:37:50,  5.41s/it]
                                                       
{'loss': 0.0843, 'learning_rate': 0.0001, 'epoch': 3.86}

 39%|███▊      | 2355/6100 [3:33:21<5:37:50,  5.41s/it]
 39%|███▊      | 2356/6100 [3:33:27<5:38:03,  5.42s/it]
 39%|███▊      | 2357/6100 [3:33:32<5:38:08,  5.42s/it]
 39%|███▊      | 2358/6100 [3:33:37<5:37:54,  5.42s/it]
 39%|███▊      | 2359/6100 [3:33:43<5:37:45,  5.42s/it]
 39%|███▊      | 2360/6100 [3:33:48<5:37:19,  5.41s/it]
                                                       
{'loss': 0.0917, 'learning_rate': 0.0001, 'epoch': 3.87}

 39%|███▊      | 2360/6100 [3:33:48<5:37:19,  5.41s/it]
 39%|███▊      | 2361/6100 [3:33:54<5:37:01,  5.41s/it]
 39%|███▊      | 2362/6100 [3:33:59<5:36:56,  5.41s/it]
 39%|███▊      | 2363/6100 [3:34:04<5:36:50,  5.41s/it]
 39%|███▉      | 2364/6100 [3:34:10<5:37:05,  5.41s/it]
 39%|███▉      | 2365/6100 [3:34:15<5:37:13,  5.42s/it]
                                                       
{'loss': 0.0907, 'learning_rate': 0.0001, 'epoch': 3.88}

 39%|███▉      | 2365/6100 [3:34:15<5:37:13,  5.42s/it]
 39%|███▉      | 2366/6100 [3:34:21<5:36:57,  5.41s/it]
 39%|███▉      | 2367/6100 [3:34:26<5:37:02,  5.42s/it]
 39%|███▉      | 2368/6100 [3:34:31<5:36:43,  5.41s/it]
 39%|███▉      | 2369/6100 [3:34:37<5:36:41,  5.41s/it]
 39%|███▉      | 2370/6100 [3:34:42<5:36:18,  5.41s/it]
                                                       
{'loss': 0.0863, 'learning_rate': 0.0001, 'epoch': 3.88}

 39%|███▉      | 2370/6100 [3:34:42<5:36:18,  5.41s/it]
 39%|███▉      | 2371/6100 [3:34:48<5:36:25,  5.41s/it]
 39%|███▉      | 2372/6100 [3:34:53<5:36:02,  5.41s/it]
 39%|███▉      | 2373/6100 [3:34:59<5:36:07,  5.41s/it]
 39%|███▉      | 2374/6100 [3:35:04<5:35:51,  5.41s/it]
 39%|███▉      | 2375/6100 [3:35:09<5:35:57,  5.41s/it]
                                                       
{'loss': 0.0832, 'learning_rate': 0.0001, 'epoch': 3.89}

 39%|███▉      | 2375/6100 [3:35:09<5:35:57,  5.41s/it]
 39%|███▉      | 2376/6100 [3:35:15<5:35:52,  5.41s/it]
 39%|███▉      | 2377/6100 [3:35:20<5:35:44,  5.41s/it]
 39%|███▉      | 2378/6100 [3:35:26<5:35:42,  5.41s/it]
 39%|███▉      | 2379/6100 [3:35:31<5:40:40,  5.49s/it]
 39%|███▉      | 2380/6100 [3:35:37<5:39:14,  5.47s/it]
                                                       
{'loss': 0.09, 'learning_rate': 0.0001, 'epoch': 3.9}

 39%|███▉      | 2380/6100 [3:35:37<5:39:14,  5.47s/it]
 39%|███▉      | 2381/6100 [3:35:42<5:37:54,  5.45s/it]
 39%|███▉      | 2382/6100 [3:35:48<5:37:13,  5.44s/it]
 39%|███▉      | 2383/6100 [3:35:53<5:36:19,  5.43s/it]
 39%|███▉      | 2384/6100 [3:35:58<5:36:10,  5.43s/it]
 39%|███▉      | 2385/6100 [3:36:04<5:35:50,  5.42s/it]
                                                       
{'loss': 0.0861, 'learning_rate': 0.0001, 'epoch': 3.91}

 39%|███▉      | 2385/6100 [3:36:04<5:35:50,  5.42s/it]
 39%|███▉      | 2386/6100 [3:36:09<5:35:36,  5.42s/it]
 39%|███▉      | 2387/6100 [3:36:15<5:35:24,  5.42s/it]
 39%|███▉      | 2388/6100 [3:36:20<5:35:50,  5.43s/it]
 39%|███▉      | 2389/6100 [3:36:25<5:35:31,  5.42s/it]
 39%|███▉      | 2390/6100 [3:36:31<5:35:08,  5.42s/it]
                                                       
{'loss': 0.0859, 'learning_rate': 0.0001, 'epoch': 3.92}

 39%|███▉      | 2390/6100 [3:36:31<5:35:08,  5.42s/it]
 39%|███▉      | 2391/6100 [3:36:36<5:34:57,  5.42s/it]
 39%|███▉      | 2392/6100 [3:36:42<5:34:46,  5.42s/it]
 39%|███▉      | 2393/6100 [3:36:47<5:34:25,  5.41s/it]
 39%|███▉      | 2394/6100 [3:36:53<5:34:15,  5.41s/it]
 39%|███▉      | 2395/6100 [3:36:58<5:34:35,  5.42s/it]
                                                       
{'loss': 0.0907, 'learning_rate': 0.0001, 'epoch': 3.93}

 39%|███▉      | 2395/6100 [3:36:58<5:34:35,  5.42s/it]
 39%|███▉      | 2396/6100 [3:37:03<5:36:07,  5.44s/it]
 39%|███▉      | 2397/6100 [3:37:09<5:35:21,  5.43s/it]
 39%|███▉      | 2398/6100 [3:37:14<5:35:03,  5.43s/it]
 39%|███▉      | 2399/6100 [3:37:20<5:34:28,  5.42s/it]
 39%|███▉      | 2400/6100 [3:37:25<5:34:16,  5.42s/it]
                                                       
{'loss': 0.0858, 'learning_rate': 0.0001, 'epoch': 3.93}

 39%|███▉      | 2400/6100 [3:37:25<5:34:16,  5.42s/it]Saving model checkpoint to ./results/checkpoint-2400
Configuration saved in ./results/checkpoint-2400/config.json
Model weights saved in ./results/checkpoint-2400/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-2400/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-2400/special_tokens_map.json
[2023-02-21 23:29:41,276] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step2401 is begin to save!
[2023-02-21 23:29:41,279] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-2400/global_step2401/mp_rank_00_model_states.pt
[2023-02-21 23:29:41,279] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-2400/global_step2401/mp_rank_00_model_states.pt...
[2023-02-21 23:29:42,098] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-2400/global_step2401/mp_rank_00_model_states.pt.
[2023-02-21 23:29:42,100] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-2400/global_step2401/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-21 23:29:42,274] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-2400/global_step2401/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-21 23:29:42,274] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-2400/global_step2401/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-21 23:29:42,274] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2401 is ready now!
Deleting older checkpoint [results/checkpoint-2250] due to args.save_total_limit

 39%|███▉      | 2401/6100 [3:37:33<6:13:38,  6.06s/it]
 39%|███▉      | 2402/6100 [3:37:38<6:01:24,  5.86s/it]
 39%|███▉      | 2403/6100 [3:37:43<5:53:02,  5.73s/it]
 39%|███▉      | 2404/6100 [3:37:49<5:46:56,  5.63s/it]
 39%|███▉      | 2405/6100 [3:37:54<5:42:33,  5.56s/it]
                                                       
{'loss': 0.0834, 'learning_rate': 0.0001, 'epoch': 3.94}

 39%|███▉      | 2405/6100 [3:37:54<5:42:33,  5.56s/it]
 39%|███▉      | 2406/6100 [3:38:00<5:39:24,  5.51s/it]
 39%|███▉      | 2407/6100 [3:38:05<5:37:13,  5.48s/it]
 39%|███▉      | 2408/6100 [3:38:11<5:36:15,  5.46s/it]
 39%|███▉      | 2409/6100 [3:38:16<5:35:01,  5.45s/it]
 40%|███▉      | 2410/6100 [3:38:21<5:34:08,  5.43s/it]
                                                       
{'loss': 0.0874, 'learning_rate': 0.0001, 'epoch': 3.95}

 40%|███▉      | 2410/6100 [3:38:21<5:34:08,  5.43s/it]
 40%|███▉      | 2411/6100 [3:38:27<5:33:37,  5.43s/it]
 40%|███▉      | 2412/6100 [3:38:32<5:33:06,  5.42s/it]
 40%|███▉      | 2413/6100 [3:38:38<5:33:02,  5.42s/it]
 40%|███▉      | 2414/6100 [3:38:43<5:32:46,  5.42s/it]
 40%|███▉      | 2415/6100 [3:38:48<5:33:51,  5.44s/it]
                                                       
{'loss': 0.0902, 'learning_rate': 0.0001, 'epoch': 3.96}

 40%|███▉      | 2415/6100 [3:38:48<5:33:51,  5.44s/it]
 40%|███▉      | 2416/6100 [3:38:54<5:33:21,  5.43s/it]
 40%|███▉      | 2417/6100 [3:38:59<5:33:07,  5.43s/it]
 40%|███▉      | 2418/6100 [3:39:05<5:32:49,  5.42s/it]
 40%|███▉      | 2419/6100 [3:39:10<5:32:20,  5.42s/it]
 40%|███▉      | 2420/6100 [3:39:15<5:31:52,  5.41s/it]
                                                       
{'loss': 0.0926, 'learning_rate': 0.0001, 'epoch': 3.97}

 40%|███▉      | 2420/6100 [3:39:15<5:31:52,  5.41s/it]
 40%|███▉      | 2421/6100 [3:39:21<5:31:54,  5.41s/it]
 40%|███▉      | 2422/6100 [3:39:26<5:31:37,  5.41s/it]
 40%|███▉      | 2423/6100 [3:39:32<5:31:35,  5.41s/it]
 40%|███▉      | 2424/6100 [3:39:37<5:31:21,  5.41s/it]
 40%|███▉      | 2425/6100 [3:39:43<5:31:21,  5.41s/it]
                                                       
{'loss': 0.089, 'learning_rate': 0.0001, 'epoch': 3.97}

 40%|███▉      | 2425/6100 [3:39:43<5:31:21,  5.41s/it]
 40%|███▉      | 2426/6100 [3:39:48<5:31:12,  5.41s/it]
 40%|███▉      | 2427/6100 [3:39:53<5:31:07,  5.41s/it]
 40%|███▉      | 2428/6100 [3:39:59<5:31:05,  5.41s/it]
 40%|███▉      | 2429/6100 [3:40:04<5:31:01,  5.41s/it]
 40%|███▉      | 2430/6100 [3:40:10<5:30:51,  5.41s/it]
                                                       
{'loss': 0.0866, 'learning_rate': 0.0001, 'epoch': 3.98}

 40%|███▉      | 2430/6100 [3:40:10<5:30:51,  5.41s/it]
 40%|███▉      | 2431/6100 [3:40:15<5:30:57,  5.41s/it]
 40%|███▉      | 2432/6100 [3:40:20<5:30:41,  5.41s/it]
 40%|███▉      | 2433/6100 [3:40:26<5:30:46,  5.41s/it]
 40%|███▉      | 2434/6100 [3:40:31<5:31:02,  5.42s/it]
 40%|███▉      | 2435/6100 [3:40:37<5:31:14,  5.42s/it]
                                                       
{'loss': 0.0837, 'learning_rate': 0.0001, 'epoch': 3.99}

 40%|███▉      | 2435/6100 [3:40:37<5:31:14,  5.42s/it]
 40%|███▉      | 2436/6100 [3:40:42<5:30:51,  5.42s/it]
 40%|███▉      | 2437/6100 [3:40:48<5:31:10,  5.42s/it]
 40%|███▉      | 2438/6100 [3:40:53<5:30:44,  5.42s/it]
 40%|███▉      | 2439/6100 [3:40:58<5:30:34,  5.42s/it]
 40%|████      | 2440/6100 [3:41:04<5:30:15,  5.41s/it]
                                                       
{'loss': 0.0845, 'learning_rate': 0.0001, 'epoch': 4.0}

 40%|████      | 2440/6100 [3:41:04<5:30:15,  5.41s/it]
 40%|████      | 2441/6100 [3:41:12<6:23:03,  6.28s/it]
 40%|████      | 2442/6100 [3:41:17<6:06:50,  6.02s/it]
 40%|████      | 2443/6100 [3:41:23<5:55:45,  5.84s/it]
 40%|████      | 2444/6100 [3:41:28<5:47:46,  5.71s/it]
 40%|████      | 2445/6100 [3:41:34<5:42:13,  5.62s/it]
                                                       
{'loss': 0.0924, 'learning_rate': 0.0001, 'epoch': 4.01}

 40%|████      | 2445/6100 [3:41:34<5:42:13,  5.62s/it]
 40%|████      | 2446/6100 [3:41:39<5:38:31,  5.56s/it]
 40%|████      | 2447/6100 [3:41:45<5:35:43,  5.51s/it]
 40%|████      | 2448/6100 [3:41:50<5:33:48,  5.48s/it]
 40%|████      | 2449/6100 [3:41:55<5:32:30,  5.46s/it]
 40%|████      | 2450/6100 [3:42:01<5:31:24,  5.45s/it]
                                                       
{'loss': 0.082, 'learning_rate': 0.0001, 'epoch': 4.02}

 40%|████      | 2450/6100 [3:42:01<5:31:24,  5.45s/it]
 40%|████      | 2451/6100 [3:42:06<5:30:41,  5.44s/it]
 40%|████      | 2452/6100 [3:42:12<5:29:58,  5.43s/it]
 40%|████      | 2453/6100 [3:42:17<5:30:03,  5.43s/it]
 40%|████      | 2454/6100 [3:42:22<5:29:28,  5.42s/it]
 40%|████      | 2455/6100 [3:42:28<5:29:16,  5.42s/it]
                                                       
{'loss': 0.0852, 'learning_rate': 0.0001, 'epoch': 4.02}

 40%|████      | 2455/6100 [3:42:28<5:29:16,  5.42s/it]
 40%|████      | 2456/6100 [3:42:33<5:28:52,  5.42s/it]
 40%|████      | 2457/6100 [3:42:39<5:28:36,  5.41s/it]
 40%|████      | 2458/6100 [3:42:44<5:28:36,  5.41s/it]
 40%|████      | 2459/6100 [3:42:49<5:28:36,  5.42s/it]
 40%|████      | 2460/6100 [3:42:55<5:28:31,  5.42s/it]
                                                       
{'loss': 0.0863, 'learning_rate': 0.0001, 'epoch': 4.03}

 40%|████      | 2460/6100 [3:42:55<5:28:31,  5.42s/it]
 40%|████      | 2461/6100 [3:43:00<5:28:24,  5.41s/it]
 40%|████      | 2462/6100 [3:43:06<5:28:21,  5.42s/it]
 40%|████      | 2463/6100 [3:43:11<5:28:01,  5.41s/it]
 40%|████      | 2464/6100 [3:43:17<5:27:55,  5.41s/it]
 40%|████      | 2465/6100 [3:43:22<5:27:46,  5.41s/it]
                                                       
{'loss': 0.077, 'learning_rate': 0.0001, 'epoch': 4.04}

 40%|████      | 2465/6100 [3:43:22<5:27:46,  5.41s/it]
 40%|████      | 2466/6100 [3:43:27<5:27:40,  5.41s/it]
 40%|████      | 2467/6100 [3:43:33<5:27:43,  5.41s/it]
 40%|████      | 2468/6100 [3:43:38<5:27:20,  5.41s/it]
 40%|████      | 2469/6100 [3:43:44<5:27:16,  5.41s/it]
 40%|████      | 2470/6100 [3:43:49<5:27:09,  5.41s/it]
                                                       
{'loss': 0.0801, 'learning_rate': 0.0001, 'epoch': 4.05}

 40%|████      | 2470/6100 [3:43:49<5:27:09,  5.41s/it]
 41%|████      | 2471/6100 [3:43:54<5:27:02,  5.41s/it]
 41%|████      | 2472/6100 [3:44:00<5:26:55,  5.41s/it]
 41%|████      | 2473/6100 [3:44:05<5:26:42,  5.40s/it]
 41%|████      | 2474/6100 [3:44:11<5:26:44,  5.41s/it]
 41%|████      | 2475/6100 [3:44:16<5:28:58,  5.45s/it]
                                                       
{'loss': 0.0892, 'learning_rate': 0.0001, 'epoch': 4.06}

 41%|████      | 2475/6100 [3:44:16<5:28:58,  5.45s/it]
 41%|████      | 2476/6100 [3:44:22<5:28:15,  5.43s/it]
 41%|████      | 2477/6100 [3:44:27<5:27:49,  5.43s/it]
 41%|████      | 2478/6100 [3:44:32<5:27:24,  5.42s/it]
 41%|████      | 2479/6100 [3:44:38<5:27:23,  5.42s/it]
 41%|████      | 2480/6100 [3:44:43<5:26:55,  5.42s/it]
                                                       
{'loss': 0.0775, 'learning_rate': 0.0001, 'epoch': 4.07}

 41%|████      | 2480/6100 [3:44:43<5:26:55,  5.42s/it]
 41%|████      | 2481/6100 [3:44:49<5:26:36,  5.41s/it]
 41%|████      | 2482/6100 [3:44:54<5:26:23,  5.41s/it]
 41%|████      | 2483/6100 [3:44:59<5:26:21,  5.41s/it]
 41%|████      | 2484/6100 [3:45:05<5:26:18,  5.41s/it]
 41%|████      | 2485/6100 [3:45:10<5:26:21,  5.42s/it]
                                                       
{'loss': 0.0812, 'learning_rate': 0.0001, 'epoch': 4.07}

 41%|████      | 2485/6100 [3:45:10<5:26:21,  5.42s/it]
 41%|████      | 2486/6100 [3:45:16<5:26:19,  5.42s/it]
 41%|████      | 2487/6100 [3:45:21<5:26:33,  5.42s/it]
 41%|████      | 2488/6100 [3:45:27<5:26:02,  5.42s/it]
 41%|████      | 2489/6100 [3:45:32<5:25:44,  5.41s/it]
 41%|████      | 2490/6100 [3:45:37<5:25:56,  5.42s/it]
                                                       
{'loss': 0.0797, 'learning_rate': 0.0001, 'epoch': 4.08}

 41%|████      | 2490/6100 [3:45:37<5:25:56,  5.42s/it]
 41%|████      | 2491/6100 [3:45:43<5:25:54,  5.42s/it]
 41%|████      | 2492/6100 [3:45:48<5:25:47,  5.42s/it]
 41%|████      | 2493/6100 [3:45:54<5:25:39,  5.42s/it]
 41%|████      | 2494/6100 [3:45:59<5:25:25,  5.41s/it]
 41%|████      | 2495/6100 [3:46:04<5:25:34,  5.42s/it]
                                                       
{'loss': 0.0807, 'learning_rate': 0.0001, 'epoch': 4.09}

 41%|████      | 2495/6100 [3:46:04<5:25:34,  5.42s/it]
 41%|████      | 2496/6100 [3:46:10<5:28:11,  5.46s/it]
 41%|████      | 2497/6100 [3:46:15<5:27:23,  5.45s/it]
 41%|████      | 2498/6100 [3:46:21<5:26:54,  5.45s/it]
 41%|████      | 2499/6100 [3:46:26<5:26:42,  5.44s/it]
 41%|████      | 2500/6100 [3:46:32<5:25:53,  5.43s/it]
                                                       
{'loss': 0.0793, 'learning_rate': 0.0001, 'epoch': 4.1}

 41%|████      | 2500/6100 [3:46:32<5:25:53,  5.43s/it]
 41%|████      | 2501/6100 [3:46:37<5:25:42,  5.43s/it]
 41%|████      | 2502/6100 [3:46:43<5:25:14,  5.42s/it]
 41%|████      | 2503/6100 [3:46:48<5:24:53,  5.42s/it]
 41%|████      | 2504/6100 [3:46:53<5:24:26,  5.41s/it]
 41%|████      | 2505/6100 [3:46:59<5:24:23,  5.41s/it]
                                                       
{'loss': 0.0813, 'learning_rate': 0.0001, 'epoch': 4.11}

 41%|████      | 2505/6100 [3:46:59<5:24:23,  5.41s/it]
 41%|████      | 2506/6100 [3:47:04<5:24:25,  5.42s/it]
 41%|████      | 2507/6100 [3:47:10<5:24:17,  5.42s/it]
 41%|████      | 2508/6100 [3:47:15<5:24:15,  5.42s/it]
 41%|████      | 2509/6100 [3:47:20<5:23:52,  5.41s/it]
 41%|████      | 2510/6100 [3:47:26<5:24:11,  5.42s/it]
                                                       
{'loss': 0.0827, 'learning_rate': 0.0001, 'epoch': 4.11}

 41%|████      | 2510/6100 [3:47:26<5:24:11,  5.42s/it]
 41%|████      | 2511/6100 [3:47:31<5:24:14,  5.42s/it]
 41%|████      | 2512/6100 [3:47:37<5:24:18,  5.42s/it]
 41%|████      | 2513/6100 [3:47:42<5:23:50,  5.42s/it]
 41%|████      | 2514/6100 [3:47:48<5:23:27,  5.41s/it]
 41%|████      | 2515/6100 [3:47:53<5:23:28,  5.41s/it]
                                                       
{'loss': 0.0847, 'learning_rate': 0.0001, 'epoch': 4.12}

 41%|████      | 2515/6100 [3:47:53<5:23:28,  5.41s/it]
 41%|████      | 2516/6100 [3:47:58<5:23:10,  5.41s/it]
 41%|████▏     | 2517/6100 [3:48:04<5:23:11,  5.41s/it]
 41%|████▏     | 2518/6100 [3:48:09<5:23:16,  5.41s/it]
 41%|████▏     | 2519/6100 [3:48:15<5:22:53,  5.41s/it]
 41%|████▏     | 2520/6100 [3:48:20<5:22:36,  5.41s/it]
                                                       
{'loss': 0.0868, 'learning_rate': 0.0001, 'epoch': 4.13}

 41%|████▏     | 2520/6100 [3:48:20<5:22:36,  5.41s/it]
 41%|████▏     | 2521/6100 [3:48:25<5:22:54,  5.41s/it]
 41%|████▏     | 2522/6100 [3:48:31<5:22:54,  5.42s/it]
 41%|████▏     | 2523/6100 [3:48:36<5:22:44,  5.41s/it]
 41%|████▏     | 2524/6100 [3:48:42<5:22:44,  5.42s/it]
 41%|████▏     | 2525/6100 [3:48:47<5:22:41,  5.42s/it]
                                                       
{'loss': 0.0842, 'learning_rate': 0.0001, 'epoch': 4.14}

 41%|████▏     | 2525/6100 [3:48:47<5:22:41,  5.42s/it]
 41%|████▏     | 2526/6100 [3:48:53<5:22:32,  5.41s/it]
 41%|████▏     | 2527/6100 [3:48:58<5:22:45,  5.42s/it]
 41%|████▏     | 2528/6100 [3:49:03<5:22:23,  5.42s/it]
 41%|████▏     | 2529/6100 [3:49:09<5:22:25,  5.42s/it]
 41%|████▏     | 2530/6100 [3:49:14<5:22:24,  5.42s/it]
                                                       
{'loss': 0.0811, 'learning_rate': 0.0001, 'epoch': 4.15}

 41%|████▏     | 2530/6100 [3:49:14<5:22:24,  5.42s/it]
 41%|████▏     | 2531/6100 [3:49:20<5:22:00,  5.41s/it]
 42%|████▏     | 2532/6100 [3:49:25<5:21:56,  5.41s/it]
 42%|████▏     | 2533/6100 [3:49:30<5:21:53,  5.41s/it]
 42%|████▏     | 2534/6100 [3:49:36<5:21:40,  5.41s/it]
 42%|████▏     | 2535/6100 [3:49:41<5:21:27,  5.41s/it]
                                                       
{'loss': 0.0806, 'learning_rate': 0.0001, 'epoch': 4.16}

 42%|████▏     | 2535/6100 [3:49:41<5:21:27,  5.41s/it]
 42%|████▏     | 2536/6100 [3:49:47<5:21:14,  5.41s/it]
 42%|████▏     | 2537/6100 [3:49:52<5:21:06,  5.41s/it]
 42%|████▏     | 2538/6100 [3:49:57<5:21:11,  5.41s/it]
 42%|████▏     | 2539/6100 [3:50:03<5:20:54,  5.41s/it]
 42%|████▏     | 2540/6100 [3:50:08<5:20:52,  5.41s/it]
                                                       
{'loss': 0.0845, 'learning_rate': 0.0001, 'epoch': 4.16}

 42%|████▏     | 2540/6100 [3:50:08<5:20:52,  5.41s/it]
 42%|████▏     | 2541/6100 [3:50:14<5:21:10,  5.41s/it]
 42%|████▏     | 2542/6100 [3:50:19<5:21:00,  5.41s/it]
 42%|████▏     | 2543/6100 [3:50:25<5:20:47,  5.41s/it]
 42%|████▏     | 2544/6100 [3:50:30<5:20:38,  5.41s/it]
 42%|████▏     | 2545/6100 [3:50:35<5:21:28,  5.43s/it]
                                                       
{'loss': 0.0792, 'learning_rate': 0.0001, 'epoch': 4.17}

 42%|████▏     | 2545/6100 [3:50:35<5:21:28,  5.43s/it]
 42%|████▏     | 2546/6100 [3:50:41<5:21:01,  5.42s/it]
 42%|████▏     | 2547/6100 [3:50:46<5:20:43,  5.42s/it]
 42%|████▏     | 2548/6100 [3:50:52<5:20:27,  5.41s/it]
 42%|████▏     | 2549/6100 [3:50:57<5:20:45,  5.42s/it]
 42%|████▏     | 2550/6100 [3:51:02<5:20:50,  5.42s/it]
                                                       
{'loss': 0.0859, 'learning_rate': 0.0001, 'epoch': 4.18}

 42%|████▏     | 2550/6100 [3:51:02<5:20:50,  5.42s/it]Saving model checkpoint to ./results/checkpoint-2550
Configuration saved in ./results/checkpoint-2550/config.json
Model weights saved in ./results/checkpoint-2550/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-2550/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-2550/special_tokens_map.json
[2023-02-21 23:43:18,642] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step2552 is begin to save!
[2023-02-21 23:43:18,645] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-2550/global_step2552/mp_rank_00_model_states.pt
[2023-02-21 23:43:18,645] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-2550/global_step2552/mp_rank_00_model_states.pt...
[2023-02-21 23:43:19,460] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-2550/global_step2552/mp_rank_00_model_states.pt.
[2023-02-21 23:43:19,462] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-2550/global_step2552/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-21 23:43:19,636] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-2550/global_step2552/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-21 23:43:19,637] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-2550/global_step2552/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-21 23:43:19,637] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2552 is ready now!
Deleting older checkpoint [results/checkpoint-2400] due to args.save_total_limit

 42%|████▏     | 2551/6100 [3:51:10<5:58:55,  6.07s/it]
 42%|████▏     | 2552/6100 [3:51:15<5:47:19,  5.87s/it]
 42%|████▏     | 2553/6100 [3:51:21<5:39:08,  5.74s/it]
 42%|████▏     | 2554/6100 [3:51:26<5:33:15,  5.64s/it]
 42%|████▏     | 2555/6100 [3:51:32<5:29:17,  5.57s/it]
                                                       
{'loss': 0.0802, 'learning_rate': 0.0001, 'epoch': 4.19}

 42%|████▏     | 2555/6100 [3:51:32<5:29:17,  5.57s/it]
 42%|████▏     | 2556/6100 [3:51:37<5:26:28,  5.53s/it]
 42%|████▏     | 2557/6100 [3:51:43<5:24:19,  5.49s/it]
 42%|████▏     | 2558/6100 [3:51:48<5:22:28,  5.46s/it]
 42%|████▏     | 2559/6100 [3:51:53<5:21:38,  5.45s/it]
 42%|████▏     | 2560/6100 [3:51:59<5:20:58,  5.44s/it]
                                                       
{'loss': 0.0786, 'learning_rate': 0.0001, 'epoch': 4.2}

 42%|████▏     | 2560/6100 [3:51:59<5:20:58,  5.44s/it]
 42%|████▏     | 2561/6100 [3:52:04<5:20:24,  5.43s/it]
 42%|████▏     | 2562/6100 [3:52:10<5:19:50,  5.42s/it]
 42%|████▏     | 2563/6100 [3:52:15<5:19:20,  5.42s/it]
 42%|████▏     | 2564/6100 [3:52:20<5:19:14,  5.42s/it]
 42%|████▏     | 2565/6100 [3:52:26<5:20:16,  5.44s/it]
                                                       
{'loss': 0.0806, 'learning_rate': 0.0001, 'epoch': 4.2}

 42%|████▏     | 2565/6100 [3:52:26<5:20:16,  5.44s/it]
 42%|████▏     | 2566/6100 [3:52:31<5:19:47,  5.43s/it]
 42%|████▏     | 2567/6100 [3:52:37<5:19:25,  5.42s/it]
 42%|████▏     | 2568/6100 [3:52:42<5:19:13,  5.42s/it]
 42%|████▏     | 2569/6100 [3:52:48<5:18:54,  5.42s/it]
 42%|████▏     | 2570/6100 [3:52:53<5:18:56,  5.42s/it]
                                                       
{'loss': 0.0788, 'learning_rate': 0.0001, 'epoch': 4.21}

 42%|████▏     | 2570/6100 [3:52:53<5:18:56,  5.42s/it]
 42%|████▏     | 2571/6100 [3:52:58<5:18:29,  5.41s/it]
 42%|████▏     | 2572/6100 [3:53:04<5:18:20,  5.41s/it]
 42%|████▏     | 2573/6100 [3:53:09<5:18:08,  5.41s/it]
 42%|████▏     | 2574/6100 [3:53:15<5:17:50,  5.41s/it]
 42%|████▏     | 2575/6100 [3:53:20<5:17:44,  5.41s/it]
                                                       
{'loss': 0.083, 'learning_rate': 0.0001, 'epoch': 4.22}

 42%|████▏     | 2575/6100 [3:53:20<5:17:44,  5.41s/it]
 42%|████▏     | 2576/6100 [3:53:25<5:17:46,  5.41s/it]
 42%|████▏     | 2577/6100 [3:53:31<5:17:34,  5.41s/it]
 42%|████▏     | 2578/6100 [3:53:36<5:17:43,  5.41s/it]
 42%|████▏     | 2579/6100 [3:53:42<5:17:33,  5.41s/it]
 42%|████▏     | 2580/6100 [3:53:47<5:17:35,  5.41s/it]
                                                       
{'loss': 0.0813, 'learning_rate': 0.0001, 'epoch': 4.23}

 42%|████▏     | 2580/6100 [3:53:47<5:17:35,  5.41s/it]
 42%|████▏     | 2581/6100 [3:53:52<5:17:42,  5.42s/it]
 42%|████▏     | 2582/6100 [3:53:58<5:17:34,  5.42s/it]
 42%|████▏     | 2583/6100 [3:54:03<5:17:14,  5.41s/it]
 42%|████▏     | 2584/6100 [3:54:09<5:17:13,  5.41s/it]
 42%|████▏     | 2585/6100 [3:54:14<5:16:52,  5.41s/it]
                                                       
{'loss': 0.0816, 'learning_rate': 0.0001, 'epoch': 4.24}

 42%|████▏     | 2585/6100 [3:54:14<5:16:52,  5.41s/it]
 42%|████▏     | 2586/6100 [3:54:20<5:16:49,  5.41s/it]
 42%|████▏     | 2587/6100 [3:54:25<5:17:00,  5.41s/it]
 42%|████▏     | 2588/6100 [3:54:30<5:16:49,  5.41s/it]
 42%|████▏     | 2589/6100 [3:54:36<5:16:40,  5.41s/it]
 42%|████▏     | 2590/6100 [3:54:41<5:16:16,  5.41s/it]
                                                       
{'loss': 0.0808, 'learning_rate': 0.0001, 'epoch': 4.25}

 42%|████▏     | 2590/6100 [3:54:41<5:16:16,  5.41s/it]
 42%|████▏     | 2591/6100 [3:54:47<5:16:22,  5.41s/it]
 42%|████▏     | 2592/6100 [3:54:52<5:16:11,  5.41s/it]
 43%|████▎     | 2593/6100 [3:54:57<5:16:09,  5.41s/it]
 43%|████▎     | 2594/6100 [3:55:03<5:16:28,  5.42s/it]
 43%|████▎     | 2595/6100 [3:55:08<5:16:30,  5.42s/it]
                                                       
{'loss': 0.085, 'learning_rate': 0.0001, 'epoch': 4.25}

 43%|████▎     | 2595/6100 [3:55:08<5:16:30,  5.42s/it]
 43%|████▎     | 2596/6100 [3:55:14<5:16:24,  5.42s/it]
 43%|████▎     | 2597/6100 [3:55:19<5:16:36,  5.42s/it]
 43%|████▎     | 2598/6100 [3:55:25<5:16:11,  5.42s/it]
 43%|████▎     | 2599/6100 [3:55:30<5:15:58,  5.42s/it]
 43%|████▎     | 2600/6100 [3:55:35<5:15:48,  5.41s/it]
                                                       
{'loss': 0.0867, 'learning_rate': 0.0001, 'epoch': 4.26}

 43%|████▎     | 2600/6100 [3:55:35<5:15:48,  5.41s/it]
 43%|████▎     | 2601/6100 [3:55:41<5:15:53,  5.42s/it]
 43%|████▎     | 2602/6100 [3:55:46<5:15:48,  5.42s/it]
 43%|████▎     | 2603/6100 [3:55:52<5:15:34,  5.41s/it]
 43%|████▎     | 2604/6100 [3:55:57<5:15:18,  5.41s/it]
 43%|████▎     | 2605/6100 [3:56:02<5:15:15,  5.41s/it]
                                                       
{'loss': 0.0821, 'learning_rate': 0.0001, 'epoch': 4.27}

 43%|████▎     | 2605/6100 [3:56:02<5:15:15,  5.41s/it]
 43%|████▎     | 2606/6100 [3:56:08<5:15:30,  5.42s/it]
 43%|████▎     | 2607/6100 [3:56:13<5:15:33,  5.42s/it]
 43%|████▎     | 2608/6100 [3:56:19<5:15:19,  5.42s/it]
 43%|████▎     | 2609/6100 [3:56:24<5:15:43,  5.43s/it]
 43%|████▎     | 2610/6100 [3:56:30<5:15:27,  5.42s/it]
                                                       
{'loss': 0.082, 'learning_rate': 0.0001, 'epoch': 4.28}

 43%|████▎     | 2610/6100 [3:56:30<5:15:27,  5.42s/it]
 43%|████▎     | 2611/6100 [3:56:35<5:15:21,  5.42s/it]
 43%|████▎     | 2612/6100 [3:56:40<5:15:11,  5.42s/it]
 43%|████▎     | 2613/6100 [3:56:46<5:14:46,  5.42s/it]
 43%|████▎     | 2614/6100 [3:56:51<5:14:36,  5.41s/it]
 43%|████▎     | 2615/6100 [3:56:57<5:14:38,  5.42s/it]
                                                       
{'loss': 0.0885, 'learning_rate': 0.0001, 'epoch': 4.29}

 43%|████▎     | 2615/6100 [3:56:57<5:14:38,  5.42s/it]
 43%|████▎     | 2616/6100 [3:57:02<5:14:24,  5.41s/it]
 43%|████▎     | 2617/6100 [3:57:07<5:14:09,  5.41s/it]
 43%|████▎     | 2618/6100 [3:57:13<5:14:07,  5.41s/it]
 43%|████▎     | 2619/6100 [3:57:18<5:13:55,  5.41s/it]
 43%|████▎     | 2620/6100 [3:57:24<5:13:36,  5.41s/it]
                                                       
{'loss': 0.0766, 'learning_rate': 0.0001, 'epoch': 4.29}

 43%|████▎     | 2620/6100 [3:57:24<5:13:36,  5.41s/it]
 43%|████▎     | 2621/6100 [3:57:29<5:13:39,  5.41s/it]
 43%|████▎     | 2622/6100 [3:57:34<5:13:41,  5.41s/it]
 43%|████▎     | 2623/6100 [3:57:40<5:13:27,  5.41s/it]
 43%|████▎     | 2624/6100 [3:57:45<5:13:26,  5.41s/it]
 43%|████▎     | 2625/6100 [3:57:51<5:13:24,  5.41s/it]
                                                       
{'loss': 0.0836, 'learning_rate': 0.0001, 'epoch': 4.3}

 43%|████▎     | 2625/6100 [3:57:51<5:13:24,  5.41s/it]
 43%|████▎     | 2626/6100 [3:57:56<5:13:35,  5.42s/it]
 43%|████▎     | 2627/6100 [3:58:02<5:13:34,  5.42s/it]
 43%|████▎     | 2628/6100 [3:58:07<5:13:32,  5.42s/it]
 43%|████▎     | 2629/6100 [3:58:12<5:13:24,  5.42s/it]
 43%|████▎     | 2630/6100 [3:58:18<5:13:31,  5.42s/it]
                                                       
{'loss': 0.0821, 'learning_rate': 0.0001, 'epoch': 4.31}

 43%|████▎     | 2630/6100 [3:58:18<5:13:31,  5.42s/it]
 43%|████▎     | 2631/6100 [3:58:23<5:13:39,  5.43s/it]
 43%|████▎     | 2632/6100 [3:58:29<5:13:17,  5.42s/it]
 43%|████▎     | 2633/6100 [3:58:34<5:13:14,  5.42s/it]
 43%|████▎     | 2634/6100 [3:58:40<5:13:00,  5.42s/it]
 43%|████▎     | 2635/6100 [3:58:45<5:12:47,  5.42s/it]
                                                       
{'loss': 0.0787, 'learning_rate': 0.0001, 'epoch': 4.32}

 43%|████▎     | 2635/6100 [3:58:45<5:12:47,  5.42s/it]
 43%|████▎     | 2636/6100 [3:58:50<5:12:35,  5.41s/it]
 43%|████▎     | 2637/6100 [3:58:56<5:12:22,  5.41s/it]
 43%|████▎     | 2638/6100 [3:59:01<5:16:37,  5.49s/it]
 43%|████▎     | 2639/6100 [3:59:07<5:15:06,  5.46s/it]
 43%|████▎     | 2640/6100 [3:59:12<5:14:13,  5.45s/it]
                                                       
{'loss': 0.0809, 'learning_rate': 0.0001, 'epoch': 4.33}

 43%|████▎     | 2640/6100 [3:59:12<5:14:13,  5.45s/it]
 43%|████▎     | 2641/6100 [3:59:18<5:13:25,  5.44s/it]
 43%|████▎     | 2642/6100 [3:59:23<5:12:51,  5.43s/it]
 43%|████▎     | 2643/6100 [3:59:28<5:12:21,  5.42s/it]
 43%|████▎     | 2644/6100 [3:59:34<5:11:58,  5.42s/it]
 43%|████▎     | 2645/6100 [3:59:39<5:11:46,  5.41s/it]
                                                       
{'loss': 0.0791, 'learning_rate': 0.0001, 'epoch': 4.34}

 43%|████▎     | 2645/6100 [3:59:39<5:11:46,  5.41s/it]
 43%|████▎     | 2646/6100 [3:59:45<5:11:41,  5.41s/it]
 43%|████▎     | 2647/6100 [3:59:50<5:11:28,  5.41s/it]
 43%|████▎     | 2648/6100 [3:59:56<5:11:34,  5.42s/it]
 43%|████▎     | 2649/6100 [4:00:01<5:11:26,  5.41s/it]
 43%|████▎     | 2650/6100 [4:00:06<5:11:26,  5.42s/it]
                                                       
{'loss': 0.0803, 'learning_rate': 0.0001, 'epoch': 4.34}

 43%|████▎     | 2650/6100 [4:00:06<5:11:26,  5.42s/it]
 43%|████▎     | 2651/6100 [4:00:12<5:11:03,  5.41s/it]
 43%|████▎     | 2652/6100 [4:00:17<5:15:59,  5.50s/it]
 43%|████▎     | 2653/6100 [4:00:23<5:14:38,  5.48s/it]
 44%|████▎     | 2654/6100 [4:00:28<5:13:20,  5.46s/it]
 44%|████▎     | 2655/6100 [4:00:34<5:12:32,  5.44s/it]
                                                       
{'loss': 0.0818, 'learning_rate': 0.0001, 'epoch': 4.35}

 44%|████▎     | 2655/6100 [4:00:34<5:12:32,  5.44s/it]
 44%|████▎     | 2656/6100 [4:00:39<5:12:16,  5.44s/it]
 44%|████▎     | 2657/6100 [4:00:45<5:11:45,  5.43s/it]
 44%|████▎     | 2658/6100 [4:00:50<5:11:08,  5.42s/it]
 44%|████▎     | 2659/6100 [4:00:55<5:11:07,  5.43s/it]
 44%|████▎     | 2660/6100 [4:01:01<5:10:36,  5.42s/it]
                                                       
{'loss': 0.087, 'learning_rate': 0.0001, 'epoch': 4.36}

 44%|████▎     | 2660/6100 [4:01:01<5:10:36,  5.42s/it]
 44%|████▎     | 2661/6100 [4:01:06<5:10:44,  5.42s/it]
 44%|████▎     | 2662/6100 [4:01:12<5:10:45,  5.42s/it]
 44%|████▎     | 2663/6100 [4:01:17<5:10:49,  5.43s/it]
 44%|████▎     | 2664/6100 [4:01:22<5:10:39,  5.42s/it]
 44%|████▎     | 2665/6100 [4:01:28<5:10:13,  5.42s/it]
                                                       
{'loss': 0.0808, 'learning_rate': 0.0001, 'epoch': 4.37}

 44%|████▎     | 2665/6100 [4:01:28<5:10:13,  5.42s/it]
 44%|████▎     | 2666/6100 [4:01:33<5:10:03,  5.42s/it]
 44%|████▎     | 2667/6100 [4:01:39<5:09:55,  5.42s/it]
 44%|████▎     | 2668/6100 [4:01:44<5:10:11,  5.42s/it]
 44%|████▍     | 2669/6100 [4:01:50<5:09:54,  5.42s/it]
 44%|████▍     | 2670/6100 [4:01:55<5:09:39,  5.42s/it]
                                                       
{'loss': 0.0841, 'learning_rate': 0.0001, 'epoch': 4.38}

 44%|████▍     | 2670/6100 [4:01:55<5:09:39,  5.42s/it]
 44%|████▍     | 2671/6100 [4:02:00<5:09:44,  5.42s/it]
 44%|████▍     | 2672/6100 [4:02:06<5:09:57,  5.43s/it]
 44%|████▍     | 2673/6100 [4:02:11<5:09:37,  5.42s/it]
 44%|████▍     | 2674/6100 [4:02:17<5:09:14,  5.42s/it]
 44%|████▍     | 2675/6100 [4:02:22<5:09:40,  5.42s/it]
                                                       
{'loss': 0.0828, 'learning_rate': 0.0001, 'epoch': 4.38}

 44%|████▍     | 2675/6100 [4:02:22<5:09:40,  5.42s/it]
 44%|████▍     | 2676/6100 [4:02:28<5:09:28,  5.42s/it]
 44%|████▍     | 2677/6100 [4:02:33<5:09:05,  5.42s/it]
 44%|████▍     | 2678/6100 [4:02:38<5:08:56,  5.42s/it]
 44%|████▍     | 2679/6100 [4:02:44<5:09:16,  5.42s/it]
 44%|████▍     | 2680/6100 [4:02:49<5:09:04,  5.42s/it]
                                                       
{'loss': 0.0789, 'learning_rate': 0.0001, 'epoch': 4.39}

 44%|████▍     | 2680/6100 [4:02:49<5:09:04,  5.42s/it]
 44%|████▍     | 2681/6100 [4:02:55<5:08:49,  5.42s/it]
 44%|████▍     | 2682/6100 [4:03:00<5:08:35,  5.42s/it]
 44%|████▍     | 2683/6100 [4:03:05<5:08:48,  5.42s/it]
 44%|████▍     | 2684/6100 [4:03:11<5:08:28,  5.42s/it]
 44%|████▍     | 2685/6100 [4:03:16<5:08:20,  5.42s/it]
                                                       
{'loss': 0.0914, 'learning_rate': 0.0001, 'epoch': 4.4}

 44%|████▍     | 2685/6100 [4:03:16<5:08:20,  5.42s/it]
 44%|████▍     | 2686/6100 [4:03:22<5:08:05,  5.41s/it]
 44%|████▍     | 2687/6100 [4:03:27<5:08:00,  5.41s/it]
 44%|████▍     | 2688/6100 [4:03:33<5:07:45,  5.41s/it]
 44%|████▍     | 2689/6100 [4:03:38<5:07:47,  5.41s/it]
 44%|████▍     | 2690/6100 [4:03:43<5:07:31,  5.41s/it]
                                                       
{'loss': 0.082, 'learning_rate': 0.0001, 'epoch': 4.41}

 44%|████▍     | 2690/6100 [4:03:43<5:07:31,  5.41s/it]
 44%|████▍     | 2691/6100 [4:03:49<5:07:31,  5.41s/it]
 44%|████▍     | 2692/6100 [4:03:54<5:07:21,  5.41s/it]
 44%|████▍     | 2693/6100 [4:04:00<5:07:25,  5.41s/it]
 44%|████▍     | 2694/6100 [4:04:05<5:07:11,  5.41s/it]
 44%|████▍     | 2695/6100 [4:04:10<5:07:13,  5.41s/it]
                                                       
{'loss': 0.0843, 'learning_rate': 0.0001, 'epoch': 4.42}

 44%|████▍     | 2695/6100 [4:04:10<5:07:13,  5.41s/it]
 44%|████▍     | 2696/6100 [4:04:16<5:07:01,  5.41s/it]
 44%|████▍     | 2697/6100 [4:04:21<5:07:14,  5.42s/it]
 44%|████▍     | 2698/6100 [4:04:27<5:06:55,  5.41s/it]
 44%|████▍     | 2699/6100 [4:04:32<5:06:32,  5.41s/it]
 44%|████▍     | 2700/6100 [4:04:37<5:06:57,  5.42s/it]
                                                       
{'loss': 0.085, 'learning_rate': 0.0001, 'epoch': 4.43}

 44%|████▍     | 2700/6100 [4:04:37<5:06:57,  5.42s/it]Saving model checkpoint to ./results/checkpoint-2700
Configuration saved in ./results/checkpoint-2700/config.json
Model weights saved in ./results/checkpoint-2700/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-2700/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-2700/special_tokens_map.json
[2023-02-21 23:56:53,653] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step2702 is begin to save!
[2023-02-21 23:56:53,656] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-2700/global_step2702/mp_rank_00_model_states.pt
[2023-02-21 23:56:53,656] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-2700/global_step2702/mp_rank_00_model_states.pt...
[2023-02-21 23:56:54,528] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-2700/global_step2702/mp_rank_00_model_states.pt.
[2023-02-21 23:56:54,529] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-2700/global_step2702/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-21 23:56:54,740] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-2700/global_step2702/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-21 23:56:54,741] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-2700/global_step2702/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-21 23:56:54,741] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2702 is ready now!
Deleting older checkpoint [results/checkpoint-2550] due to args.save_total_limit

 44%|████▍     | 2701/6100 [4:04:45<5:44:34,  6.08s/it]
 44%|████▍     | 2702/6100 [4:04:51<5:32:55,  5.88s/it]
 44%|████▍     | 2703/6100 [4:04:56<5:24:39,  5.73s/it]
 44%|████▍     | 2704/6100 [4:05:01<5:19:03,  5.64s/it]
 44%|████▍     | 2705/6100 [4:05:07<5:14:59,  5.57s/it]
                                                       
{'loss': 0.0849, 'learning_rate': 0.0001, 'epoch': 4.43}

 44%|████▍     | 2705/6100 [4:05:07<5:14:59,  5.57s/it]
 44%|████▍     | 2706/6100 [4:05:12<5:12:26,  5.52s/it]
 44%|████▍     | 2707/6100 [4:05:18<5:10:23,  5.49s/it]
 44%|████▍     | 2708/6100 [4:05:23<5:09:04,  5.47s/it]
 44%|████▍     | 2709/6100 [4:05:28<5:08:11,  5.45s/it]
 44%|████▍     | 2710/6100 [4:05:34<5:07:15,  5.44s/it]
                                                       
{'loss': 0.0803, 'learning_rate': 0.0001, 'epoch': 4.44}

 44%|████▍     | 2710/6100 [4:05:34<5:07:15,  5.44s/it]
 44%|████▍     | 2711/6100 [4:05:39<5:06:43,  5.43s/it]
 44%|████▍     | 2712/6100 [4:05:45<5:06:38,  5.43s/it]
 44%|████▍     | 2713/6100 [4:05:50<5:06:41,  5.43s/it]
 44%|████▍     | 2714/6100 [4:05:55<5:06:05,  5.42s/it]
 45%|████▍     | 2715/6100 [4:06:01<5:05:35,  5.42s/it]
                                                       
{'loss': 0.0794, 'learning_rate': 0.0001, 'epoch': 4.45}

 45%|████▍     | 2715/6100 [4:06:01<5:05:35,  5.42s/it]
 45%|████▍     | 2716/6100 [4:06:06<5:05:25,  5.42s/it]
 45%|████▍     | 2717/6100 [4:06:12<5:05:04,  5.41s/it]
 45%|████▍     | 2718/6100 [4:06:17<5:05:04,  5.41s/it]
 45%|████▍     | 2719/6100 [4:06:23<5:04:59,  5.41s/it]
 45%|████▍     | 2720/6100 [4:06:28<5:04:40,  5.41s/it]
                                                       
{'loss': 0.0802, 'learning_rate': 0.0001, 'epoch': 4.46}

 45%|████▍     | 2720/6100 [4:06:28<5:04:40,  5.41s/it]
 45%|████▍     | 2721/6100 [4:06:33<5:04:47,  5.41s/it]
 45%|████▍     | 2722/6100 [4:06:39<5:04:30,  5.41s/it]
 45%|████▍     | 2723/6100 [4:06:44<5:04:30,  5.41s/it]
 45%|████▍     | 2724/6100 [4:06:50<5:04:13,  5.41s/it]
 45%|████▍     | 2725/6100 [4:06:55<5:04:11,  5.41s/it]
                                                       
{'loss': 0.0797, 'learning_rate': 0.0001, 'epoch': 4.47}

 45%|████▍     | 2725/6100 [4:06:55<5:04:11,  5.41s/it]
 45%|████▍     | 2726/6100 [4:07:01<5:14:18,  5.59s/it]
 45%|████▍     | 2727/6100 [4:07:06<5:11:13,  5.54s/it]
 45%|████▍     | 2728/6100 [4:07:12<5:08:55,  5.50s/it]
 45%|████▍     | 2729/6100 [4:07:17<5:07:23,  5.47s/it]
 45%|████▍     | 2730/6100 [4:07:23<5:06:03,  5.45s/it]
                                                       
{'loss': 0.0824, 'learning_rate': 0.0001, 'epoch': 4.48}

 45%|████▍     | 2730/6100 [4:07:23<5:06:03,  5.45s/it]
 45%|████▍     | 2731/6100 [4:07:28<5:05:29,  5.44s/it]
 45%|████▍     | 2732/6100 [4:07:33<5:05:07,  5.44s/it]
 45%|████▍     | 2733/6100 [4:07:39<5:04:32,  5.43s/it]
 45%|████▍     | 2734/6100 [4:07:44<5:04:21,  5.43s/it]
 45%|████▍     | 2735/6100 [4:07:50<5:04:01,  5.42s/it]
                                                       
{'loss': 0.0845, 'learning_rate': 0.0001, 'epoch': 4.48}

 45%|████▍     | 2735/6100 [4:07:50<5:04:01,  5.42s/it]
 45%|████▍     | 2736/6100 [4:07:55<5:03:42,  5.42s/it]
 45%|████▍     | 2737/6100 [4:08:01<5:03:39,  5.42s/it]
 45%|████▍     | 2738/6100 [4:08:06<5:04:12,  5.43s/it]
 45%|████▍     | 2739/6100 [4:08:11<5:03:48,  5.42s/it]
 45%|████▍     | 2740/6100 [4:08:17<5:03:39,  5.42s/it]
                                                       
{'loss': 0.0849, 'learning_rate': 0.0001, 'epoch': 4.49}

 45%|████▍     | 2740/6100 [4:08:17<5:03:39,  5.42s/it]
 45%|████▍     | 2741/6100 [4:08:22<5:03:25,  5.42s/it]
 45%|████▍     | 2742/6100 [4:08:28<5:03:21,  5.42s/it]
 45%|████▍     | 2743/6100 [4:08:33<5:03:02,  5.42s/it]
 45%|████▍     | 2744/6100 [4:08:38<5:02:46,  5.41s/it]
 45%|████▌     | 2745/6100 [4:08:44<5:02:51,  5.42s/it]
                                                       
{'loss': 0.0825, 'learning_rate': 0.0001, 'epoch': 4.5}

 45%|████▌     | 2745/6100 [4:08:44<5:02:51,  5.42s/it]
 45%|████▌     | 2746/6100 [4:08:49<5:03:02,  5.42s/it]
 45%|████▌     | 2747/6100 [4:08:55<5:03:12,  5.43s/it]
 45%|████▌     | 2748/6100 [4:09:00<5:02:57,  5.42s/it]
 45%|████▌     | 2749/6100 [4:09:06<5:03:01,  5.43s/it]
 45%|████▌     | 2750/6100 [4:09:11<5:04:45,  5.46s/it]
                                                       
{'loss': 0.0799, 'learning_rate': 0.0001, 'epoch': 4.51}

 45%|████▌     | 2750/6100 [4:09:11<5:04:45,  5.46s/it]
 45%|████▌     | 2751/6100 [4:09:17<5:03:52,  5.44s/it]
 45%|████▌     | 2752/6100 [4:09:22<5:03:21,  5.44s/it]
 45%|████▌     | 2753/6100 [4:09:27<5:02:46,  5.43s/it]
 45%|████▌     | 2754/6100 [4:09:33<5:02:18,  5.42s/it]
 45%|████▌     | 2755/6100 [4:09:38<5:02:05,  5.42s/it]
                                                       
{'loss': 0.0825, 'learning_rate': 0.0001, 'epoch': 4.52}

 45%|████▌     | 2755/6100 [4:09:38<5:02:05,  5.42s/it]
 45%|████▌     | 2756/6100 [4:09:44<5:01:44,  5.41s/it]
 45%|████▌     | 2757/6100 [4:09:49<5:01:38,  5.41s/it]
 45%|████▌     | 2758/6100 [4:09:54<5:01:33,  5.41s/it]
 45%|████▌     | 2759/6100 [4:10:00<5:01:37,  5.42s/it]
 45%|████▌     | 2760/6100 [4:10:05<5:01:33,  5.42s/it]
                                                       
{'loss': 0.0885, 'learning_rate': 0.0001, 'epoch': 4.52}

 45%|████▌     | 2760/6100 [4:10:05<5:01:33,  5.42s/it]
 45%|████▌     | 2761/6100 [4:10:11<5:01:32,  5.42s/it]
 45%|████▌     | 2762/6100 [4:10:16<5:01:19,  5.42s/it]
 45%|████▌     | 2763/6100 [4:10:21<5:01:16,  5.42s/it]
 45%|████▌     | 2764/6100 [4:10:27<5:01:13,  5.42s/it]
 45%|████▌     | 2765/6100 [4:10:32<5:01:04,  5.42s/it]
                                                       
{'loss': 0.0778, 'learning_rate': 0.0001, 'epoch': 4.53}

 45%|████▌     | 2765/6100 [4:10:32<5:01:04,  5.42s/it]
 45%|████▌     | 2766/6100 [4:10:38<5:00:47,  5.41s/it]
 45%|████▌     | 2767/6100 [4:10:43<5:00:38,  5.41s/it]
 45%|████▌     | 2768/6100 [4:10:49<5:00:28,  5.41s/it]
 45%|████▌     | 2769/6100 [4:10:54<5:00:27,  5.41s/it]
 45%|████▌     | 2770/6100 [4:10:59<5:00:29,  5.41s/it]
                                                       
{'loss': 0.0794, 'learning_rate': 0.0001, 'epoch': 4.54}

 45%|████▌     | 2770/6100 [4:10:59<5:00:29,  5.41s/it]
 45%|████▌     | 2771/6100 [4:11:05<5:00:19,  5.41s/it]
 45%|████▌     | 2772/6100 [4:11:10<5:00:17,  5.41s/it]
 45%|████▌     | 2773/6100 [4:11:16<5:00:06,  5.41s/it]
 45%|████▌     | 2774/6100 [4:11:21<4:59:56,  5.41s/it]
 45%|████▌     | 2775/6100 [4:11:26<5:00:18,  5.42s/it]
                                                       
{'loss': 0.0798, 'learning_rate': 0.0001, 'epoch': 4.55}

 45%|████▌     | 2775/6100 [4:11:26<5:00:18,  5.42s/it]
 46%|████▌     | 2776/6100 [4:11:32<5:00:17,  5.42s/it]
 46%|████▌     | 2777/6100 [4:11:37<5:00:11,  5.42s/it]
 46%|████▌     | 2778/6100 [4:11:43<5:00:01,  5.42s/it]
 46%|████▌     | 2779/6100 [4:11:48<4:59:45,  5.42s/it]
 46%|████▌     | 2780/6100 [4:11:54<4:59:51,  5.42s/it]
                                                       
{'loss': 0.0794, 'learning_rate': 0.0001, 'epoch': 4.56}

 46%|████▌     | 2780/6100 [4:11:54<4:59:51,  5.42s/it]
 46%|████▌     | 2781/6100 [4:11:59<4:59:42,  5.42s/it]
 46%|████▌     | 2782/6100 [4:12:04<5:00:04,  5.43s/it]
 46%|████▌     | 2783/6100 [4:12:10<4:59:47,  5.42s/it]
 46%|████▌     | 2784/6100 [4:12:15<4:59:28,  5.42s/it]
 46%|████▌     | 2785/6100 [4:12:21<4:59:20,  5.42s/it]
                                                       
{'loss': 0.0807, 'learning_rate': 0.0001, 'epoch': 4.57}

 46%|████▌     | 2785/6100 [4:12:21<4:59:20,  5.42s/it]
 46%|████▌     | 2786/6100 [4:12:26<4:59:03,  5.41s/it]
 46%|████▌     | 2787/6100 [4:12:31<4:58:49,  5.41s/it]
 46%|████▌     | 2788/6100 [4:12:37<4:58:45,  5.41s/it]
 46%|████▌     | 2789/6100 [4:12:42<4:58:36,  5.41s/it]
 46%|████▌     | 2790/6100 [4:12:48<4:58:31,  5.41s/it]
                                                       
{'loss': 0.0736, 'learning_rate': 0.0001, 'epoch': 4.57}

 46%|████▌     | 2790/6100 [4:12:48<4:58:31,  5.41s/it]
 46%|████▌     | 2791/6100 [4:12:53<4:58:36,  5.41s/it]
 46%|████▌     | 2792/6100 [4:12:59<4:58:36,  5.42s/it]
 46%|████▌     | 2793/6100 [4:13:04<4:58:45,  5.42s/it]
 46%|████▌     | 2794/6100 [4:13:09<4:58:50,  5.42s/it]
 46%|████▌     | 2795/6100 [4:13:15<4:58:31,  5.42s/it]
                                                       
{'loss': 0.0771, 'learning_rate': 0.0001, 'epoch': 4.58}

 46%|████▌     | 2795/6100 [4:13:15<4:58:31,  5.42s/it]
 46%|████▌     | 2796/6100 [4:13:20<4:58:27,  5.42s/it]
 46%|████▌     | 2797/6100 [4:13:26<4:58:11,  5.42s/it]
 46%|████▌     | 2798/6100 [4:13:31<4:58:12,  5.42s/it]
 46%|████▌     | 2799/6100 [4:13:36<4:58:00,  5.42s/it]
 46%|████▌     | 2800/6100 [4:13:42<4:57:52,  5.42s/it]
                                                       
{'loss': 0.0779, 'learning_rate': 0.0001, 'epoch': 4.59}

 46%|████▌     | 2800/6100 [4:13:42<4:57:52,  5.42s/it]
 46%|████▌     | 2801/6100 [4:13:47<4:57:44,  5.41s/it]
 46%|████▌     | 2802/6100 [4:13:53<4:57:54,  5.42s/it]
 46%|████▌     | 2803/6100 [4:13:58<4:57:33,  5.41s/it]
 46%|████▌     | 2804/6100 [4:14:04<4:57:38,  5.42s/it]
 46%|████▌     | 2805/6100 [4:14:09<4:57:04,  5.41s/it]
                                                       
{'loss': 0.0832, 'learning_rate': 0.0001, 'epoch': 4.6}

 46%|████▌     | 2805/6100 [4:14:09<4:57:04,  5.41s/it]
 46%|████▌     | 2806/6100 [4:14:14<4:56:54,  5.41s/it]
 46%|████▌     | 2807/6100 [4:14:20<4:56:51,  5.41s/it]
 46%|████▌     | 2808/6100 [4:14:25<4:56:56,  5.41s/it]
 46%|████▌     | 2809/6100 [4:14:31<4:56:46,  5.41s/it]
 46%|████▌     | 2810/6100 [4:14:36<4:56:41,  5.41s/it]
                                                       
{'loss': 0.0794, 'learning_rate': 0.0001, 'epoch': 4.61}

 46%|████▌     | 2810/6100 [4:14:36<4:56:41,  5.41s/it]
 46%|████▌     | 2811/6100 [4:14:41<4:56:39,  5.41s/it]
 46%|████▌     | 2812/6100 [4:14:47<4:56:37,  5.41s/it]
 46%|████▌     | 2813/6100 [4:14:52<4:56:28,  5.41s/it]
 46%|████▌     | 2814/6100 [4:14:58<4:56:25,  5.41s/it]
 46%|████▌     | 2815/6100 [4:15:03<4:56:29,  5.42s/it]
                                                       
{'loss': 0.079, 'learning_rate': 0.0001, 'epoch': 4.61}

 46%|████▌     | 2815/6100 [4:15:03<4:56:29,  5.42s/it]
 46%|████▌     | 2816/6100 [4:15:09<4:56:22,  5.41s/it]
 46%|████▌     | 2817/6100 [4:15:14<4:56:14,  5.41s/it]
 46%|████▌     | 2818/6100 [4:15:19<4:56:05,  5.41s/it]
 46%|████▌     | 2819/6100 [4:15:25<4:55:55,  5.41s/it]
 46%|████▌     | 2820/6100 [4:15:30<4:55:53,  5.41s/it]
                                                       
{'loss': 0.0788, 'learning_rate': 0.0001, 'epoch': 4.62}

 46%|████▌     | 2820/6100 [4:15:30<4:55:53,  5.41s/it]
 46%|████▌     | 2821/6100 [4:15:36<4:55:53,  5.41s/it]
 46%|████▋     | 2822/6100 [4:15:41<4:56:11,  5.42s/it]
 46%|████▋     | 2823/6100 [4:15:46<4:56:11,  5.42s/it]
 46%|████▋     | 2824/6100 [4:15:52<4:55:56,  5.42s/it]
 46%|████▋     | 2825/6100 [4:15:57<4:56:10,  5.43s/it]
                                                       
{'loss': 0.084, 'learning_rate': 0.0001, 'epoch': 4.63}

 46%|████▋     | 2825/6100 [4:15:57<4:56:10,  5.43s/it]
 46%|████▋     | 2826/6100 [4:16:03<4:56:04,  5.43s/it]
 46%|████▋     | 2827/6100 [4:16:08<4:55:57,  5.43s/it]
 46%|████▋     | 2828/6100 [4:16:14<4:55:35,  5.42s/it]
 46%|████▋     | 2829/6100 [4:16:19<4:55:19,  5.42s/it]
 46%|████▋     | 2830/6100 [4:16:24<4:55:16,  5.42s/it]
                                                       
{'loss': 0.0821, 'learning_rate': 0.0001, 'epoch': 4.64}

 46%|████▋     | 2830/6100 [4:16:24<4:55:16,  5.42s/it]
 46%|████▋     | 2831/6100 [4:16:30<4:55:02,  5.42s/it]
 46%|████▋     | 2832/6100 [4:16:35<4:55:02,  5.42s/it]
 46%|████▋     | 2833/6100 [4:16:41<4:54:42,  5.41s/it]
 46%|████▋     | 2834/6100 [4:16:46<4:54:50,  5.42s/it]
 46%|████▋     | 2835/6100 [4:16:51<4:54:54,  5.42s/it]
                                                       
{'loss': 0.0803, 'learning_rate': 0.0001, 'epoch': 4.65}

 46%|████▋     | 2835/6100 [4:16:51<4:54:54,  5.42s/it]
 46%|████▋     | 2836/6100 [4:16:57<4:54:39,  5.42s/it]
 47%|████▋     | 2837/6100 [4:17:02<4:54:27,  5.41s/it]
 47%|████▋     | 2838/6100 [4:17:08<4:54:15,  5.41s/it]
 47%|████▋     | 2839/6100 [4:17:13<4:54:08,  5.41s/it]
 47%|████▋     | 2840/6100 [4:17:19<4:54:31,  5.42s/it]
                                                       
{'loss': 0.0847, 'learning_rate': 0.0001, 'epoch': 4.66}

 47%|████▋     | 2840/6100 [4:17:19<4:54:31,  5.42s/it]
 47%|████▋     | 2841/6100 [4:17:24<4:54:20,  5.42s/it]
 47%|████▋     | 2842/6100 [4:17:29<4:54:15,  5.42s/it]
 47%|████▋     | 2843/6100 [4:17:35<4:54:11,  5.42s/it]
 47%|████▋     | 2844/6100 [4:17:40<4:54:10,  5.42s/it]
 47%|████▋     | 2845/6100 [4:17:46<4:54:24,  5.43s/it]
                                                       
{'loss': 0.0852, 'learning_rate': 0.0001, 'epoch': 4.66}

 47%|████▋     | 2845/6100 [4:17:46<4:54:24,  5.43s/it]
 47%|████▋     | 2846/6100 [4:17:51<4:54:00,  5.42s/it]
 47%|████▋     | 2847/6100 [4:17:56<4:53:55,  5.42s/it]
 47%|████▋     | 2848/6100 [4:18:02<4:53:52,  5.42s/it]
 47%|████▋     | 2849/6100 [4:18:07<4:53:27,  5.42s/it]
 47%|████▋     | 2850/6100 [4:18:13<4:53:12,  5.41s/it]
                                                       
{'loss': 0.0799, 'learning_rate': 0.0001, 'epoch': 4.67}

 47%|████▋     | 2850/6100 [4:18:13<4:53:12,  5.41s/it]Saving model checkpoint to ./results/checkpoint-2850
Configuration saved in ./results/checkpoint-2850/config.json
Model weights saved in ./results/checkpoint-2850/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-2850/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-2850/special_tokens_map.json
[2023-02-22 00:10:28,903] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step2852 is begin to save!
[2023-02-22 00:10:28,906] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-2850/global_step2852/mp_rank_00_model_states.pt
[2023-02-22 00:10:28,906] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-2850/global_step2852/mp_rank_00_model_states.pt...
[2023-02-22 00:10:29,726] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-2850/global_step2852/mp_rank_00_model_states.pt.
[2023-02-22 00:10:29,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-2850/global_step2852/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 00:10:29,901] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-2850/global_step2852/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 00:10:29,902] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-2850/global_step2852/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 00:10:29,902] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2852 is ready now!
Deleting older checkpoint [results/checkpoint-2700] due to args.save_total_limit

 47%|████▋     | 2851/6100 [4:18:20<5:27:50,  6.05s/it]
 47%|████▋     | 2852/6100 [4:18:26<5:17:14,  5.86s/it]
 47%|████▋     | 2853/6100 [4:18:31<5:09:41,  5.72s/it]
 47%|████▋     | 2854/6100 [4:18:37<5:04:38,  5.63s/it]
 47%|████▋     | 2855/6100 [4:18:42<5:01:05,  5.57s/it]
                                                       
{'loss': 0.0848, 'learning_rate': 0.0001, 'epoch': 4.68}

 47%|████▋     | 2855/6100 [4:18:42<5:01:05,  5.57s/it]
 47%|████▋     | 2856/6100 [4:18:47<4:58:26,  5.52s/it]
 47%|████▋     | 2857/6100 [4:18:53<4:56:23,  5.48s/it]
 47%|████▋     | 2858/6100 [4:18:58<4:55:10,  5.46s/it]
 47%|████▋     | 2859/6100 [4:19:04<4:54:26,  5.45s/it]
 47%|████▋     | 2860/6100 [4:19:09<4:53:39,  5.44s/it]
                                                       
{'loss': 0.0793, 'learning_rate': 0.0001, 'epoch': 4.69}

 47%|████▋     | 2860/6100 [4:19:09<4:53:39,  5.44s/it]
 47%|████▋     | 2861/6100 [4:19:14<4:53:01,  5.43s/it]
 47%|████▋     | 2862/6100 [4:19:20<4:52:47,  5.43s/it]
 47%|████▋     | 2863/6100 [4:19:25<4:52:28,  5.42s/it]
 47%|████▋     | 2864/6100 [4:19:31<4:53:57,  5.45s/it]
 47%|████▋     | 2865/6100 [4:19:36<4:53:06,  5.44s/it]
                                                       
{'loss': 0.0759, 'learning_rate': 0.0001, 'epoch': 4.7}

 47%|████▋     | 2865/6100 [4:19:36<4:53:06,  5.44s/it]
 47%|████▋     | 2866/6100 [4:19:42<4:52:53,  5.43s/it]
 47%|████▋     | 2867/6100 [4:19:47<4:52:22,  5.43s/it]
 47%|████▋     | 2868/6100 [4:19:52<4:52:02,  5.42s/it]
 47%|████▋     | 2869/6100 [4:19:58<4:51:39,  5.42s/it]
 47%|████▋     | 2870/6100 [4:20:03<4:51:20,  5.41s/it]
                                                       
{'loss': 0.0797, 'learning_rate': 0.0001, 'epoch': 4.7}

 47%|████▋     | 2870/6100 [4:20:03<4:51:20,  5.41s/it]
 47%|████▋     | 2871/6100 [4:20:09<4:51:36,  5.42s/it]
 47%|████▋     | 2872/6100 [4:20:14<4:51:15,  5.41s/it]
 47%|████▋     | 2873/6100 [4:20:19<4:51:14,  5.42s/it]
 47%|████▋     | 2874/6100 [4:20:25<4:50:55,  5.41s/it]
 47%|████▋     | 2875/6100 [4:20:30<4:50:44,  5.41s/it]
                                                       
{'loss': 0.0853, 'learning_rate': 0.0001, 'epoch': 4.71}

 47%|████▋     | 2875/6100 [4:20:30<4:50:44,  5.41s/it]
 47%|████▋     | 2876/6100 [4:20:36<4:50:44,  5.41s/it]
 47%|████▋     | 2877/6100 [4:20:41<4:50:43,  5.41s/it]
 47%|████▋     | 2878/6100 [4:20:46<4:50:48,  5.42s/it]
 47%|████▋     | 2879/6100 [4:20:52<4:50:40,  5.41s/it]
 47%|████▋     | 2880/6100 [4:20:57<4:50:45,  5.42s/it]
                                                       
{'loss': 0.0777, 'learning_rate': 0.0001, 'epoch': 4.72}

 47%|████▋     | 2880/6100 [4:20:57<4:50:45,  5.42s/it]
 47%|████▋     | 2881/6100 [4:21:03<4:50:27,  5.41s/it]
 47%|████▋     | 2882/6100 [4:21:08<4:50:10,  5.41s/it]
 47%|████▋     | 2883/6100 [4:21:14<4:50:05,  5.41s/it]
 47%|████▋     | 2884/6100 [4:21:19<4:49:59,  5.41s/it]
 47%|████▋     | 2885/6100 [4:21:24<4:50:06,  5.41s/it]
                                                       
{'loss': 0.0781, 'learning_rate': 0.0001, 'epoch': 4.73}

 47%|████▋     | 2885/6100 [4:21:24<4:50:06,  5.41s/it]
 47%|████▋     | 2886/6100 [4:21:30<4:49:49,  5.41s/it]
 47%|████▋     | 2887/6100 [4:21:35<4:49:43,  5.41s/it]
 47%|████▋     | 2888/6100 [4:21:41<4:49:28,  5.41s/it]
 47%|████▋     | 2889/6100 [4:21:46<4:49:27,  5.41s/it]
 47%|████▋     | 2890/6100 [4:21:51<4:49:29,  5.41s/it]
                                                       
{'loss': 0.0887, 'learning_rate': 0.0001, 'epoch': 4.74}

 47%|████▋     | 2890/6100 [4:21:51<4:49:29,  5.41s/it]
 47%|████▋     | 2891/6100 [4:21:57<4:49:45,  5.42s/it]
 47%|████▋     | 2892/6100 [4:22:02<4:49:37,  5.42s/it]
 47%|████▋     | 2893/6100 [4:22:08<4:49:22,  5.41s/it]
 47%|████▋     | 2894/6100 [4:22:13<4:49:08,  5.41s/it]
 47%|████▋     | 2895/6100 [4:22:19<4:49:19,  5.42s/it]
                                                       
{'loss': 0.0796, 'learning_rate': 0.0001, 'epoch': 4.75}

 47%|████▋     | 2895/6100 [4:22:19<4:49:19,  5.42s/it]
 47%|████▋     | 2896/6100 [4:22:24<4:49:06,  5.41s/it]
 47%|████▋     | 2897/6100 [4:22:29<4:49:02,  5.41s/it]
 48%|████▊     | 2898/6100 [4:22:35<4:48:55,  5.41s/it]
 48%|████▊     | 2899/6100 [4:22:40<4:48:46,  5.41s/it]
 48%|████▊     | 2900/6100 [4:22:46<4:48:44,  5.41s/it]
                                                       
{'loss': 0.0839, 'learning_rate': 0.0001, 'epoch': 4.75}

 48%|████▊     | 2900/6100 [4:22:46<4:48:44,  5.41s/it]
 48%|████▊     | 2901/6100 [4:22:51<4:48:27,  5.41s/it]
 48%|████▊     | 2902/6100 [4:22:56<4:48:25,  5.41s/it]
 48%|████▊     | 2903/6100 [4:23:02<4:48:30,  5.41s/it]
 48%|████▊     | 2904/6100 [4:23:07<4:48:36,  5.42s/it]
 48%|████▊     | 2905/6100 [4:23:13<4:48:27,  5.42s/it]
                                                       
{'loss': 0.0798, 'learning_rate': 0.0001, 'epoch': 4.76}

 48%|████▊     | 2905/6100 [4:23:13<4:48:27,  5.42s/it]
 48%|████▊     | 2906/6100 [4:23:18<4:48:36,  5.42s/it]
 48%|████▊     | 2907/6100 [4:23:23<4:48:12,  5.42s/it]
 48%|████▊     | 2908/6100 [4:23:29<4:48:00,  5.41s/it]
 48%|████▊     | 2909/6100 [4:23:34<4:47:58,  5.41s/it]
 48%|████▊     | 2910/6100 [4:23:40<4:48:48,  5.43s/it]
                                                       
{'loss': 0.0832, 'learning_rate': 0.0001, 'epoch': 4.77}

 48%|████▊     | 2910/6100 [4:23:40<4:48:48,  5.43s/it]
 48%|████▊     | 2911/6100 [4:23:45<4:48:30,  5.43s/it]
 48%|████▊     | 2912/6100 [4:23:51<4:48:23,  5.43s/it]
 48%|████▊     | 2913/6100 [4:23:56<4:48:43,  5.44s/it]
 48%|████▊     | 2914/6100 [4:24:02<4:48:20,  5.43s/it]
 48%|████▊     | 2915/6100 [4:24:07<4:48:02,  5.43s/it]
                                                       
{'loss': 0.0837, 'learning_rate': 0.0001, 'epoch': 4.78}

 48%|████▊     | 2915/6100 [4:24:07<4:48:02,  5.43s/it]
 48%|████▊     | 2916/6100 [4:24:12<4:47:39,  5.42s/it]
 48%|████▊     | 2917/6100 [4:24:18<4:47:10,  5.41s/it]
 48%|████▊     | 2918/6100 [4:24:23<4:47:09,  5.41s/it]
 48%|████▊     | 2919/6100 [4:24:29<4:47:02,  5.41s/it]
 48%|████▊     | 2920/6100 [4:24:34<4:47:03,  5.42s/it]
                                                       
{'loss': 0.0813, 'learning_rate': 0.0001, 'epoch': 4.79}

 48%|████▊     | 2920/6100 [4:24:34<4:47:03,  5.42s/it]
 48%|████▊     | 2921/6100 [4:24:39<4:46:55,  5.42s/it]
 48%|████▊     | 2922/6100 [4:24:45<4:47:00,  5.42s/it]
 48%|████▊     | 2923/6100 [4:24:50<4:46:46,  5.42s/it]
 48%|████▊     | 2924/6100 [4:24:56<4:46:27,  5.41s/it]
 48%|████▊     | 2925/6100 [4:25:01<4:46:31,  5.41s/it]
                                                       
{'loss': 0.0823, 'learning_rate': 0.0001, 'epoch': 4.79}

 48%|████▊     | 2925/6100 [4:25:01<4:46:31,  5.41s/it]
 48%|████▊     | 2926/6100 [4:25:06<4:46:26,  5.41s/it]
 48%|████▊     | 2927/6100 [4:25:12<4:46:17,  5.41s/it]
 48%|████▊     | 2928/6100 [4:25:17<4:46:22,  5.42s/it]
 48%|████▊     | 2929/6100 [4:25:23<4:46:25,  5.42s/it]
 48%|████▊     | 2930/6100 [4:25:28<4:46:19,  5.42s/it]
                                                       
{'loss': 0.0836, 'learning_rate': 0.0001, 'epoch': 4.8}

 48%|████▊     | 2930/6100 [4:25:28<4:46:19,  5.42s/it]
 48%|████▊     | 2931/6100 [4:25:34<4:46:19,  5.42s/it]
 48%|████▊     | 2932/6100 [4:25:39<4:46:23,  5.42s/it]
 48%|████▊     | 2933/6100 [4:25:44<4:46:19,  5.42s/it]
 48%|████▊     | 2934/6100 [4:25:50<4:45:54,  5.42s/it]
 48%|████▊     | 2935/6100 [4:25:55<4:46:14,  5.43s/it]
                                                       
{'loss': 0.0886, 'learning_rate': 0.0001, 'epoch': 4.81}

 48%|████▊     | 2935/6100 [4:25:55<4:46:14,  5.43s/it]
 48%|████▊     | 2936/6100 [4:26:01<4:45:56,  5.42s/it]
 48%|████▊     | 2937/6100 [4:26:06<4:45:42,  5.42s/it]
 48%|████▊     | 2938/6100 [4:26:12<4:45:52,  5.42s/it]
 48%|████▊     | 2939/6100 [4:26:17<4:45:49,  5.43s/it]
 48%|████▊     | 2940/6100 [4:26:22<4:45:32,  5.42s/it]
                                                       
{'loss': 0.0853, 'learning_rate': 0.0001, 'epoch': 4.82}

 48%|████▊     | 2940/6100 [4:26:22<4:45:32,  5.42s/it]
 48%|████▊     | 2941/6100 [4:26:28<4:45:30,  5.42s/it]
 48%|████▊     | 2942/6100 [4:26:33<4:45:29,  5.42s/it]
 48%|████▊     | 2943/6100 [4:26:39<4:45:00,  5.42s/it]
 48%|████▊     | 2944/6100 [4:26:44<4:45:02,  5.42s/it]
 48%|████▊     | 2945/6100 [4:26:49<4:44:46,  5.42s/it]
                                                       
{'loss': 0.0756, 'learning_rate': 0.0001, 'epoch': 4.83}

 48%|████▊     | 2945/6100 [4:26:49<4:44:46,  5.42s/it]
 48%|████▊     | 2946/6100 [4:26:55<4:44:40,  5.42s/it]
 48%|████▊     | 2947/6100 [4:27:00<4:44:23,  5.41s/it]
 48%|████▊     | 2948/6100 [4:27:06<4:44:21,  5.41s/it]
 48%|████▊     | 2949/6100 [4:27:11<4:44:33,  5.42s/it]
 48%|████▊     | 2950/6100 [4:27:17<4:44:24,  5.42s/it]
                                                       
{'loss': 0.075, 'learning_rate': 0.0001, 'epoch': 4.84}

 48%|████▊     | 2950/6100 [4:27:17<4:44:24,  5.42s/it]
 48%|████▊     | 2951/6100 [4:27:22<4:44:17,  5.42s/it]
 48%|████▊     | 2952/6100 [4:27:27<4:44:22,  5.42s/it]
 48%|████▊     | 2953/6100 [4:27:33<4:44:22,  5.42s/it]
 48%|████▊     | 2954/6100 [4:27:38<4:44:10,  5.42s/it]
 48%|████▊     | 2955/6100 [4:27:44<4:44:07,  5.42s/it]
                                                       
{'loss': 0.0757, 'learning_rate': 0.0001, 'epoch': 4.84}

 48%|████▊     | 2955/6100 [4:27:44<4:44:07,  5.42s/it]
 48%|████▊     | 2956/6100 [4:27:49<4:43:54,  5.42s/it]
 48%|████▊     | 2957/6100 [4:27:54<4:43:39,  5.42s/it]
 48%|████▊     | 2958/6100 [4:28:00<4:43:31,  5.41s/it]
 49%|████▊     | 2959/6100 [4:28:05<4:43:21,  5.41s/it]
 49%|████▊     | 2960/6100 [4:28:11<4:43:13,  5.41s/it]
                                                       
{'loss': 0.0805, 'learning_rate': 0.0001, 'epoch': 4.85}

 49%|████▊     | 2960/6100 [4:28:11<4:43:13,  5.41s/it]
 49%|████▊     | 2961/6100 [4:28:16<4:43:12,  5.41s/it]
 49%|████▊     | 2962/6100 [4:28:22<4:43:01,  5.41s/it]
 49%|████▊     | 2963/6100 [4:28:27<4:42:58,  5.41s/it]
 49%|████▊     | 2964/6100 [4:28:32<4:43:08,  5.42s/it]
 49%|████▊     | 2965/6100 [4:28:38<4:43:04,  5.42s/it]
                                                       
{'loss': 0.0823, 'learning_rate': 0.0001, 'epoch': 4.86}

 49%|████▊     | 2965/6100 [4:28:38<4:43:04,  5.42s/it]
 49%|████▊     | 2966/6100 [4:28:43<4:42:58,  5.42s/it]
 49%|████▊     | 2967/6100 [4:28:49<4:43:09,  5.42s/it]
 49%|████▊     | 2968/6100 [4:28:54<4:42:55,  5.42s/it]
 49%|████▊     | 2969/6100 [4:28:59<4:42:38,  5.42s/it]
 49%|████▊     | 2970/6100 [4:29:05<4:42:40,  5.42s/it]
                                                       
{'loss': 0.0826, 'learning_rate': 0.0001, 'epoch': 4.87}

 49%|████▊     | 2970/6100 [4:29:05<4:42:40,  5.42s/it]
 49%|████▊     | 2971/6100 [4:29:10<4:42:40,  5.42s/it]
 49%|████▊     | 2972/6100 [4:29:16<4:42:31,  5.42s/it]
 49%|████▊     | 2973/6100 [4:29:22<4:50:43,  5.58s/it]
 49%|████▉     | 2974/6100 [4:29:27<4:48:07,  5.53s/it]
 49%|████▉     | 2975/6100 [4:29:33<4:46:01,  5.49s/it]
                                                       
{'loss': 0.0796, 'learning_rate': 0.0001, 'epoch': 4.88}

 49%|████▉     | 2975/6100 [4:29:33<4:46:01,  5.49s/it]
 49%|████▉     | 2976/6100 [4:29:38<4:44:57,  5.47s/it]
 49%|████▉     | 2977/6100 [4:29:43<4:43:46,  5.45s/it]
 49%|████▉     | 2978/6100 [4:29:49<4:43:05,  5.44s/it]
 49%|████▉     | 2979/6100 [4:29:54<4:42:35,  5.43s/it]
 49%|████▉     | 2980/6100 [4:30:00<4:42:27,  5.43s/it]
                                                       
{'loss': 0.0825, 'learning_rate': 0.0001, 'epoch': 4.88}

 49%|████▉     | 2980/6100 [4:30:00<4:42:27,  5.43s/it]
 49%|████▉     | 2981/6100 [4:30:05<4:41:58,  5.42s/it]
 49%|████▉     | 2982/6100 [4:30:10<4:41:38,  5.42s/it]
 49%|████▉     | 2983/6100 [4:30:16<4:41:27,  5.42s/it]
 49%|████▉     | 2984/6100 [4:30:21<4:41:22,  5.42s/it]
 49%|████▉     | 2985/6100 [4:30:27<4:41:01,  5.41s/it]
                                                       
{'loss': 0.08, 'learning_rate': 0.0001, 'epoch': 4.89}

 49%|████▉     | 2985/6100 [4:30:27<4:41:01,  5.41s/it]
 49%|████▉     | 2986/6100 [4:30:32<4:40:49,  5.41s/it]
 49%|████▉     | 2987/6100 [4:30:37<4:40:36,  5.41s/it]
 49%|████▉     | 2988/6100 [4:30:43<4:40:30,  5.41s/it]
 49%|████▉     | 2989/6100 [4:30:49<4:51:56,  5.63s/it]
 49%|████▉     | 2990/6100 [4:30:54<4:48:27,  5.57s/it]
                                                       
{'loss': 0.0797, 'learning_rate': 0.0001, 'epoch': 4.9}

 49%|████▉     | 2990/6100 [4:30:54<4:48:27,  5.57s/it]
 49%|████▉     | 2991/6100 [4:31:00<4:46:04,  5.52s/it]
 49%|████▉     | 2992/6100 [4:31:05<4:44:15,  5.49s/it]
 49%|████▉     | 2993/6100 [4:31:11<4:42:55,  5.46s/it]
 49%|████▉     | 2994/6100 [4:31:16<4:42:06,  5.45s/it]
 49%|████▉     | 2995/6100 [4:31:21<4:41:23,  5.44s/it]
                                                       
{'loss': 0.0764, 'learning_rate': 0.0001, 'epoch': 4.91}

 49%|████▉     | 2995/6100 [4:31:21<4:41:23,  5.44s/it]
 49%|████▉     | 2996/6100 [4:31:27<4:40:56,  5.43s/it]
 49%|████▉     | 2997/6100 [4:31:32<4:40:45,  5.43s/it]
 49%|████▉     | 2998/6100 [4:31:38<4:40:24,  5.42s/it]
 49%|████▉     | 2999/6100 [4:31:43<4:40:16,  5.42s/it]
 49%|████▉     | 3000/6100 [4:31:49<4:39:54,  5.42s/it]
                                                       
{'loss': 0.0818, 'learning_rate': 0.0001, 'epoch': 4.92}

 49%|████▉     | 3000/6100 [4:31:49<4:39:54,  5.42s/it]Saving model checkpoint to ./results/checkpoint-3000
Configuration saved in ./results/checkpoint-3000/config.json
Model weights saved in ./results/checkpoint-3000/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-3000/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-3000/special_tokens_map.json
[2023-02-22 00:24:04,725] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step3002 is begin to save!
[2023-02-22 00:24:04,728] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-3000/global_step3002/mp_rank_00_model_states.pt
[2023-02-22 00:24:04,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-3000/global_step3002/mp_rank_00_model_states.pt...
[2023-02-22 00:24:05,542] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-3000/global_step3002/mp_rank_00_model_states.pt.
[2023-02-22 00:24:05,544] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-3000/global_step3002/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 00:24:05,717] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-3000/global_step3002/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 00:24:05,717] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-3000/global_step3002/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 00:24:05,718] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3002 is ready now!
Deleting older checkpoint [results/checkpoint-2850] due to args.save_total_limit

 49%|████▉     | 3001/6100 [4:31:56<5:12:47,  6.06s/it]
 49%|████▉     | 3002/6100 [4:32:02<5:02:35,  5.86s/it]
 49%|████▉     | 3003/6100 [4:32:07<4:55:50,  5.73s/it]
 49%|████▉     | 3004/6100 [4:32:12<4:50:37,  5.63s/it]
 49%|████▉     | 3005/6100 [4:32:18<4:48:36,  5.60s/it]
                                                       
{'loss': 0.0775, 'learning_rate': 0.0001, 'epoch': 4.93}

 49%|████▉     | 3005/6100 [4:32:18<4:48:36,  5.60s/it]
 49%|████▉     | 3006/6100 [4:32:23<4:45:35,  5.54s/it]
 49%|████▉     | 3007/6100 [4:32:29<4:43:37,  5.50s/it]
 49%|████▉     | 3008/6100 [4:32:34<4:42:08,  5.47s/it]
 49%|████▉     | 3009/6100 [4:32:39<4:41:09,  5.46s/it]
 49%|████▉     | 3010/6100 [4:32:45<4:40:03,  5.44s/it]
                                                       
{'loss': 0.0828, 'learning_rate': 0.0001, 'epoch': 4.93}

 49%|████▉     | 3010/6100 [4:32:45<4:40:03,  5.44s/it]
 49%|████▉     | 3011/6100 [4:32:50<4:39:28,  5.43s/it]
 49%|████▉     | 3012/6100 [4:32:56<4:39:12,  5.43s/it]
 49%|████▉     | 3013/6100 [4:33:01<4:39:17,  5.43s/it]
 49%|████▉     | 3014/6100 [4:33:07<4:39:09,  5.43s/it]
 49%|████▉     | 3015/6100 [4:33:12<4:38:44,  5.42s/it]
                                                       
{'loss': 0.0767, 'learning_rate': 0.0001, 'epoch': 4.94}

 49%|████▉     | 3015/6100 [4:33:12<4:38:44,  5.42s/it]
 49%|████▉     | 3016/6100 [4:33:17<4:38:26,  5.42s/it]
 49%|████▉     | 3017/6100 [4:33:23<4:38:05,  5.41s/it]
 49%|████▉     | 3018/6100 [4:33:28<4:38:17,  5.42s/it]
 49%|████▉     | 3019/6100 [4:33:34<4:37:56,  5.41s/it]
 50%|████▉     | 3020/6100 [4:33:39<4:37:49,  5.41s/it]
                                                       
{'loss': 0.0774, 'learning_rate': 0.0001, 'epoch': 4.95}

 50%|████▉     | 3020/6100 [4:33:39<4:37:49,  5.41s/it]
 50%|████▉     | 3021/6100 [4:33:44<4:37:52,  5.41s/it]
 50%|████▉     | 3022/6100 [4:33:50<4:37:41,  5.41s/it]
 50%|████▉     | 3023/6100 [4:33:55<4:37:27,  5.41s/it]
 50%|████▉     | 3024/6100 [4:34:01<4:37:28,  5.41s/it]
 50%|████▉     | 3025/6100 [4:34:06<4:37:19,  5.41s/it]
                                                       
{'loss': 0.0772, 'learning_rate': 0.0001, 'epoch': 4.96}

 50%|████▉     | 3025/6100 [4:34:06<4:37:19,  5.41s/it]
 50%|████▉     | 3026/6100 [4:34:12<4:37:47,  5.42s/it]
 50%|████▉     | 3027/6100 [4:34:17<4:40:21,  5.47s/it]
 50%|████▉     | 3028/6100 [4:34:23<4:39:09,  5.45s/it]
 50%|████▉     | 3029/6100 [4:34:28<4:38:27,  5.44s/it]
 50%|████▉     | 3030/6100 [4:34:33<4:38:00,  5.43s/it]
                                                       
{'loss': 0.0854, 'learning_rate': 0.0001, 'epoch': 4.97}

 50%|████▉     | 3030/6100 [4:34:33<4:38:00,  5.43s/it]
 50%|████▉     | 3031/6100 [4:34:39<4:37:34,  5.43s/it]
 50%|████▉     | 3032/6100 [4:34:44<4:37:20,  5.42s/it]
 50%|████▉     | 3033/6100 [4:34:50<4:37:06,  5.42s/it]
 50%|████▉     | 3034/6100 [4:34:55<4:36:57,  5.42s/it]
 50%|████▉     | 3035/6100 [4:35:00<4:36:38,  5.42s/it]
                                                       
{'loss': 0.0747, 'learning_rate': 0.0001, 'epoch': 4.97}

 50%|████▉     | 3035/6100 [4:35:00<4:36:38,  5.42s/it]
 50%|████▉     | 3036/6100 [4:35:06<4:36:38,  5.42s/it]
 50%|████▉     | 3037/6100 [4:35:11<4:36:26,  5.41s/it]
 50%|████▉     | 3038/6100 [4:35:17<4:36:22,  5.42s/it]
 50%|████▉     | 3039/6100 [4:35:22<4:36:07,  5.41s/it]
 50%|████▉     | 3040/6100 [4:35:28<4:36:17,  5.42s/it]
                                                       
{'loss': 0.0789, 'learning_rate': 0.0001, 'epoch': 4.98}

 50%|████▉     | 3040/6100 [4:35:28<4:36:17,  5.42s/it]
 50%|████▉     | 3041/6100 [4:35:33<4:36:03,  5.41s/it]
 50%|████▉     | 3042/6100 [4:35:38<4:35:57,  5.41s/it]
 50%|████▉     | 3043/6100 [4:35:44<4:35:53,  5.42s/it]
 50%|████▉     | 3044/6100 [4:35:49<4:35:44,  5.41s/it]
 50%|████▉     | 3045/6100 [4:35:55<4:35:41,  5.41s/it]
                                                       
{'loss': 0.0798, 'learning_rate': 0.0001, 'epoch': 4.99}

 50%|████▉     | 3045/6100 [4:35:55<4:35:41,  5.41s/it]
 50%|████▉     | 3046/6100 [4:36:00<4:35:44,  5.42s/it]
 50%|████▉     | 3047/6100 [4:36:05<4:35:29,  5.41s/it]
 50%|████▉     | 3048/6100 [4:36:11<4:35:13,  5.41s/it]
 50%|████▉     | 3049/6100 [4:36:16<4:35:00,  5.41s/it]
 50%|█████     | 3050/6100 [4:36:22<4:34:56,  5.41s/it]
                                                       
{'loss': 0.0803, 'learning_rate': 0.0001, 'epoch': 5.0}

 50%|█████     | 3050/6100 [4:36:22<4:34:56,  5.41s/it]
 50%|█████     | 3051/6100 [4:36:30<5:14:32,  6.19s/it]
 50%|█████     | 3052/6100 [4:36:35<5:02:27,  5.95s/it]
 50%|█████     | 3053/6100 [4:36:40<4:54:10,  5.79s/it]
 50%|█████     | 3054/6100 [4:36:46<4:48:07,  5.68s/it]
 50%|█████     | 3055/6100 [4:36:51<4:44:00,  5.60s/it]
                                                       
{'loss': 0.0815, 'learning_rate': 0.0001, 'epoch': 5.01}

 50%|█████     | 3055/6100 [4:36:51<4:44:00,  5.60s/it]
 50%|█████     | 3056/6100 [4:36:57<4:41:03,  5.54s/it]
 50%|█████     | 3057/6100 [4:37:02<4:39:00,  5.50s/it]
 50%|█████     | 3058/6100 [4:37:08<4:37:35,  5.48s/it]
 50%|█████     | 3059/6100 [4:37:13<4:36:36,  5.46s/it]
 50%|█████     | 3060/6100 [4:37:18<4:37:57,  5.49s/it]
                                                       
{'loss': 0.0794, 'learning_rate': 0.0001, 'epoch': 5.02}

 50%|█████     | 3060/6100 [4:37:18<4:37:57,  5.49s/it]
 50%|█████     | 3061/6100 [4:37:24<4:36:43,  5.46s/it]
 50%|█████     | 3062/6100 [4:37:29<4:35:45,  5.45s/it]
 50%|█████     | 3063/6100 [4:37:35<4:34:57,  5.43s/it]
 50%|█████     | 3064/6100 [4:37:40<4:34:32,  5.43s/it]
 50%|█████     | 3065/6100 [4:37:46<4:34:22,  5.42s/it]
                                                       
{'loss': 0.0727, 'learning_rate': 0.0001, 'epoch': 5.02}

 50%|█████     | 3065/6100 [4:37:46<4:34:22,  5.42s/it]
 50%|█████     | 3066/6100 [4:37:51<4:34:26,  5.43s/it]
 50%|█████     | 3067/6100 [4:37:56<4:34:20,  5.43s/it]
 50%|█████     | 3068/6100 [4:38:02<4:34:20,  5.43s/it]
 50%|█████     | 3069/6100 [4:38:07<4:33:56,  5.42s/it]
 50%|█████     | 3070/6100 [4:38:13<4:33:31,  5.42s/it]
                                                       
{'loss': 0.0787, 'learning_rate': 0.0001, 'epoch': 5.03}

 50%|█████     | 3070/6100 [4:38:13<4:33:31,  5.42s/it]
 50%|█████     | 3071/6100 [4:38:18<4:33:28,  5.42s/it]
 50%|█████     | 3072/6100 [4:38:23<4:33:16,  5.41s/it]
 50%|█████     | 3073/6100 [4:38:29<4:33:22,  5.42s/it]
 50%|█████     | 3074/6100 [4:38:34<4:33:24,  5.42s/it]
 50%|█████     | 3075/6100 [4:38:40<4:33:08,  5.42s/it]
                                                       
{'loss': 0.0781, 'learning_rate': 0.0001, 'epoch': 5.04}

 50%|█████     | 3075/6100 [4:38:40<4:33:08,  5.42s/it]
 50%|█████     | 3076/6100 [4:38:45<4:33:06,  5.42s/it]
 50%|█████     | 3077/6100 [4:38:51<4:33:05,  5.42s/it]
 50%|█████     | 3078/6100 [4:38:56<4:33:03,  5.42s/it]
 50%|█████     | 3079/6100 [4:39:01<4:32:42,  5.42s/it]
 50%|█████     | 3080/6100 [4:39:07<4:32:22,  5.41s/it]
                                                       
{'loss': 0.0711, 'learning_rate': 0.0001, 'epoch': 5.05}

 50%|█████     | 3080/6100 [4:39:07<4:32:22,  5.41s/it]
 51%|█████     | 3081/6100 [4:39:12<4:32:27,  5.41s/it]
 51%|█████     | 3082/6100 [4:39:18<4:32:21,  5.41s/it]
 51%|█████     | 3083/6100 [4:39:23<4:32:10,  5.41s/it]
 51%|█████     | 3084/6100 [4:39:28<4:32:13,  5.42s/it]
 51%|█████     | 3085/6100 [4:39:34<4:32:07,  5.42s/it]
                                                       
{'loss': 0.0709, 'learning_rate': 0.0001, 'epoch': 5.06}

 51%|█████     | 3085/6100 [4:39:34<4:32:07,  5.42s/it]
 51%|█████     | 3086/6100 [4:39:39<4:32:57,  5.43s/it]
 51%|█████     | 3087/6100 [4:39:45<4:32:26,  5.43s/it]
 51%|█████     | 3088/6100 [4:39:50<4:32:08,  5.42s/it]
 51%|█████     | 3089/6100 [4:39:56<4:31:44,  5.41s/it]
 51%|█████     | 3090/6100 [4:40:01<4:31:48,  5.42s/it]
                                                       
{'loss': 0.08, 'learning_rate': 0.0001, 'epoch': 5.07}

 51%|█████     | 3090/6100 [4:40:01<4:31:48,  5.42s/it]
 51%|█████     | 3091/6100 [4:40:06<4:31:43,  5.42s/it]
 51%|█████     | 3092/6100 [4:40:12<4:31:35,  5.42s/it]
 51%|█████     | 3093/6100 [4:40:17<4:31:30,  5.42s/it]
 51%|█████     | 3094/6100 [4:40:23<4:31:26,  5.42s/it]
 51%|█████     | 3095/6100 [4:40:28<4:31:11,  5.41s/it]
                                                       
{'loss': 0.0739, 'learning_rate': 0.0001, 'epoch': 5.07}

 51%|█████     | 3095/6100 [4:40:28<4:31:11,  5.41s/it]
 51%|█████     | 3096/6100 [4:40:33<4:31:05,  5.41s/it]
 51%|█████     | 3097/6100 [4:40:39<4:30:49,  5.41s/it]
 51%|█████     | 3098/6100 [4:40:44<4:30:39,  5.41s/it]
 51%|█████     | 3099/6100 [4:40:50<4:30:24,  5.41s/it]
 51%|█████     | 3100/6100 [4:40:55<4:30:30,  5.41s/it]
                                                       
{'loss': 0.074, 'learning_rate': 0.0001, 'epoch': 5.08}

 51%|█████     | 3100/6100 [4:40:55<4:30:30,  5.41s/it]
 51%|█████     | 3101/6100 [4:41:01<4:30:42,  5.42s/it]
 51%|█████     | 3102/6100 [4:41:06<4:30:44,  5.42s/it]
 51%|█████     | 3103/6100 [4:41:11<4:30:43,  5.42s/it]
 51%|█████     | 3104/6100 [4:41:17<4:30:24,  5.42s/it]
 51%|█████     | 3105/6100 [4:41:22<4:30:10,  5.41s/it]
                                                       
{'loss': 0.079, 'learning_rate': 0.0001, 'epoch': 5.09}

 51%|█████     | 3105/6100 [4:41:22<4:30:10,  5.41s/it]
 51%|█████     | 3106/6100 [4:41:28<4:30:08,  5.41s/it]
 51%|█████     | 3107/6100 [4:41:33<4:30:12,  5.42s/it]
 51%|█████     | 3108/6100 [4:41:38<4:30:09,  5.42s/it]
 51%|█████     | 3109/6100 [4:41:44<4:30:03,  5.42s/it]
 51%|█████     | 3110/6100 [4:41:49<4:32:31,  5.47s/it]
                                                       
{'loss': 0.0748, 'learning_rate': 0.0001, 'epoch': 5.1}

 51%|█████     | 3110/6100 [4:41:49<4:32:31,  5.47s/it]
 51%|█████     | 3111/6100 [4:41:55<4:31:27,  5.45s/it]
 51%|█████     | 3112/6100 [4:42:00<4:30:55,  5.44s/it]
 51%|█████     | 3113/6100 [4:42:06<4:30:30,  5.43s/it]
 51%|█████     | 3114/6100 [4:42:11<4:30:00,  5.43s/it]
 51%|█████     | 3115/6100 [4:42:17<4:29:44,  5.42s/it]
                                                       
{'loss': 0.0759, 'learning_rate': 0.0001, 'epoch': 5.11}

 51%|█████     | 3115/6100 [4:42:17<4:29:44,  5.42s/it]
 51%|█████     | 3116/6100 [4:42:22<4:29:27,  5.42s/it]
 51%|█████     | 3117/6100 [4:42:27<4:29:28,  5.42s/it]
 51%|█████     | 3118/6100 [4:42:33<4:29:21,  5.42s/it]
 51%|█████     | 3119/6100 [4:42:38<4:29:18,  5.42s/it]
 51%|█████     | 3120/6100 [4:42:44<4:29:40,  5.43s/it]
                                                       
{'loss': 0.0729, 'learning_rate': 0.0001, 'epoch': 5.11}

 51%|█████     | 3120/6100 [4:42:44<4:29:40,  5.43s/it]
 51%|█████     | 3121/6100 [4:42:49<4:29:28,  5.43s/it]
 51%|█████     | 3122/6100 [4:42:55<4:29:20,  5.43s/it]
 51%|█████     | 3123/6100 [4:43:00<4:29:35,  5.43s/it]
 51%|█████     | 3124/6100 [4:43:05<4:29:57,  5.44s/it]
 51%|█████     | 3125/6100 [4:43:11<4:29:39,  5.44s/it]
                                                       
{'loss': 0.0783, 'learning_rate': 0.0001, 'epoch': 5.12}

 51%|█████     | 3125/6100 [4:43:11<4:29:39,  5.44s/it]
 51%|█████     | 3126/6100 [4:43:16<4:29:51,  5.44s/it]
 51%|█████▏    | 3127/6100 [4:43:22<4:29:13,  5.43s/it]
 51%|█████▏    | 3128/6100 [4:43:27<4:28:52,  5.43s/it]
 51%|█████▏    | 3129/6100 [4:43:33<4:28:47,  5.43s/it]
 51%|█████▏    | 3130/6100 [4:43:38<4:28:44,  5.43s/it]
                                                       
{'loss': 0.075, 'learning_rate': 0.0001, 'epoch': 5.13}

 51%|█████▏    | 3130/6100 [4:43:38<4:28:44,  5.43s/it]
 51%|█████▏    | 3131/6100 [4:43:43<4:28:28,  5.43s/it]
 51%|█████▏    | 3132/6100 [4:43:49<4:28:14,  5.42s/it]
 51%|█████▏    | 3133/6100 [4:43:54<4:28:02,  5.42s/it]
 51%|█████▏    | 3134/6100 [4:44:00<4:27:57,  5.42s/it]
 51%|█████▏    | 3135/6100 [4:44:05<4:27:59,  5.42s/it]
                                                       
{'loss': 0.0794, 'learning_rate': 0.0001, 'epoch': 5.14}

 51%|█████▏    | 3135/6100 [4:44:05<4:27:59,  5.42s/it]
 51%|█████▏    | 3136/6100 [4:44:11<4:27:55,  5.42s/it]
 51%|█████▏    | 3137/6100 [4:44:16<4:27:43,  5.42s/it]
 51%|█████▏    | 3138/6100 [4:44:21<4:27:36,  5.42s/it]
 51%|█████▏    | 3139/6100 [4:44:27<4:27:21,  5.42s/it]
 51%|█████▏    | 3140/6100 [4:44:32<4:27:07,  5.41s/it]
                                                       
{'loss': 0.077, 'learning_rate': 0.0001, 'epoch': 5.15}

 51%|█████▏    | 3140/6100 [4:44:32<4:27:07,  5.41s/it]
 51%|█████▏    | 3141/6100 [4:44:38<4:27:07,  5.42s/it]
 52%|█████▏    | 3142/6100 [4:44:43<4:27:05,  5.42s/it]
 52%|█████▏    | 3143/6100 [4:44:48<4:27:07,  5.42s/it]
 52%|█████▏    | 3144/6100 [4:44:54<4:30:44,  5.50s/it]
 52%|█████▏    | 3145/6100 [4:45:00<4:29:34,  5.47s/it]
                                                       
{'loss': 0.0781, 'learning_rate': 0.0001, 'epoch': 5.16}

 52%|█████▏    | 3145/6100 [4:45:00<4:29:34,  5.47s/it]
 52%|█████▏    | 3146/6100 [4:45:05<4:28:43,  5.46s/it]
 52%|█████▏    | 3147/6100 [4:45:10<4:28:09,  5.45s/it]
 52%|█████▏    | 3148/6100 [4:45:16<4:27:31,  5.44s/it]
 52%|█████▏    | 3149/6100 [4:45:21<4:27:09,  5.43s/it]
 52%|█████▏    | 3150/6100 [4:45:27<4:26:41,  5.42s/it]
                                                       
{'loss': 0.0795, 'learning_rate': 0.0001, 'epoch': 5.16}

 52%|█████▏    | 3150/6100 [4:45:27<4:26:41,  5.42s/it]Saving model checkpoint to ./results/checkpoint-3150
Configuration saved in ./results/checkpoint-3150/config.json
Model weights saved in ./results/checkpoint-3150/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-3150/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-3150/special_tokens_map.json
[2023-02-22 00:37:42,780] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step3152 is begin to save!
[2023-02-22 00:37:42,783] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-3150/global_step3152/mp_rank_00_model_states.pt
[2023-02-22 00:37:42,783] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-3150/global_step3152/mp_rank_00_model_states.pt...
[2023-02-22 00:37:43,596] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-3150/global_step3152/mp_rank_00_model_states.pt.
[2023-02-22 00:37:43,597] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-3150/global_step3152/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 00:37:43,770] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-3150/global_step3152/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 00:37:43,770] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-3150/global_step3152/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 00:37:43,770] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3152 is ready now!
Deleting older checkpoint [results/checkpoint-3000] due to args.save_total_limit

 52%|█████▏    | 3151/6100 [4:45:34<4:57:34,  6.05s/it]
 52%|█████▏    | 3152/6100 [4:45:40<4:47:57,  5.86s/it]
 52%|█████▏    | 3153/6100 [4:45:45<4:41:12,  5.73s/it]
 52%|█████▏    | 3154/6100 [4:45:50<4:36:08,  5.62s/it]
 52%|█████▏    | 3155/6100 [4:45:56<4:32:54,  5.56s/it]
                                                       
{'loss': 0.0747, 'learning_rate': 0.0001, 'epoch': 5.17}

 52%|█████▏    | 3155/6100 [4:45:56<4:32:54,  5.56s/it]
 52%|█████▏    | 3156/6100 [4:46:01<4:30:45,  5.52s/it]
 52%|█████▏    | 3157/6100 [4:46:07<4:29:11,  5.49s/it]
 52%|█████▏    | 3158/6100 [4:46:12<4:27:58,  5.47s/it]
 52%|█████▏    | 3159/6100 [4:46:17<4:27:16,  5.45s/it]
 52%|█████▏    | 3160/6100 [4:46:23<4:26:28,  5.44s/it]
                                                       
{'loss': 0.0774, 'learning_rate': 0.0001, 'epoch': 5.18}

 52%|█████▏    | 3160/6100 [4:46:23<4:26:28,  5.44s/it]
 52%|█████▏    | 3161/6100 [4:46:28<4:26:05,  5.43s/it]
 52%|█████▏    | 3162/6100 [4:46:34<4:25:45,  5.43s/it]
 52%|█████▏    | 3163/6100 [4:46:39<4:25:21,  5.42s/it]
 52%|█████▏    | 3164/6100 [4:46:44<4:25:06,  5.42s/it]
 52%|█████▏    | 3165/6100 [4:46:50<4:24:51,  5.41s/it]
                                                       
{'loss': 0.0718, 'learning_rate': 0.0001, 'epoch': 5.19}

 52%|█████▏    | 3165/6100 [4:46:50<4:24:51,  5.41s/it]
 52%|█████▏    | 3166/6100 [4:46:55<4:24:39,  5.41s/it]
 52%|█████▏    | 3167/6100 [4:47:01<4:24:24,  5.41s/it]
 52%|█████▏    | 3168/6100 [4:47:06<4:24:32,  5.41s/it]
 52%|█████▏    | 3169/6100 [4:47:12<4:24:27,  5.41s/it]
 52%|█████▏    | 3170/6100 [4:47:17<4:24:15,  5.41s/it]
                                                       
{'loss': 0.0731, 'learning_rate': 0.0001, 'epoch': 5.2}

 52%|█████▏    | 3170/6100 [4:47:17<4:24:15,  5.41s/it]
 52%|█████▏    | 3171/6100 [4:47:22<4:24:20,  5.42s/it]
 52%|█████▏    | 3172/6100 [4:47:28<4:24:12,  5.41s/it]
 52%|█████▏    | 3173/6100 [4:47:33<4:23:47,  5.41s/it]
 52%|█████▏    | 3174/6100 [4:47:39<4:23:49,  5.41s/it]
 52%|█████▏    | 3175/6100 [4:47:44<4:23:41,  5.41s/it]
                                                       
{'loss': 0.0786, 'learning_rate': 0.0001, 'epoch': 5.2}

 52%|█████▏    | 3175/6100 [4:47:44<4:23:41,  5.41s/it]
 52%|█████▏    | 3176/6100 [4:47:49<4:23:35,  5.41s/it]
 52%|█████▏    | 3177/6100 [4:47:55<4:23:41,  5.41s/it]
 52%|█████▏    | 3178/6100 [4:48:00<4:23:26,  5.41s/it]
 52%|█████▏    | 3179/6100 [4:48:06<4:23:29,  5.41s/it]
 52%|█████▏    | 3180/6100 [4:48:11<4:25:22,  5.45s/it]
                                                       
{'loss': 0.0824, 'learning_rate': 0.0001, 'epoch': 5.21}

 52%|█████▏    | 3180/6100 [4:48:11<4:25:22,  5.45s/it]
 52%|█████▏    | 3181/6100 [4:48:17<4:24:54,  5.45s/it]
 52%|█████▏    | 3182/6100 [4:48:22<4:24:20,  5.44s/it]
 52%|█████▏    | 3183/6100 [4:48:27<4:23:47,  5.43s/it]
 52%|█████▏    | 3184/6100 [4:48:33<4:23:25,  5.42s/it]
 52%|█████▏    | 3185/6100 [4:48:38<4:23:12,  5.42s/it]
                                                       
{'loss': 0.072, 'learning_rate': 0.0001, 'epoch': 5.22}

 52%|█████▏    | 3185/6100 [4:48:38<4:23:12,  5.42s/it]
 52%|█████▏    | 3186/6100 [4:48:44<4:23:22,  5.42s/it]
 52%|█████▏    | 3187/6100 [4:48:49<4:23:12,  5.42s/it]
 52%|█████▏    | 3188/6100 [4:48:55<4:23:00,  5.42s/it]
 52%|█████▏    | 3189/6100 [4:49:00<4:22:57,  5.42s/it]
 52%|█████▏    | 3190/6100 [4:49:05<4:22:51,  5.42s/it]
                                                       
{'loss': 0.0765, 'learning_rate': 0.0001, 'epoch': 5.23}

 52%|█████▏    | 3190/6100 [4:49:05<4:22:51,  5.42s/it]
 52%|█████▏    | 3191/6100 [4:49:11<4:22:51,  5.42s/it]
 52%|█████▏    | 3192/6100 [4:49:16<4:23:18,  5.43s/it]
 52%|█████▏    | 3193/6100 [4:49:22<4:23:00,  5.43s/it]
 52%|█████▏    | 3194/6100 [4:49:27<4:22:32,  5.42s/it]
 52%|█████▏    | 3195/6100 [4:49:32<4:22:10,  5.41s/it]
                                                       
{'loss': 0.0788, 'learning_rate': 0.0001, 'epoch': 5.24}

 52%|█████▏    | 3195/6100 [4:49:32<4:22:10,  5.41s/it]
 52%|█████▏    | 3196/6100 [4:49:38<4:22:17,  5.42s/it]
 52%|█████▏    | 3197/6100 [4:49:43<4:22:11,  5.42s/it]
 52%|█████▏    | 3198/6100 [4:49:49<4:22:03,  5.42s/it]
 52%|█████▏    | 3199/6100 [4:49:54<4:21:52,  5.42s/it]
 52%|█████▏    | 3200/6100 [4:50:00<4:21:45,  5.42s/it]
                                                       
{'loss': 0.0767, 'learning_rate': 0.0001, 'epoch': 5.25}

 52%|█████▏    | 3200/6100 [4:50:00<4:21:45,  5.42s/it]
 52%|█████▏    | 3201/6100 [4:50:05<4:21:52,  5.42s/it]
 52%|█████▏    | 3202/6100 [4:50:10<4:21:51,  5.42s/it]
 53%|█████▎    | 3203/6100 [4:50:16<4:21:28,  5.42s/it]
 53%|█████▎    | 3204/6100 [4:50:21<4:21:25,  5.42s/it]
 53%|█████▎    | 3205/6100 [4:50:27<4:21:19,  5.42s/it]
                                                       
{'loss': 0.0752, 'learning_rate': 0.0001, 'epoch': 5.25}

 53%|█████▎    | 3205/6100 [4:50:27<4:21:19,  5.42s/it]
 53%|█████▎    | 3206/6100 [4:50:32<4:21:10,  5.41s/it]
 53%|█████▎    | 3207/6100 [4:50:37<4:21:02,  5.41s/it]
 53%|█████▎    | 3208/6100 [4:50:43<4:21:02,  5.42s/it]
 53%|█████▎    | 3209/6100 [4:50:48<4:21:01,  5.42s/it]
 53%|█████▎    | 3210/6100 [4:50:54<4:20:45,  5.41s/it]
                                                       
{'loss': 0.0771, 'learning_rate': 0.0001, 'epoch': 5.26}

 53%|█████▎    | 3210/6100 [4:50:54<4:20:45,  5.41s/it]
 53%|█████▎    | 3211/6100 [4:50:59<4:20:42,  5.41s/it]
 53%|█████▎    | 3212/6100 [4:51:05<4:20:36,  5.41s/it]
 53%|█████▎    | 3213/6100 [4:51:10<4:20:28,  5.41s/it]
 53%|█████▎    | 3214/6100 [4:51:15<4:20:29,  5.42s/it]
 53%|█████▎    | 3215/6100 [4:51:21<4:20:25,  5.42s/it]
                                                       
{'loss': 0.0764, 'learning_rate': 0.0001, 'epoch': 5.27}

 53%|█████▎    | 3215/6100 [4:51:21<4:20:25,  5.42s/it]
 53%|█████▎    | 3216/6100 [4:51:26<4:20:28,  5.42s/it]
 53%|█████▎    | 3217/6100 [4:51:32<4:20:14,  5.42s/it]
 53%|█████▎    | 3218/6100 [4:51:37<4:20:03,  5.41s/it]
 53%|█████▎    | 3219/6100 [4:51:42<4:20:14,  5.42s/it]
 53%|█████▎    | 3220/6100 [4:51:48<4:19:59,  5.42s/it]
                                                       
{'loss': 0.0797, 'learning_rate': 0.0001, 'epoch': 5.28}

 53%|█████▎    | 3220/6100 [4:51:48<4:19:59,  5.42s/it]
 53%|█████▎    | 3221/6100 [4:51:53<4:19:48,  5.41s/it]
 53%|█████▎    | 3222/6100 [4:51:59<4:19:36,  5.41s/it]
 53%|█████▎    | 3223/6100 [4:52:04<4:19:33,  5.41s/it]
 53%|█████▎    | 3224/6100 [4:52:10<4:19:32,  5.41s/it]
 53%|█████▎    | 3225/6100 [4:52:15<4:19:16,  5.41s/it]
                                                       
{'loss': 0.0734, 'learning_rate': 0.0001, 'epoch': 5.29}

 53%|█████▎    | 3225/6100 [4:52:15<4:19:16,  5.41s/it]
 53%|█████▎    | 3226/6100 [4:52:20<4:19:21,  5.41s/it]
 53%|█████▎    | 3227/6100 [4:52:26<4:19:08,  5.41s/it]
 53%|█████▎    | 3228/6100 [4:52:31<4:19:05,  5.41s/it]
 53%|█████▎    | 3229/6100 [4:52:37<4:18:57,  5.41s/it]
 53%|█████▎    | 3230/6100 [4:52:42<4:18:47,  5.41s/it]
                                                       
{'loss': 0.0769, 'learning_rate': 0.0001, 'epoch': 5.29}

 53%|█████▎    | 3230/6100 [4:52:42<4:18:47,  5.41s/it]
 53%|█████▎    | 3231/6100 [4:52:47<4:18:46,  5.41s/it]
 53%|█████▎    | 3232/6100 [4:52:53<4:18:38,  5.41s/it]
 53%|█████▎    | 3233/6100 [4:52:58<4:18:59,  5.42s/it]
 53%|█████▎    | 3234/6100 [4:53:04<4:18:58,  5.42s/it]
 53%|█████▎    | 3235/6100 [4:53:09<4:18:43,  5.42s/it]
                                                       
{'loss': 0.0777, 'learning_rate': 0.0001, 'epoch': 5.3}

 53%|█████▎    | 3235/6100 [4:53:09<4:18:43,  5.42s/it]
 53%|█████▎    | 3236/6100 [4:53:15<4:18:46,  5.42s/it]
 53%|█████▎    | 3237/6100 [4:53:20<4:18:48,  5.42s/it]
 53%|█████▎    | 3238/6100 [4:53:25<4:18:25,  5.42s/it]
 53%|█████▎    | 3239/6100 [4:53:31<4:18:22,  5.42s/it]
 53%|█████▎    | 3240/6100 [4:53:36<4:18:09,  5.42s/it]
                                                       
{'loss': 0.0724, 'learning_rate': 0.0001, 'epoch': 5.31}

 53%|█████▎    | 3240/6100 [4:53:36<4:18:09,  5.42s/it]
 53%|█████▎    | 3241/6100 [4:53:42<4:18:09,  5.42s/it]
 53%|█████▎    | 3242/6100 [4:53:47<4:18:03,  5.42s/it]
 53%|█████▎    | 3243/6100 [4:53:52<4:17:49,  5.41s/it]
 53%|█████▎    | 3244/6100 [4:53:58<4:17:35,  5.41s/it]
 53%|█████▎    | 3245/6100 [4:54:03<4:17:22,  5.41s/it]
                                                       
{'loss': 0.0749, 'learning_rate': 0.0001, 'epoch': 5.32}

 53%|█████▎    | 3245/6100 [4:54:03<4:17:22,  5.41s/it]
 53%|█████▎    | 3246/6100 [4:54:09<4:17:27,  5.41s/it]
 53%|█████▎    | 3247/6100 [4:54:14<4:17:13,  5.41s/it]
 53%|█████▎    | 3248/6100 [4:54:19<4:17:04,  5.41s/it]
 53%|█████▎    | 3249/6100 [4:54:25<4:16:52,  5.41s/it]
 53%|█████▎    | 3250/6100 [4:54:30<4:16:57,  5.41s/it]
                                                       
{'loss': 0.0756, 'learning_rate': 0.0001, 'epoch': 5.33}

 53%|█████▎    | 3250/6100 [4:54:30<4:16:57,  5.41s/it]
 53%|█████▎    | 3251/6100 [4:54:36<4:17:02,  5.41s/it]
 53%|█████▎    | 3252/6100 [4:54:41<4:16:58,  5.41s/it]
 53%|█████▎    | 3253/6100 [4:54:47<4:16:45,  5.41s/it]
 53%|█████▎    | 3254/6100 [4:54:52<4:16:39,  5.41s/it]
 53%|█████▎    | 3255/6100 [4:54:57<4:16:34,  5.41s/it]
                                                       
{'loss': 0.0755, 'learning_rate': 0.0001, 'epoch': 5.34}

 53%|█████▎    | 3255/6100 [4:54:57<4:16:34,  5.41s/it]
 53%|█████▎    | 3256/6100 [4:55:03<4:16:42,  5.42s/it]
 53%|█████▎    | 3257/6100 [4:55:08<4:16:35,  5.42s/it]
 53%|█████▎    | 3258/6100 [4:55:14<4:16:38,  5.42s/it]
 53%|█████▎    | 3259/6100 [4:55:19<4:16:37,  5.42s/it]
 53%|█████▎    | 3260/6100 [4:55:24<4:16:24,  5.42s/it]
                                                       
{'loss': 0.0748, 'learning_rate': 0.0001, 'epoch': 5.34}

 53%|█████▎    | 3260/6100 [4:55:24<4:16:24,  5.42s/it]
 53%|█████▎    | 3261/6100 [4:55:30<4:16:12,  5.41s/it]
 53%|█████▎    | 3262/6100 [4:55:35<4:16:07,  5.41s/it]
 53%|█████▎    | 3263/6100 [4:55:41<4:15:57,  5.41s/it]
 54%|█████▎    | 3264/6100 [4:55:46<4:15:52,  5.41s/it]
 54%|█████▎    | 3265/6100 [4:55:52<4:15:45,  5.41s/it]
                                                       
{'loss': 0.0779, 'learning_rate': 0.0001, 'epoch': 5.35}

 54%|█████▎    | 3265/6100 [4:55:52<4:15:45,  5.41s/it]
 54%|█████▎    | 3266/6100 [4:55:57<4:15:45,  5.41s/it]
 54%|█████▎    | 3267/6100 [4:56:02<4:15:35,  5.41s/it]
 54%|█████▎    | 3268/6100 [4:56:08<4:15:26,  5.41s/it]
 54%|█████▎    | 3269/6100 [4:56:13<4:15:31,  5.42s/it]
 54%|█████▎    | 3270/6100 [4:56:19<4:15:18,  5.41s/it]
                                                       
{'loss': 0.0719, 'learning_rate': 0.0001, 'epoch': 5.36}

 54%|█████▎    | 3270/6100 [4:56:19<4:15:18,  5.41s/it]
 54%|█████▎    | 3271/6100 [4:56:24<4:15:05,  5.41s/it]
 54%|█████▎    | 3272/6100 [4:56:29<4:15:11,  5.41s/it]
 54%|█████▎    | 3273/6100 [4:56:35<4:15:04,  5.41s/it]
 54%|█████▎    | 3274/6100 [4:56:40<4:14:55,  5.41s/it]
 54%|█████▎    | 3275/6100 [4:56:46<4:14:44,  5.41s/it]
                                                       
{'loss': 0.0739, 'learning_rate': 0.0001, 'epoch': 5.37}

 54%|█████▎    | 3275/6100 [4:56:46<4:14:44,  5.41s/it]
 54%|█████▎    | 3276/6100 [4:56:51<4:14:46,  5.41s/it]
 54%|█████▎    | 3277/6100 [4:56:56<4:14:50,  5.42s/it]
 54%|█████▎    | 3278/6100 [4:57:02<4:14:56,  5.42s/it]
 54%|█████▍    | 3279/6100 [4:57:07<4:15:02,  5.42s/it]
 54%|█████▍    | 3280/6100 [4:57:13<4:15:00,  5.43s/it]
                                                       
{'loss': 0.0742, 'learning_rate': 0.0001, 'epoch': 5.38}

 54%|█████▍    | 3280/6100 [4:57:13<4:15:00,  5.43s/it]
 54%|█████▍    | 3281/6100 [4:57:18<4:14:56,  5.43s/it]
 54%|█████▍    | 3282/6100 [4:57:24<4:14:45,  5.42s/it]
 54%|█████▍    | 3283/6100 [4:57:29<4:14:22,  5.42s/it]
 54%|█████▍    | 3284/6100 [4:57:34<4:14:05,  5.41s/it]
 54%|█████▍    | 3285/6100 [4:57:40<4:13:57,  5.41s/it]
                                                       
{'loss': 0.0724, 'learning_rate': 0.0001, 'epoch': 5.38}

 54%|█████▍    | 3285/6100 [4:57:40<4:13:57,  5.41s/it]
 54%|█████▍    | 3286/6100 [4:57:45<4:13:56,  5.41s/it]
 54%|█████▍    | 3287/6100 [4:57:51<4:13:55,  5.42s/it]
 54%|█████▍    | 3288/6100 [4:57:56<4:13:39,  5.41s/it]
 54%|█████▍    | 3289/6100 [4:58:01<4:13:24,  5.41s/it]
 54%|█████▍    | 3290/6100 [4:58:07<4:13:36,  5.41s/it]
                                                       
{'loss': 0.0727, 'learning_rate': 0.0001, 'epoch': 5.39}

 54%|█████▍    | 3290/6100 [4:58:07<4:13:36,  5.41s/it]
 54%|█████▍    | 3291/6100 [4:58:12<4:13:22,  5.41s/it]
 54%|█████▍    | 3292/6100 [4:58:18<4:13:18,  5.41s/it]
 54%|█████▍    | 3293/6100 [4:58:23<4:13:20,  5.42s/it]
 54%|█████▍    | 3294/6100 [4:58:29<4:13:11,  5.41s/it]
 54%|█████▍    | 3295/6100 [4:58:34<4:13:12,  5.42s/it]
                                                       
{'loss': 0.0799, 'learning_rate': 0.0001, 'epoch': 5.4}

 54%|█████▍    | 3295/6100 [4:58:34<4:13:12,  5.42s/it]
 54%|█████▍    | 3296/6100 [4:58:39<4:12:56,  5.41s/it]
 54%|█████▍    | 3297/6100 [4:58:45<4:12:42,  5.41s/it]
 54%|█████▍    | 3298/6100 [4:58:50<4:12:36,  5.41s/it]
 54%|█████▍    | 3299/6100 [4:58:56<4:12:29,  5.41s/it]
 54%|█████▍    | 3300/6100 [4:59:01<4:12:37,  5.41s/it]
                                                       
{'loss': 0.0728, 'learning_rate': 0.0001, 'epoch': 5.41}

 54%|█████▍    | 3300/6100 [4:59:01<4:12:37,  5.41s/it]Saving model checkpoint to ./results/checkpoint-3300
Configuration saved in ./results/checkpoint-3300/config.json
Model weights saved in ./results/checkpoint-3300/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-3300/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-3300/special_tokens_map.json
[2023-02-22 00:51:17,212] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step3302 is begin to save!
[2023-02-22 00:51:17,215] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-3300/global_step3302/mp_rank_00_model_states.pt
[2023-02-22 00:51:17,215] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-3300/global_step3302/mp_rank_00_model_states.pt...
[2023-02-22 00:51:18,034] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-3300/global_step3302/mp_rank_00_model_states.pt.
[2023-02-22 00:51:18,036] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-3300/global_step3302/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 00:51:18,237] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-3300/global_step3302/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 00:51:18,237] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-3300/global_step3302/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 00:51:18,237] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3302 is ready now!
Deleting older checkpoint [results/checkpoint-3150] due to args.save_total_limit

 54%|█████▍    | 3301/6100 [4:59:09<4:42:48,  6.06s/it]
 54%|█████▍    | 3302/6100 [4:59:14<4:33:32,  5.87s/it]
 54%|█████▍    | 3303/6100 [4:59:19<4:27:03,  5.73s/it]
 54%|█████▍    | 3304/6100 [4:59:25<4:22:27,  5.63s/it]
 54%|█████▍    | 3305/6100 [4:59:30<4:19:22,  5.57s/it]
                                                       
{'loss': 0.0828, 'learning_rate': 0.0001, 'epoch': 5.42}

 54%|█████▍    | 3305/6100 [4:59:30<4:19:22,  5.57s/it]
 54%|█████▍    | 3306/6100 [4:59:36<4:17:22,  5.53s/it]
 54%|█████▍    | 3307/6100 [4:59:41<4:15:39,  5.49s/it]
 54%|█████▍    | 3308/6100 [4:59:47<4:14:25,  5.47s/it]
 54%|█████▍    | 3309/6100 [4:59:52<4:13:25,  5.45s/it]
 54%|█████▍    | 3310/6100 [4:59:57<4:12:42,  5.43s/it]
                                                       
{'loss': 0.0757, 'learning_rate': 0.0001, 'epoch': 5.43}

 54%|█████▍    | 3310/6100 [4:59:57<4:12:42,  5.43s/it]
 54%|█████▍    | 3311/6100 [5:00:03<4:12:26,  5.43s/it]
 54%|█████▍    | 3312/6100 [5:00:08<4:11:53,  5.42s/it]
 54%|█████▍    | 3313/6100 [5:00:14<4:11:42,  5.42s/it]
 54%|█████▍    | 3314/6100 [5:00:19<4:11:30,  5.42s/it]
 54%|█████▍    | 3315/6100 [5:00:24<4:11:18,  5.41s/it]
                                                       
{'loss': 0.0763, 'learning_rate': 0.0001, 'epoch': 5.43}

 54%|█████▍    | 3315/6100 [5:00:24<4:11:18,  5.41s/it]
 54%|█████▍    | 3316/6100 [5:00:30<4:11:28,  5.42s/it]
 54%|█████▍    | 3317/6100 [5:00:35<4:11:13,  5.42s/it]
 54%|█████▍    | 3318/6100 [5:00:41<4:11:02,  5.41s/it]
 54%|█████▍    | 3319/6100 [5:00:46<4:10:58,  5.41s/it]
 54%|█████▍    | 3320/6100 [5:00:51<4:10:49,  5.41s/it]
                                                       
{'loss': 0.0788, 'learning_rate': 0.0001, 'epoch': 5.44}

 54%|█████▍    | 3320/6100 [5:00:51<4:10:49,  5.41s/it]
 54%|█████▍    | 3321/6100 [5:00:57<4:10:38,  5.41s/it]
 54%|█████▍    | 3322/6100 [5:01:02<4:10:25,  5.41s/it]
 54%|█████▍    | 3323/6100 [5:01:08<4:10:20,  5.41s/it]
 54%|█████▍    | 3324/6100 [5:01:13<4:10:04,  5.41s/it]
 55%|█████▍    | 3325/6100 [5:01:18<4:09:58,  5.40s/it]
                                                       
{'loss': 0.0764, 'learning_rate': 0.0001, 'epoch': 5.45}

 55%|█████▍    | 3325/6100 [5:01:18<4:09:58,  5.40s/it]
 55%|█████▍    | 3326/6100 [5:01:24<4:09:59,  5.41s/it]
 55%|█████▍    | 3327/6100 [5:01:29<4:09:53,  5.41s/it]
 55%|█████▍    | 3328/6100 [5:01:35<4:09:38,  5.40s/it]
 55%|█████▍    | 3329/6100 [5:01:40<4:09:50,  5.41s/it]
 55%|█████▍    | 3330/6100 [5:01:46<4:09:48,  5.41s/it]
                                                       
{'loss': 0.0739, 'learning_rate': 0.0001, 'epoch': 5.46}

 55%|█████▍    | 3330/6100 [5:01:46<4:09:48,  5.41s/it]
 55%|█████▍    | 3331/6100 [5:01:51<4:09:51,  5.41s/it]
 55%|█████▍    | 3332/6100 [5:01:56<4:09:40,  5.41s/it]
 55%|█████▍    | 3333/6100 [5:02:02<4:09:37,  5.41s/it]
 55%|█████▍    | 3334/6100 [5:02:07<4:09:31,  5.41s/it]
 55%|█████▍    | 3335/6100 [5:02:13<4:09:23,  5.41s/it]
                                                       
{'loss': 0.0768, 'learning_rate': 0.0001, 'epoch': 5.47}

 55%|█████▍    | 3335/6100 [5:02:13<4:09:23,  5.41s/it]
 55%|█████▍    | 3336/6100 [5:02:18<4:09:40,  5.42s/it]
 55%|█████▍    | 3337/6100 [5:02:23<4:09:15,  5.41s/it]
 55%|█████▍    | 3338/6100 [5:02:29<4:09:24,  5.42s/it]
 55%|█████▍    | 3339/6100 [5:02:34<4:09:32,  5.42s/it]
 55%|█████▍    | 3340/6100 [5:02:40<4:09:23,  5.42s/it]
                                                       
{'loss': 0.0801, 'learning_rate': 0.0001, 'epoch': 5.48}

 55%|█████▍    | 3340/6100 [5:02:40<4:09:23,  5.42s/it]
 55%|█████▍    | 3341/6100 [5:02:45<4:09:03,  5.42s/it]
 55%|█████▍    | 3342/6100 [5:02:51<4:08:57,  5.42s/it]
 55%|█████▍    | 3343/6100 [5:02:56<4:08:49,  5.42s/it]
 55%|█████▍    | 3344/6100 [5:03:01<4:08:54,  5.42s/it]
 55%|█████▍    | 3345/6100 [5:03:07<4:09:00,  5.42s/it]
                                                       
{'loss': 0.079, 'learning_rate': 0.0001, 'epoch': 5.48}

 55%|█████▍    | 3345/6100 [5:03:07<4:09:00,  5.42s/it]
 55%|█████▍    | 3346/6100 [5:03:12<4:08:26,  5.41s/it]
 55%|█████▍    | 3347/6100 [5:03:18<4:08:07,  5.41s/it]
 55%|█████▍    | 3348/6100 [5:03:23<4:08:21,  5.41s/it]
 55%|█████▍    | 3349/6100 [5:03:28<4:08:19,  5.42s/it]
 55%|█████▍    | 3350/6100 [5:03:34<4:08:16,  5.42s/it]
                                                       
{'loss': 0.0749, 'learning_rate': 0.0001, 'epoch': 5.49}

 55%|█████▍    | 3350/6100 [5:03:34<4:08:16,  5.42s/it]
 55%|█████▍    | 3351/6100 [5:03:39<4:07:58,  5.41s/it]
 55%|█████▍    | 3352/6100 [5:03:45<4:07:56,  5.41s/it]
 55%|█████▍    | 3353/6100 [5:03:50<4:08:06,  5.42s/it]
 55%|█████▍    | 3354/6100 [5:03:56<4:08:07,  5.42s/it]
 55%|█████▌    | 3355/6100 [5:04:01<4:07:57,  5.42s/it]
                                                       
{'loss': 0.0759, 'learning_rate': 0.0001, 'epoch': 5.5}

 55%|█████▌    | 3355/6100 [5:04:01<4:07:57,  5.42s/it]
 55%|█████▌    | 3356/6100 [5:04:06<4:07:53,  5.42s/it]
 55%|█████▌    | 3357/6100 [5:04:12<4:07:39,  5.42s/it]
 55%|█████▌    | 3358/6100 [5:04:17<4:07:22,  5.41s/it]
 55%|█████▌    | 3359/6100 [5:04:23<4:07:18,  5.41s/it]
 55%|█████▌    | 3360/6100 [5:04:28<4:07:13,  5.41s/it]
                                                       
{'loss': 0.08, 'learning_rate': 0.0001, 'epoch': 5.51}

 55%|█████▌    | 3360/6100 [5:04:28<4:07:13,  5.41s/it]
 55%|█████▌    | 3361/6100 [5:04:33<4:07:02,  5.41s/it]
 55%|█████▌    | 3362/6100 [5:04:39<4:06:56,  5.41s/it]
 55%|█████▌    | 3363/6100 [5:04:44<4:07:01,  5.42s/it]
 55%|█████▌    | 3364/6100 [5:04:50<4:07:08,  5.42s/it]
 55%|█████▌    | 3365/6100 [5:04:55<4:06:50,  5.42s/it]
                                                       
{'loss': 0.0739, 'learning_rate': 0.0001, 'epoch': 5.52}

 55%|█████▌    | 3365/6100 [5:04:55<4:06:50,  5.42s/it]
 55%|█████▌    | 3366/6100 [5:05:00<4:06:53,  5.42s/it]
 55%|█████▌    | 3367/6100 [5:05:06<4:06:44,  5.42s/it]
 55%|█████▌    | 3368/6100 [5:05:11<4:06:26,  5.41s/it]
 55%|█████▌    | 3369/6100 [5:05:17<4:06:18,  5.41s/it]
 55%|█████▌    | 3370/6100 [5:05:22<4:06:10,  5.41s/it]
                                                       
{'loss': 0.073, 'learning_rate': 0.0001, 'epoch': 5.52}

 55%|█████▌    | 3370/6100 [5:05:22<4:06:10,  5.41s/it]
 55%|█████▌    | 3371/6100 [5:05:28<4:06:14,  5.41s/it]
 55%|█████▌    | 3372/6100 [5:05:33<4:06:15,  5.42s/it]
 55%|█████▌    | 3373/6100 [5:05:38<4:06:05,  5.41s/it]
 55%|█████▌    | 3374/6100 [5:05:44<4:06:01,  5.41s/it]
 55%|█████▌    | 3375/6100 [5:05:49<4:05:54,  5.41s/it]
                                                       
{'loss': 0.0812, 'learning_rate': 0.0001, 'epoch': 5.53}

 55%|█████▌    | 3375/6100 [5:05:49<4:05:54,  5.41s/it]
 55%|█████▌    | 3376/6100 [5:05:55<4:05:45,  5.41s/it]
 55%|█████▌    | 3377/6100 [5:06:00<4:05:30,  5.41s/it]
 55%|█████▌    | 3378/6100 [5:06:05<4:05:16,  5.41s/it]
 55%|█████▌    | 3379/6100 [5:06:11<4:05:21,  5.41s/it]
 55%|█████▌    | 3380/6100 [5:06:16<4:05:09,  5.41s/it]
                                                       
{'loss': 0.0727, 'learning_rate': 0.0001, 'epoch': 5.54}

 55%|█████▌    | 3380/6100 [5:06:16<4:05:09,  5.41s/it]
 55%|█████▌    | 3381/6100 [5:06:22<4:05:04,  5.41s/it]
 55%|█████▌    | 3382/6100 [5:06:27<4:05:01,  5.41s/it]
 55%|█████▌    | 3383/6100 [5:06:32<4:05:06,  5.41s/it]
 55%|█████▌    | 3384/6100 [5:06:38<4:05:17,  5.42s/it]
 55%|█████▌    | 3385/6100 [5:06:43<4:05:07,  5.42s/it]
                                                       
{'loss': 0.0756, 'learning_rate': 0.0001, 'epoch': 5.55}

 55%|█████▌    | 3385/6100 [5:06:43<4:05:07,  5.42s/it]
 56%|█████▌    | 3386/6100 [5:06:49<4:04:49,  5.41s/it]
 56%|█████▌    | 3387/6100 [5:06:54<4:05:01,  5.42s/it]
 56%|█████▌    | 3388/6100 [5:07:00<4:05:00,  5.42s/it]
 56%|█████▌    | 3389/6100 [5:07:05<4:04:56,  5.42s/it]
 56%|█████▌    | 3390/6100 [5:07:10<4:04:50,  5.42s/it]
                                                       
{'loss': 0.0781, 'learning_rate': 0.0001, 'epoch': 5.56}

 56%|█████▌    | 3390/6100 [5:07:10<4:04:50,  5.42s/it]
 56%|█████▌    | 3391/6100 [5:07:16<4:04:44,  5.42s/it]
 56%|█████▌    | 3392/6100 [5:07:21<4:04:54,  5.43s/it]
 56%|█████▌    | 3393/6100 [5:07:27<4:04:48,  5.43s/it]
 56%|█████▌    | 3394/6100 [5:07:32<4:04:44,  5.43s/it]
 56%|█████▌    | 3395/6100 [5:07:38<4:04:23,  5.42s/it]
                                                       
{'loss': 0.0772, 'learning_rate': 0.0001, 'epoch': 5.57}

 56%|█████▌    | 3395/6100 [5:07:38<4:04:23,  5.42s/it]
 56%|█████▌    | 3396/6100 [5:07:43<4:04:28,  5.42s/it]
 56%|█████▌    | 3397/6100 [5:07:48<4:04:08,  5.42s/it]
 56%|█████▌    | 3398/6100 [5:07:54<4:03:59,  5.42s/it]
 56%|█████▌    | 3399/6100 [5:07:59<4:03:52,  5.42s/it]
 56%|█████▌    | 3400/6100 [5:08:05<4:03:34,  5.41s/it]
                                                       
{'loss': 0.0784, 'learning_rate': 0.0001, 'epoch': 5.57}

 56%|█████▌    | 3400/6100 [5:08:05<4:03:34,  5.41s/it]
 56%|█████▌    | 3401/6100 [5:08:10<4:03:31,  5.41s/it]
 56%|█████▌    | 3402/6100 [5:08:15<4:03:29,  5.41s/it]
 56%|█████▌    | 3403/6100 [5:08:21<4:03:07,  5.41s/it]
 56%|█████▌    | 3404/6100 [5:08:26<4:03:04,  5.41s/it]
 56%|█████▌    | 3405/6100 [5:08:32<4:03:14,  5.42s/it]
                                                       
{'loss': 0.0738, 'learning_rate': 0.0001, 'epoch': 5.58}

 56%|█████▌    | 3405/6100 [5:08:32<4:03:14,  5.42s/it]
 56%|█████▌    | 3406/6100 [5:08:37<4:03:13,  5.42s/it]
 56%|█████▌    | 3407/6100 [5:08:43<4:03:17,  5.42s/it]
 56%|█████▌    | 3408/6100 [5:08:48<4:03:23,  5.42s/it]
 56%|█████▌    | 3409/6100 [5:08:53<4:03:06,  5.42s/it]
 56%|█████▌    | 3410/6100 [5:08:59<4:02:53,  5.42s/it]
                                                       
{'loss': 0.0742, 'learning_rate': 0.0001, 'epoch': 5.59}

 56%|█████▌    | 3410/6100 [5:08:59<4:02:53,  5.42s/it]
 56%|█████▌    | 3411/6100 [5:09:04<4:02:49,  5.42s/it]
 56%|█████▌    | 3412/6100 [5:09:10<4:02:47,  5.42s/it]
 56%|█████▌    | 3413/6100 [5:09:15<4:02:35,  5.42s/it]
 56%|█████▌    | 3414/6100 [5:09:20<4:02:22,  5.41s/it]
 56%|█████▌    | 3415/6100 [5:09:26<4:02:30,  5.42s/it]
                                                       
{'loss': 0.0813, 'learning_rate': 0.0001, 'epoch': 5.6}

 56%|█████▌    | 3415/6100 [5:09:26<4:02:30,  5.42s/it]
 56%|█████▌    | 3416/6100 [5:09:31<4:02:19,  5.42s/it]
 56%|█████▌    | 3417/6100 [5:09:37<4:02:09,  5.42s/it]
 56%|█████▌    | 3418/6100 [5:09:42<4:02:20,  5.42s/it]
 56%|█████▌    | 3419/6100 [5:09:48<4:02:15,  5.42s/it]
 56%|█████▌    | 3420/6100 [5:09:53<4:02:04,  5.42s/it]
                                                       
{'loss': 0.0787, 'learning_rate': 0.0001, 'epoch': 5.61}

 56%|█████▌    | 3420/6100 [5:09:53<4:02:04,  5.42s/it]
 56%|█████▌    | 3421/6100 [5:09:58<4:01:48,  5.42s/it]
 56%|█████▌    | 3422/6100 [5:10:04<4:01:46,  5.42s/it]
 56%|█████▌    | 3423/6100 [5:10:09<4:01:39,  5.42s/it]
 56%|█████▌    | 3424/6100 [5:10:15<4:01:30,  5.42s/it]
 56%|█████▌    | 3425/6100 [5:10:20<4:01:34,  5.42s/it]
                                                       
{'loss': 0.0754, 'learning_rate': 0.0001, 'epoch': 5.61}

 56%|█████▌    | 3425/6100 [5:10:20<4:01:34,  5.42s/it]
 56%|█████▌    | 3426/6100 [5:10:25<4:01:38,  5.42s/it]
 56%|█████▌    | 3427/6100 [5:10:31<4:01:23,  5.42s/it]
 56%|█████▌    | 3428/6100 [5:10:36<4:01:12,  5.42s/it]
 56%|█████▌    | 3429/6100 [5:10:42<4:01:23,  5.42s/it]
 56%|█████▌    | 3430/6100 [5:10:47<4:01:02,  5.42s/it]
                                                       
{'loss': 0.0801, 'learning_rate': 0.0001, 'epoch': 5.62}

 56%|█████▌    | 3430/6100 [5:10:47<4:01:02,  5.42s/it]
 56%|█████▌    | 3431/6100 [5:10:53<4:00:53,  5.42s/it]
 56%|█████▋    | 3432/6100 [5:10:58<4:00:42,  5.41s/it]
 56%|█████▋    | 3433/6100 [5:11:03<4:00:51,  5.42s/it]
 56%|█████▋    | 3434/6100 [5:11:09<4:00:51,  5.42s/it]
 56%|█████▋    | 3435/6100 [5:11:14<4:00:47,  5.42s/it]
                                                       
{'loss': 0.0813, 'learning_rate': 0.0001, 'epoch': 5.63}

 56%|█████▋    | 3435/6100 [5:11:14<4:00:47,  5.42s/it]
 56%|█████▋    | 3436/6100 [5:11:20<4:00:44,  5.42s/it]
 56%|█████▋    | 3437/6100 [5:11:25<4:00:32,  5.42s/it]
 56%|█████▋    | 3438/6100 [5:11:31<4:00:13,  5.41s/it]
 56%|█████▋    | 3439/6100 [5:11:36<4:00:09,  5.42s/it]
 56%|█████▋    | 3440/6100 [5:11:41<4:00:02,  5.41s/it]
                                                       
{'loss': 0.0752, 'learning_rate': 0.0001, 'epoch': 5.64}

 56%|█████▋    | 3440/6100 [5:11:41<4:00:02,  5.41s/it]
 56%|█████▋    | 3441/6100 [5:11:47<3:59:52,  5.41s/it]
 56%|█████▋    | 3442/6100 [5:11:52<3:59:51,  5.41s/it]
 56%|█████▋    | 3443/6100 [5:11:58<3:59:42,  5.41s/it]
 56%|█████▋    | 3444/6100 [5:12:03<3:59:54,  5.42s/it]
 56%|█████▋    | 3445/6100 [5:12:08<3:59:35,  5.41s/it]
                                                       
{'loss': 0.0717, 'learning_rate': 0.0001, 'epoch': 5.65}

 56%|█████▋    | 3445/6100 [5:12:08<3:59:35,  5.41s/it]
 56%|█████▋    | 3446/6100 [5:12:14<3:59:41,  5.42s/it]
 57%|█████▋    | 3447/6100 [5:12:19<3:59:27,  5.42s/it]
 57%|█████▋    | 3448/6100 [5:12:25<3:59:28,  5.42s/it]
 57%|█████▋    | 3449/6100 [5:12:30<3:59:40,  5.42s/it]
 57%|█████▋    | 3450/6100 [5:12:36<3:59:27,  5.42s/it]
                                                       
{'loss': 0.0815, 'learning_rate': 0.0001, 'epoch': 5.66}

 57%|█████▋    | 3450/6100 [5:12:36<3:59:27,  5.42s/it]Saving model checkpoint to ./results/checkpoint-3450
Configuration saved in ./results/checkpoint-3450/config.json
Model weights saved in ./results/checkpoint-3450/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-3450/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-3450/special_tokens_map.json
[2023-02-22 01:04:51,710] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step3452 is begin to save!
[2023-02-22 01:04:51,713] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-3450/global_step3452/mp_rank_00_model_states.pt
[2023-02-22 01:04:51,713] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-3450/global_step3452/mp_rank_00_model_states.pt...
[2023-02-22 01:04:52,600] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-3450/global_step3452/mp_rank_00_model_states.pt.
[2023-02-22 01:04:52,602] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-3450/global_step3452/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 01:04:52,814] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-3450/global_step3452/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 01:04:52,815] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-3450/global_step3452/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 01:04:52,815] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3452 is ready now!
Deleting older checkpoint [results/checkpoint-3300] due to args.save_total_limit

 57%|█████▋    | 3451/6100 [5:12:43<4:29:08,  6.10s/it]
 57%|█████▋    | 3452/6100 [5:12:49<4:19:54,  5.89s/it]
 57%|█████▋    | 3453/6100 [5:12:54<4:13:28,  5.75s/it]
 57%|█████▋    | 3454/6100 [5:12:59<4:09:13,  5.65s/it]
 57%|█████▋    | 3455/6100 [5:13:05<4:05:55,  5.58s/it]
                                                       
{'loss': 0.079, 'learning_rate': 0.0001, 'epoch': 5.66}

 57%|█████▋    | 3455/6100 [5:13:05<4:05:55,  5.58s/it]
 57%|█████▋    | 3456/6100 [5:13:10<4:03:34,  5.53s/it]
 57%|█████▋    | 3457/6100 [5:13:16<4:01:48,  5.49s/it]
 57%|█████▋    | 3458/6100 [5:13:21<4:01:34,  5.49s/it]
 57%|█████▋    | 3459/6100 [5:13:27<4:00:33,  5.47s/it]
 57%|█████▋    | 3460/6100 [5:13:32<3:59:39,  5.45s/it]
                                                       
{'loss': 0.0756, 'learning_rate': 0.0001, 'epoch': 5.67}

 57%|█████▋    | 3460/6100 [5:13:32<3:59:39,  5.45s/it]
 57%|█████▋    | 3461/6100 [5:13:37<3:59:02,  5.43s/it]
 57%|█████▋    | 3462/6100 [5:13:43<3:58:34,  5.43s/it]
 57%|█████▋    | 3463/6100 [5:13:48<3:58:06,  5.42s/it]
 57%|█████▋    | 3464/6100 [5:13:54<3:57:48,  5.41s/it]
 57%|█████▋    | 3465/6100 [5:13:59<3:57:37,  5.41s/it]
                                                       
{'loss': 0.0783, 'learning_rate': 0.0001, 'epoch': 5.68}

 57%|█████▋    | 3465/6100 [5:13:59<3:57:37,  5.41s/it]
 57%|█████▋    | 3466/6100 [5:14:04<3:57:27,  5.41s/it]
 57%|█████▋    | 3467/6100 [5:14:10<3:57:15,  5.41s/it]
 57%|█████▋    | 3468/6100 [5:14:15<3:57:05,  5.40s/it]
 57%|█████▋    | 3469/6100 [5:14:21<3:57:06,  5.41s/it]
 57%|█████▋    | 3470/6100 [5:14:26<3:57:09,  5.41s/it]
                                                       
{'loss': 0.0739, 'learning_rate': 0.0001, 'epoch': 5.69}

 57%|█████▋    | 3470/6100 [5:14:26<3:57:09,  5.41s/it]
 57%|█████▋    | 3471/6100 [5:14:31<3:57:00,  5.41s/it]
 57%|█████▋    | 3472/6100 [5:14:37<3:56:55,  5.41s/it]
 57%|█████▋    | 3473/6100 [5:14:42<3:56:56,  5.41s/it]
 57%|█████▋    | 3474/6100 [5:14:48<3:56:57,  5.41s/it]
 57%|█████▋    | 3475/6100 [5:14:53<3:56:37,  5.41s/it]
                                                       
{'loss': 0.082, 'learning_rate': 0.0001, 'epoch': 5.7}

 57%|█████▋    | 3475/6100 [5:14:53<3:56:37,  5.41s/it]
 57%|█████▋    | 3476/6100 [5:14:58<3:56:34,  5.41s/it]
 57%|█████▋    | 3477/6100 [5:15:04<3:56:31,  5.41s/it]
 57%|█████▋    | 3478/6100 [5:15:09<3:56:28,  5.41s/it]
 57%|█████▋    | 3479/6100 [5:15:15<3:56:22,  5.41s/it]
 57%|█████▋    | 3480/6100 [5:15:20<3:56:17,  5.41s/it]
                                                       
{'loss': 0.0745, 'learning_rate': 0.0001, 'epoch': 5.7}

 57%|█████▋    | 3480/6100 [5:15:20<3:56:17,  5.41s/it]
 57%|█████▋    | 3481/6100 [5:15:26<3:56:24,  5.42s/it]
 57%|█████▋    | 3482/6100 [5:15:31<3:56:13,  5.41s/it]
 57%|█████▋    | 3483/6100 [5:15:36<3:56:04,  5.41s/it]
 57%|█████▋    | 3484/6100 [5:15:42<3:55:53,  5.41s/it]
 57%|█████▋    | 3485/6100 [5:15:47<3:55:52,  5.41s/it]
                                                       
{'loss': 0.0743, 'learning_rate': 0.0001, 'epoch': 5.71}

 57%|█████▋    | 3485/6100 [5:15:47<3:55:52,  5.41s/it]
 57%|█████▋    | 3486/6100 [5:15:53<3:55:46,  5.41s/it]
 57%|█████▋    | 3487/6100 [5:15:58<3:55:43,  5.41s/it]
 57%|█████▋    | 3488/6100 [5:16:03<3:55:34,  5.41s/it]
 57%|█████▋    | 3489/6100 [5:16:09<3:56:08,  5.43s/it]
 57%|█████▋    | 3490/6100 [5:16:14<3:55:47,  5.42s/it]
                                                       
{'loss': 0.0826, 'learning_rate': 0.0001, 'epoch': 5.72}

 57%|█████▋    | 3490/6100 [5:16:14<3:55:47,  5.42s/it]
 57%|█████▋    | 3491/6100 [5:16:20<3:55:37,  5.42s/it]
 57%|█████▋    | 3492/6100 [5:16:25<3:55:29,  5.42s/it]
 57%|█████▋    | 3493/6100 [5:16:31<3:55:15,  5.41s/it]
 57%|█████▋    | 3494/6100 [5:16:36<4:01:45,  5.57s/it]
 57%|█████▋    | 3495/6100 [5:16:42<3:59:38,  5.52s/it]
                                                       
{'loss': 0.0836, 'learning_rate': 0.0001, 'epoch': 5.73}

 57%|█████▋    | 3495/6100 [5:16:42<3:59:38,  5.52s/it]
 57%|█████▋    | 3496/6100 [5:16:47<3:58:03,  5.49s/it]
 57%|█████▋    | 3497/6100 [5:16:53<3:56:53,  5.46s/it]
 57%|█████▋    | 3498/6100 [5:16:58<3:56:01,  5.44s/it]
 57%|█████▋    | 3499/6100 [5:17:03<3:55:29,  5.43s/it]
 57%|█████▋    | 3500/6100 [5:17:09<3:55:07,  5.43s/it]
                                                       
{'loss': 0.079, 'learning_rate': 0.0001, 'epoch': 5.74}

 57%|█████▋    | 3500/6100 [5:17:09<3:55:07,  5.43s/it]
 57%|█████▋    | 3501/6100 [5:17:14<3:54:49,  5.42s/it]
 57%|█████▋    | 3502/6100 [5:17:20<3:54:28,  5.42s/it]
 57%|█████▋    | 3503/6100 [5:17:25<3:54:19,  5.41s/it]
 57%|█████▋    | 3504/6100 [5:17:31<3:54:02,  5.41s/it]
 57%|█████▋    | 3505/6100 [5:17:36<3:53:50,  5.41s/it]
                                                       
{'loss': 0.0745, 'learning_rate': 0.0001, 'epoch': 5.75}

 57%|█████▋    | 3505/6100 [5:17:36<3:53:50,  5.41s/it]
 57%|█████▋    | 3506/6100 [5:17:41<3:53:49,  5.41s/it]
 57%|█████▋    | 3507/6100 [5:17:47<3:53:57,  5.41s/it]
 58%|█████▊    | 3508/6100 [5:17:52<3:53:46,  5.41s/it]
 58%|█████▊    | 3509/6100 [5:17:58<3:53:40,  5.41s/it]
 58%|█████▊    | 3510/6100 [5:18:03<3:53:39,  5.41s/it]
                                                       
{'loss': 0.071, 'learning_rate': 0.0001, 'epoch': 5.75}

 58%|█████▊    | 3510/6100 [5:18:03<3:53:39,  5.41s/it]
 58%|█████▊    | 3511/6100 [5:18:08<3:53:38,  5.41s/it]
 58%|█████▊    | 3512/6100 [5:18:14<3:53:40,  5.42s/it]
 58%|█████▊    | 3513/6100 [5:18:19<3:53:27,  5.41s/it]
 58%|█████▊    | 3514/6100 [5:18:25<3:53:18,  5.41s/it]
 58%|█████▊    | 3515/6100 [5:18:30<3:53:13,  5.41s/it]
                                                       
{'loss': 0.0732, 'learning_rate': 0.0001, 'epoch': 5.76}

 58%|█████▊    | 3515/6100 [5:18:30<3:53:13,  5.41s/it]
 58%|█████▊    | 3516/6100 [5:18:35<3:53:12,  5.41s/it]
 58%|█████▊    | 3517/6100 [5:18:41<3:53:10,  5.42s/it]
 58%|█████▊    | 3518/6100 [5:18:46<3:53:59,  5.44s/it]
 58%|█████▊    | 3519/6100 [5:18:52<3:53:17,  5.42s/it]
 58%|█████▊    | 3520/6100 [5:18:57<3:53:07,  5.42s/it]
                                                       
{'loss': 0.075, 'learning_rate': 0.0001, 'epoch': 5.77}

 58%|█████▊    | 3520/6100 [5:18:57<3:53:07,  5.42s/it]
 58%|█████▊    | 3521/6100 [5:19:03<3:53:07,  5.42s/it]
 58%|█████▊    | 3522/6100 [5:19:08<3:53:06,  5.43s/it]
 58%|█████▊    | 3523/6100 [5:19:13<3:52:49,  5.42s/it]
 58%|█████▊    | 3524/6100 [5:19:19<3:52:29,  5.42s/it]
 58%|█████▊    | 3525/6100 [5:19:24<3:52:29,  5.42s/it]
                                                       
{'loss': 0.0772, 'learning_rate': 0.0001, 'epoch': 5.78}

 58%|█████▊    | 3525/6100 [5:19:24<3:52:29,  5.42s/it]
 58%|█████▊    | 3526/6100 [5:19:30<3:52:18,  5.42s/it]
 58%|█████▊    | 3527/6100 [5:19:35<3:52:17,  5.42s/it]
 58%|█████▊    | 3528/6100 [5:19:41<3:52:11,  5.42s/it]
 58%|█████▊    | 3529/6100 [5:19:46<3:52:03,  5.42s/it]
 58%|█████▊    | 3530/6100 [5:19:51<3:52:02,  5.42s/it]
                                                       
{'loss': 0.0749, 'learning_rate': 0.0001, 'epoch': 5.79}

 58%|█████▊    | 3530/6100 [5:19:51<3:52:02,  5.42s/it]
 58%|█████▊    | 3531/6100 [5:19:57<3:51:55,  5.42s/it]
 58%|█████▊    | 3532/6100 [5:20:02<3:51:57,  5.42s/it]
 58%|█████▊    | 3533/6100 [5:20:08<3:51:58,  5.42s/it]
 58%|█████▊    | 3534/6100 [5:20:13<3:51:55,  5.42s/it]
 58%|█████▊    | 3535/6100 [5:20:18<3:51:41,  5.42s/it]
                                                       
{'loss': 0.0767, 'learning_rate': 0.0001, 'epoch': 5.79}

 58%|█████▊    | 3535/6100 [5:20:18<3:51:41,  5.42s/it]
 58%|█████▊    | 3536/6100 [5:20:24<3:51:34,  5.42s/it]
 58%|█████▊    | 3537/6100 [5:20:29<3:51:15,  5.41s/it]
 58%|█████▊    | 3538/6100 [5:20:35<3:51:07,  5.41s/it]
 58%|█████▊    | 3539/6100 [5:20:40<3:51:05,  5.41s/it]
 58%|█████▊    | 3540/6100 [5:20:46<3:51:13,  5.42s/it]
                                                       
{'loss': 0.075, 'learning_rate': 0.0001, 'epoch': 5.8}

 58%|█████▊    | 3540/6100 [5:20:46<3:51:13,  5.42s/it]
 58%|█████▊    | 3541/6100 [5:20:51<3:50:58,  5.42s/it]
 58%|█████▊    | 3542/6100 [5:20:56<3:50:48,  5.41s/it]
 58%|█████▊    | 3543/6100 [5:21:02<3:50:57,  5.42s/it]
 58%|█████▊    | 3544/6100 [5:21:07<3:50:44,  5.42s/it]
 58%|█████▊    | 3545/6100 [5:21:13<3:50:39,  5.42s/it]
                                                       
{'loss': 0.0763, 'learning_rate': 0.0001, 'epoch': 5.81}

 58%|█████▊    | 3545/6100 [5:21:13<3:50:39,  5.42s/it]
 58%|█████▊    | 3546/6100 [5:21:18<3:50:31,  5.42s/it]
 58%|█████▊    | 3547/6100 [5:21:23<3:50:18,  5.41s/it]
 58%|█████▊    | 3548/6100 [5:21:29<3:50:22,  5.42s/it]
 58%|█████▊    | 3549/6100 [5:21:34<3:50:05,  5.41s/it]
 58%|█████▊    | 3550/6100 [5:21:40<3:50:04,  5.41s/it]
                                                       
{'loss': 0.0833, 'learning_rate': 0.0001, 'epoch': 5.82}

 58%|█████▊    | 3550/6100 [5:21:40<3:50:04,  5.41s/it]
 58%|█████▊    | 3551/6100 [5:21:45<3:50:02,  5.41s/it]
 58%|█████▊    | 3552/6100 [5:21:51<3:49:56,  5.41s/it]
 58%|█████▊    | 3553/6100 [5:21:56<3:50:03,  5.42s/it]
 58%|█████▊    | 3554/6100 [5:22:02<3:53:03,  5.49s/it]
 58%|█████▊    | 3555/6100 [5:22:07<3:51:59,  5.47s/it]
                                                       
{'loss': 0.0768, 'learning_rate': 0.0001, 'epoch': 5.83}

 58%|█████▊    | 3555/6100 [5:22:07<3:51:59,  5.47s/it]
 58%|█████▊    | 3556/6100 [5:22:12<3:51:15,  5.45s/it]
 58%|█████▊    | 3557/6100 [5:22:18<3:50:34,  5.44s/it]
 58%|█████▊    | 3558/6100 [5:22:23<3:49:58,  5.43s/it]
 58%|█████▊    | 3559/6100 [5:22:29<3:49:55,  5.43s/it]
 58%|█████▊    | 3560/6100 [5:22:34<3:49:42,  5.43s/it]
                                                       
{'loss': 0.0803, 'learning_rate': 0.0001, 'epoch': 5.84}

 58%|█████▊    | 3560/6100 [5:22:34<3:49:42,  5.43s/it]
 58%|█████▊    | 3561/6100 [5:22:40<3:58:49,  5.64s/it]
 58%|█████▊    | 3562/6100 [5:22:46<3:55:40,  5.57s/it]
 58%|█████▊    | 3563/6100 [5:22:51<3:53:34,  5.52s/it]
 58%|█████▊    | 3564/6100 [5:22:57<3:52:25,  5.50s/it]
 58%|█████▊    | 3565/6100 [5:23:02<3:51:19,  5.48s/it]
                                                       
{'loss': 0.0761, 'learning_rate': 0.0001, 'epoch': 5.84}

 58%|█████▊    | 3565/6100 [5:23:02<3:51:19,  5.48s/it]
 58%|█████▊    | 3566/6100 [5:23:07<3:50:28,  5.46s/it]
 58%|█████▊    | 3567/6100 [5:23:13<3:49:53,  5.45s/it]
 58%|█████▊    | 3568/6100 [5:23:18<3:49:24,  5.44s/it]
 59%|█████▊    | 3569/6100 [5:23:24<3:48:59,  5.43s/it]
 59%|█████▊    | 3570/6100 [5:23:29<3:48:32,  5.42s/it]
                                                       
{'loss': 0.076, 'learning_rate': 0.0001, 'epoch': 5.85}

 59%|█████▊    | 3570/6100 [5:23:29<3:48:32,  5.42s/it]
 59%|█████▊    | 3571/6100 [5:23:34<3:48:26,  5.42s/it]
 59%|█████▊    | 3572/6100 [5:23:40<3:48:07,  5.41s/it]
 59%|█████▊    | 3573/6100 [5:23:45<3:48:01,  5.41s/it]
 59%|█████▊    | 3574/6100 [5:23:51<3:47:55,  5.41s/it]
 59%|█████▊    | 3575/6100 [5:23:56<3:47:46,  5.41s/it]
                                                       
{'loss': 0.079, 'learning_rate': 0.0001, 'epoch': 5.86}

 59%|█████▊    | 3575/6100 [5:23:56<3:47:46,  5.41s/it]
 59%|█████▊    | 3576/6100 [5:24:01<3:47:45,  5.41s/it]
 59%|█████▊    | 3577/6100 [5:24:07<3:47:48,  5.42s/it]
 59%|█████▊    | 3578/6100 [5:24:12<3:47:33,  5.41s/it]
 59%|█████▊    | 3579/6100 [5:24:18<3:47:31,  5.42s/it]
 59%|█████▊    | 3580/6100 [5:24:23<3:47:24,  5.41s/it]
                                                       
{'loss': 0.0714, 'learning_rate': 0.0001, 'epoch': 5.87}

 59%|█████▊    | 3580/6100 [5:24:23<3:47:24,  5.41s/it]
 59%|█████▊    | 3581/6100 [5:24:29<3:47:21,  5.42s/it]
 59%|█████▊    | 3582/6100 [5:24:34<3:47:22,  5.42s/it]
 59%|█████▊    | 3583/6100 [5:24:39<3:47:19,  5.42s/it]
 59%|█████▉    | 3584/6100 [5:24:45<3:47:04,  5.42s/it]
 59%|█████▉    | 3585/6100 [5:24:50<3:47:07,  5.42s/it]
                                                       
{'loss': 0.0724, 'learning_rate': 0.0001, 'epoch': 5.88}

 59%|█████▉    | 3585/6100 [5:24:50<3:47:07,  5.42s/it]
 59%|█████▉    | 3586/6100 [5:24:56<3:47:05,  5.42s/it]
 59%|█████▉    | 3587/6100 [5:25:01<3:47:23,  5.43s/it]
 59%|█████▉    | 3588/6100 [5:25:07<3:47:04,  5.42s/it]
 59%|█████▉    | 3589/6100 [5:25:12<3:46:38,  5.42s/it]
 59%|█████▉    | 3590/6100 [5:25:17<3:46:33,  5.42s/it]
                                                       
{'loss': 0.0744, 'learning_rate': 0.0001, 'epoch': 5.88}

 59%|█████▉    | 3590/6100 [5:25:17<3:46:33,  5.42s/it]
 59%|█████▉    | 3591/6100 [5:25:23<3:46:33,  5.42s/it]
 59%|█████▉    | 3592/6100 [5:25:28<3:46:25,  5.42s/it]
 59%|█████▉    | 3593/6100 [5:25:34<3:46:19,  5.42s/it]
 59%|█████▉    | 3594/6100 [5:25:39<3:46:04,  5.41s/it]
 59%|█████▉    | 3595/6100 [5:25:44<3:45:54,  5.41s/it]
                                                       
{'loss': 0.0757, 'learning_rate': 0.0001, 'epoch': 5.89}

 59%|█████▉    | 3595/6100 [5:25:44<3:45:54,  5.41s/it]
 59%|█████▉    | 3596/6100 [5:25:50<3:46:07,  5.42s/it]
 59%|█████▉    | 3597/6100 [5:25:55<3:46:08,  5.42s/it]
 59%|█████▉    | 3598/6100 [5:26:01<3:46:09,  5.42s/it]
 59%|█████▉    | 3599/6100 [5:26:06<3:46:03,  5.42s/it]
 59%|█████▉    | 3600/6100 [5:26:12<3:45:53,  5.42s/it]
                                                       
{'loss': 0.0711, 'learning_rate': 0.0001, 'epoch': 5.9}

 59%|█████▉    | 3600/6100 [5:26:12<3:45:53,  5.42s/it]Saving model checkpoint to ./results/checkpoint-3600
Configuration saved in ./results/checkpoint-3600/config.json
Model weights saved in ./results/checkpoint-3600/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-3600/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-3600/special_tokens_map.json
[2023-02-22 01:18:27,687] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step3602 is begin to save!
[2023-02-22 01:18:27,691] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-3600/global_step3602/mp_rank_00_model_states.pt
[2023-02-22 01:18:27,691] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-3600/global_step3602/mp_rank_00_model_states.pt...
[2023-02-22 01:18:28,507] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-3600/global_step3602/mp_rank_00_model_states.pt.
[2023-02-22 01:18:28,509] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-3600/global_step3602/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 01:18:28,682] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-3600/global_step3602/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 01:18:28,682] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-3600/global_step3602/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 01:18:28,682] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3602 is ready now!
Deleting older checkpoint [results/checkpoint-3450] due to args.save_total_limit

 59%|█████▉    | 3601/6100 [5:26:19<4:13:01,  6.08s/it]
 59%|█████▉    | 3602/6100 [5:26:25<4:04:33,  5.87s/it]
 59%|█████▉    | 3603/6100 [5:26:30<3:58:39,  5.73s/it]
 59%|█████▉    | 3604/6100 [5:26:35<3:54:25,  5.64s/it]
 59%|█████▉    | 3605/6100 [5:26:41<3:51:28,  5.57s/it]
                                                       
{'loss': 0.0718, 'learning_rate': 0.0001, 'epoch': 5.91}

 59%|█████▉    | 3605/6100 [5:26:41<3:51:28,  5.57s/it]
 59%|█████▉    | 3606/6100 [5:26:46<3:49:29,  5.52s/it]
 59%|█████▉    | 3607/6100 [5:26:52<3:48:11,  5.49s/it]
 59%|█████▉    | 3608/6100 [5:26:57<3:47:16,  5.47s/it]
 59%|█████▉    | 3609/6100 [5:27:02<3:46:23,  5.45s/it]
 59%|█████▉    | 3610/6100 [5:27:08<3:45:50,  5.44s/it]
                                                       
{'loss': 0.0752, 'learning_rate': 0.0001, 'epoch': 5.92}

 59%|█████▉    | 3610/6100 [5:27:08<3:45:50,  5.44s/it]
 59%|█████▉    | 3611/6100 [5:27:13<3:45:24,  5.43s/it]
 59%|█████▉    | 3612/6100 [5:27:19<3:45:02,  5.43s/it]
 59%|█████▉    | 3613/6100 [5:27:24<3:44:45,  5.42s/it]
 59%|█████▉    | 3614/6100 [5:27:29<3:44:37,  5.42s/it]
 59%|█████▉    | 3615/6100 [5:27:35<3:44:20,  5.42s/it]
                                                       
{'loss': 0.0722, 'learning_rate': 0.0001, 'epoch': 5.93}

 59%|█████▉    | 3615/6100 [5:27:35<3:44:20,  5.42s/it]
 59%|█████▉    | 3616/6100 [5:27:40<3:44:10,  5.42s/it]
 59%|█████▉    | 3617/6100 [5:27:46<3:43:58,  5.41s/it]
 59%|█████▉    | 3618/6100 [5:27:51<3:43:48,  5.41s/it]
 59%|█████▉    | 3619/6100 [5:27:57<3:43:45,  5.41s/it]
 59%|█████▉    | 3620/6100 [5:28:03<3:50:58,  5.59s/it]
                                                       
{'loss': 0.0755, 'learning_rate': 0.0001, 'epoch': 5.93}

 59%|█████▉    | 3620/6100 [5:28:03<3:50:58,  5.59s/it]
 59%|█████▉    | 3621/6100 [5:28:08<3:48:43,  5.54s/it]
 59%|█████▉    | 3622/6100 [5:28:13<3:47:06,  5.50s/it]
 59%|█████▉    | 3623/6100 [5:28:19<3:45:56,  5.47s/it]
 59%|█████▉    | 3624/6100 [5:28:24<3:45:12,  5.46s/it]
 59%|█████▉    | 3625/6100 [5:28:30<3:44:34,  5.44s/it]
                                                       
{'loss': 0.0808, 'learning_rate': 0.0001, 'epoch': 5.94}

 59%|█████▉    | 3625/6100 [5:28:30<3:44:34,  5.44s/it]
 59%|█████▉    | 3626/6100 [5:28:35<3:43:59,  5.43s/it]
 59%|█████▉    | 3627/6100 [5:28:40<3:43:33,  5.42s/it]
 59%|█████▉    | 3628/6100 [5:28:46<3:43:13,  5.42s/it]
 59%|█████▉    | 3629/6100 [5:28:51<3:42:58,  5.41s/it]
 60%|█████▉    | 3630/6100 [5:28:57<3:42:58,  5.42s/it]
                                                       
{'loss': 0.0712, 'learning_rate': 0.0001, 'epoch': 5.95}

 60%|█████▉    | 3630/6100 [5:28:57<3:42:58,  5.42s/it]
 60%|█████▉    | 3631/6100 [5:29:02<3:42:49,  5.42s/it]
 60%|█████▉    | 3632/6100 [5:29:07<3:42:36,  5.41s/it]
 60%|█████▉    | 3633/6100 [5:29:13<3:42:46,  5.42s/it]
 60%|█████▉    | 3634/6100 [5:29:18<3:42:37,  5.42s/it]
 60%|█████▉    | 3635/6100 [5:29:24<3:42:26,  5.41s/it]
                                                       
{'loss': 0.071, 'learning_rate': 0.0001, 'epoch': 5.96}

 60%|█████▉    | 3635/6100 [5:29:24<3:42:26,  5.41s/it]
 60%|█████▉    | 3636/6100 [5:29:29<3:42:22,  5.42s/it]
 60%|█████▉    | 3637/6100 [5:29:35<3:42:21,  5.42s/it]
 60%|█████▉    | 3638/6100 [5:29:40<3:42:09,  5.41s/it]
 60%|█████▉    | 3639/6100 [5:29:45<3:42:03,  5.41s/it]
 60%|█████▉    | 3640/6100 [5:29:51<3:42:03,  5.42s/it]
                                                       
{'loss': 0.0742, 'learning_rate': 0.0001, 'epoch': 5.97}

 60%|█████▉    | 3640/6100 [5:29:51<3:42:03,  5.42s/it]
 60%|█████▉    | 3641/6100 [5:29:56<3:42:00,  5.42s/it]
 60%|█████▉    | 3642/6100 [5:30:02<3:41:50,  5.42s/it]
 60%|█████▉    | 3643/6100 [5:30:07<3:41:32,  5.41s/it]
 60%|█████▉    | 3644/6100 [5:30:12<3:41:29,  5.41s/it]
 60%|█████▉    | 3645/6100 [5:30:18<3:41:21,  5.41s/it]
                                                       
{'loss': 0.0808, 'learning_rate': 0.0001, 'epoch': 5.97}

 60%|█████▉    | 3645/6100 [5:30:18<3:41:21,  5.41s/it]
 60%|█████▉    | 3646/6100 [5:30:23<3:41:19,  5.41s/it]
 60%|█████▉    | 3647/6100 [5:30:29<3:41:13,  5.41s/it]
 60%|█████▉    | 3648/6100 [5:30:34<3:41:09,  5.41s/it]
 60%|█████▉    | 3649/6100 [5:30:39<3:41:02,  5.41s/it]
 60%|█████▉    | 3650/6100 [5:30:45<3:41:08,  5.42s/it]
                                                       
{'loss': 0.0752, 'learning_rate': 0.0001, 'epoch': 5.98}

 60%|█████▉    | 3650/6100 [5:30:45<3:41:08,  5.42s/it]
 60%|█████▉    | 3651/6100 [5:30:50<3:41:06,  5.42s/it]
 60%|█████▉    | 3652/6100 [5:30:56<3:42:24,  5.45s/it]
 60%|█████▉    | 3653/6100 [5:31:01<3:41:51,  5.44s/it]
 60%|█████▉    | 3654/6100 [5:31:07<3:41:30,  5.43s/it]
 60%|█████▉    | 3655/6100 [5:31:12<3:41:17,  5.43s/it]
                                                       
{'loss': 0.0742, 'learning_rate': 0.0001, 'epoch': 5.99}

 60%|█████▉    | 3655/6100 [5:31:12<3:41:17,  5.43s/it]
 60%|█████▉    | 3656/6100 [5:31:18<3:41:13,  5.43s/it]
 60%|█████▉    | 3657/6100 [5:31:23<3:40:59,  5.43s/it]
 60%|█████▉    | 3658/6100 [5:31:28<3:40:55,  5.43s/it]
 60%|█████▉    | 3659/6100 [5:31:34<3:40:47,  5.43s/it]
 60%|██████    | 3660/6100 [5:31:39<3:40:42,  5.43s/it]
                                                       
{'loss': 0.073, 'learning_rate': 0.0001, 'epoch': 6.0}

 60%|██████    | 3660/6100 [5:31:39<3:40:42,  5.43s/it]
 60%|██████    | 3661/6100 [5:31:48<4:15:44,  6.29s/it]
 60%|██████    | 3662/6100 [5:31:53<4:04:49,  6.03s/it]
 60%|██████    | 3663/6100 [5:31:58<3:57:13,  5.84s/it]
 60%|██████    | 3664/6100 [5:32:04<3:51:46,  5.71s/it]
 60%|██████    | 3665/6100 [5:32:09<3:48:15,  5.62s/it]
                                                       
{'loss': 0.0796, 'learning_rate': 0.0001, 'epoch': 6.01}

 60%|██████    | 3665/6100 [5:32:09<3:48:15,  5.62s/it]
 60%|██████    | 3666/6100 [5:32:15<3:45:40,  5.56s/it]
 60%|██████    | 3667/6100 [5:32:20<3:43:41,  5.52s/it]
 60%|██████    | 3668/6100 [5:32:25<3:42:29,  5.49s/it]
 60%|██████    | 3669/6100 [5:32:31<3:41:23,  5.46s/it]
 60%|██████    | 3670/6100 [5:32:36<3:40:33,  5.45s/it]
                                                       
{'loss': 0.0698, 'learning_rate': 0.0001, 'epoch': 6.02}

 60%|██████    | 3670/6100 [5:32:36<3:40:33,  5.45s/it]
 60%|██████    | 3671/6100 [5:32:42<3:40:06,  5.44s/it]
 60%|██████    | 3672/6100 [5:32:47<3:39:42,  5.43s/it]
 60%|██████    | 3673/6100 [5:32:53<3:39:27,  5.43s/it]
 60%|██████    | 3674/6100 [5:32:58<3:39:11,  5.42s/it]
 60%|██████    | 3675/6100 [5:33:03<3:39:04,  5.42s/it]
                                                       
{'loss': 0.074, 'learning_rate': 0.0001, 'epoch': 6.02}

 60%|██████    | 3675/6100 [5:33:03<3:39:04,  5.42s/it]
 60%|██████    | 3676/6100 [5:33:09<3:39:16,  5.43s/it]
 60%|██████    | 3677/6100 [5:33:14<3:38:57,  5.42s/it]
 60%|██████    | 3678/6100 [5:33:20<3:38:45,  5.42s/it]
 60%|██████    | 3679/6100 [5:33:25<3:38:35,  5.42s/it]
 60%|██████    | 3680/6100 [5:33:30<3:38:39,  5.42s/it]
                                                       
{'loss': 0.0698, 'learning_rate': 0.0001, 'epoch': 6.03}

 60%|██████    | 3680/6100 [5:33:30<3:38:39,  5.42s/it]
 60%|██████    | 3681/6100 [5:33:36<3:38:33,  5.42s/it]
 60%|██████    | 3682/6100 [5:33:41<3:38:31,  5.42s/it]
 60%|██████    | 3683/6100 [5:33:47<3:38:36,  5.43s/it]
 60%|██████    | 3684/6100 [5:33:52<3:38:23,  5.42s/it]
 60%|██████    | 3685/6100 [5:33:58<3:39:08,  5.44s/it]
                                                       
{'loss': 0.0737, 'learning_rate': 0.0001, 'epoch': 6.04}

 60%|██████    | 3685/6100 [5:33:58<3:39:08,  5.44s/it]
 60%|██████    | 3686/6100 [5:34:03<3:38:42,  5.44s/it]
 60%|██████    | 3687/6100 [5:34:08<3:38:21,  5.43s/it]
 60%|██████    | 3688/6100 [5:34:14<3:37:57,  5.42s/it]
 60%|██████    | 3689/6100 [5:34:19<3:37:42,  5.42s/it]
 60%|██████    | 3690/6100 [5:34:25<3:37:26,  5.41s/it]
                                                       
{'loss': 0.0735, 'learning_rate': 0.0001, 'epoch': 6.05}

 60%|██████    | 3690/6100 [5:34:25<3:37:26,  5.41s/it]
 61%|██████    | 3691/6100 [5:34:30<3:37:42,  5.42s/it]
 61%|██████    | 3692/6100 [5:34:36<3:37:41,  5.42s/it]
 61%|██████    | 3693/6100 [5:34:41<3:37:30,  5.42s/it]
 61%|██████    | 3694/6100 [5:34:46<3:37:20,  5.42s/it]
 61%|██████    | 3695/6100 [5:34:52<3:37:17,  5.42s/it]
                                                       
{'loss': 0.0765, 'learning_rate': 0.0001, 'epoch': 6.06}

 61%|██████    | 3695/6100 [5:34:52<3:37:17,  5.42s/it]
 61%|██████    | 3696/6100 [5:34:57<3:37:00,  5.42s/it]
 61%|██████    | 3697/6100 [5:35:03<3:37:04,  5.42s/it]
 61%|██████    | 3698/6100 [5:35:08<3:37:03,  5.42s/it]
 61%|██████    | 3699/6100 [5:35:13<3:36:45,  5.42s/it]
 61%|██████    | 3700/6100 [5:35:19<3:36:37,  5.42s/it]
                                                       
{'loss': 0.0708, 'learning_rate': 0.0001, 'epoch': 6.07}

 61%|██████    | 3700/6100 [5:35:19<3:36:37,  5.42s/it]
 61%|██████    | 3701/6100 [5:35:24<3:36:43,  5.42s/it]
 61%|██████    | 3702/6100 [5:35:30<3:36:34,  5.42s/it]
 61%|██████    | 3703/6100 [5:35:35<3:36:27,  5.42s/it]
 61%|██████    | 3704/6100 [5:35:41<3:36:12,  5.41s/it]
 61%|██████    | 3705/6100 [5:35:46<3:36:01,  5.41s/it]
                                                       
{'loss': 0.0739, 'learning_rate': 0.0001, 'epoch': 6.07}

 61%|██████    | 3705/6100 [5:35:46<3:36:01,  5.41s/it]
 61%|██████    | 3706/6100 [5:35:51<3:35:51,  5.41s/it]
 61%|██████    | 3707/6100 [5:35:57<3:35:48,  5.41s/it]
 61%|██████    | 3708/6100 [5:36:02<3:35:45,  5.41s/it]
 61%|██████    | 3709/6100 [5:36:08<3:35:46,  5.41s/it]
 61%|██████    | 3710/6100 [5:36:13<3:35:40,  5.41s/it]
                                                       
{'loss': 0.0742, 'learning_rate': 0.0001, 'epoch': 6.08}

 61%|██████    | 3710/6100 [5:36:13<3:35:40,  5.41s/it]
 61%|██████    | 3711/6100 [5:36:18<3:35:42,  5.42s/it]
 61%|██████    | 3712/6100 [5:36:24<3:35:40,  5.42s/it]
 61%|██████    | 3713/6100 [5:36:29<3:35:35,  5.42s/it]
 61%|██████    | 3714/6100 [5:36:35<3:35:24,  5.42s/it]
 61%|██████    | 3715/6100 [5:36:40<3:35:14,  5.41s/it]
                                                       
{'loss': 0.069, 'learning_rate': 0.0001, 'epoch': 6.09}

 61%|██████    | 3715/6100 [5:36:40<3:35:14,  5.41s/it]
 61%|██████    | 3716/6100 [5:36:46<3:38:18,  5.49s/it]
 61%|██████    | 3717/6100 [5:36:51<3:37:07,  5.47s/it]
 61%|██████    | 3718/6100 [5:36:57<3:36:26,  5.45s/it]
 61%|██████    | 3719/6100 [5:37:02<3:35:55,  5.44s/it]
 61%|██████    | 3720/6100 [5:37:07<3:35:33,  5.43s/it]
                                                       
{'loss': 0.0732, 'learning_rate': 0.0001, 'epoch': 6.1}

 61%|██████    | 3720/6100 [5:37:07<3:35:33,  5.43s/it]
 61%|██████    | 3721/6100 [5:37:13<3:35:10,  5.43s/it]
 61%|██████    | 3722/6100 [5:37:18<3:35:01,  5.43s/it]
 61%|██████    | 3723/6100 [5:37:24<3:34:58,  5.43s/it]
 61%|██████    | 3724/6100 [5:37:29<3:34:46,  5.42s/it]
 61%|██████    | 3725/6100 [5:37:35<3:34:33,  5.42s/it]
                                                       
{'loss': 0.0699, 'learning_rate': 0.0001, 'epoch': 6.11}

 61%|██████    | 3725/6100 [5:37:35<3:34:33,  5.42s/it]
 61%|██████    | 3726/6100 [5:37:40<3:34:23,  5.42s/it]
 61%|██████    | 3727/6100 [5:37:45<3:34:10,  5.42s/it]
 61%|██████    | 3728/6100 [5:37:51<3:34:00,  5.41s/it]
 61%|██████    | 3729/6100 [5:37:56<3:33:59,  5.42s/it]
 61%|██████    | 3730/6100 [5:38:02<3:33:44,  5.41s/it]
                                                       
{'loss': 0.0704, 'learning_rate': 0.0001, 'epoch': 6.11}

 61%|██████    | 3730/6100 [5:38:02<3:33:44,  5.41s/it]
 61%|██████    | 3731/6100 [5:38:07<3:33:48,  5.42s/it]
 61%|██████    | 3732/6100 [5:38:12<3:34:17,  5.43s/it]
 61%|██████    | 3733/6100 [5:38:18<3:33:56,  5.42s/it]
 61%|██████    | 3734/6100 [5:38:23<3:33:47,  5.42s/it]
 61%|██████    | 3735/6100 [5:38:29<3:33:42,  5.42s/it]
                                                       
{'loss': 0.0755, 'learning_rate': 0.0001, 'epoch': 6.12}

 61%|██████    | 3735/6100 [5:38:29<3:33:42,  5.42s/it]
 61%|██████    | 3736/6100 [5:38:34<3:33:28,  5.42s/it]
 61%|██████▏   | 3737/6100 [5:38:40<3:33:17,  5.42s/it]
 61%|██████▏   | 3738/6100 [5:38:45<3:33:30,  5.42s/it]
 61%|██████▏   | 3739/6100 [5:38:50<3:33:24,  5.42s/it]
 61%|██████▏   | 3740/6100 [5:38:56<3:33:12,  5.42s/it]
                                                       
{'loss': 0.0731, 'learning_rate': 0.0001, 'epoch': 6.13}

 61%|██████▏   | 3740/6100 [5:38:56<3:33:12,  5.42s/it]
 61%|██████▏   | 3741/6100 [5:39:01<3:33:00,  5.42s/it]
 61%|██████▏   | 3742/6100 [5:39:07<3:32:48,  5.42s/it]
 61%|██████▏   | 3743/6100 [5:39:12<3:32:43,  5.42s/it]
 61%|██████▏   | 3744/6100 [5:39:17<3:32:35,  5.41s/it]
 61%|██████▏   | 3745/6100 [5:39:23<3:32:32,  5.41s/it]
                                                       
{'loss': 0.0693, 'learning_rate': 0.0001, 'epoch': 6.14}

 61%|██████▏   | 3745/6100 [5:39:23<3:32:32,  5.41s/it]
 61%|██████▏   | 3746/6100 [5:39:28<3:32:34,  5.42s/it]
 61%|██████▏   | 3747/6100 [5:39:34<3:32:23,  5.42s/it]
 61%|██████▏   | 3748/6100 [5:39:39<3:32:23,  5.42s/it]
 61%|██████▏   | 3749/6100 [5:39:45<3:32:14,  5.42s/it]
 61%|██████▏   | 3750/6100 [5:39:50<3:32:21,  5.42s/it]
                                                       
{'loss': 0.0735, 'learning_rate': 0.0001, 'epoch': 6.15}

 61%|██████▏   | 3750/6100 [5:39:50<3:32:21,  5.42s/it]Saving model checkpoint to ./results/checkpoint-3750
Configuration saved in ./results/checkpoint-3750/config.json
Model weights saved in ./results/checkpoint-3750/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-3750/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-3750/special_tokens_map.json
[2023-02-22 01:32:06,172] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step3753 is begin to save!
[2023-02-22 01:32:06,175] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-3750/global_step3753/mp_rank_00_model_states.pt
[2023-02-22 01:32:06,175] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-3750/global_step3753/mp_rank_00_model_states.pt...
[2023-02-22 01:32:06,988] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-3750/global_step3753/mp_rank_00_model_states.pt.
[2023-02-22 01:32:06,990] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-3750/global_step3753/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 01:32:07,161] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-3750/global_step3753/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 01:32:07,162] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-3750/global_step3753/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 01:32:07,162] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3753 is ready now!
Deleting older checkpoint [results/checkpoint-3600] due to args.save_total_limit

 61%|██████▏   | 3751/6100 [5:39:58<3:57:15,  6.06s/it]
 62%|██████▏   | 3752/6100 [5:40:03<3:49:32,  5.87s/it]
 62%|██████▏   | 3753/6100 [5:40:08<3:44:08,  5.73s/it]
 62%|██████▏   | 3754/6100 [5:40:14<3:40:17,  5.63s/it]
 62%|██████▏   | 3755/6100 [5:40:19<3:37:37,  5.57s/it]
                                                       
{'loss': 0.0719, 'learning_rate': 0.0001, 'epoch': 6.16}

 62%|██████▏   | 3755/6100 [5:40:19<3:37:37,  5.57s/it]
 62%|██████▏   | 3756/6100 [5:40:25<3:35:32,  5.52s/it]
 62%|██████▏   | 3757/6100 [5:40:30<3:34:13,  5.49s/it]
 62%|██████▏   | 3758/6100 [5:40:35<3:33:11,  5.46s/it]
 62%|██████▏   | 3759/6100 [5:40:41<3:32:36,  5.45s/it]
 62%|██████▏   | 3760/6100 [5:40:46<3:31:57,  5.43s/it]
                                                       
{'loss': 0.0709, 'learning_rate': 0.0001, 'epoch': 6.16}

 62%|██████▏   | 3760/6100 [5:40:46<3:31:57,  5.43s/it]
 62%|██████▏   | 3761/6100 [5:40:52<3:31:53,  5.44s/it]
 62%|██████▏   | 3762/6100 [5:40:57<3:31:20,  5.42s/it]
 62%|██████▏   | 3763/6100 [5:41:02<3:31:05,  5.42s/it]
 62%|██████▏   | 3764/6100 [5:41:08<3:31:11,  5.42s/it]
 62%|██████▏   | 3765/6100 [5:41:13<3:31:04,  5.42s/it]
                                                       
{'loss': 0.0723, 'learning_rate': 0.0001, 'epoch': 6.17}

 62%|██████▏   | 3765/6100 [5:41:13<3:31:04,  5.42s/it]
 62%|██████▏   | 3766/6100 [5:41:19<3:30:50,  5.42s/it]
 62%|██████▏   | 3767/6100 [5:41:24<3:30:41,  5.42s/it]
 62%|██████▏   | 3768/6100 [5:41:30<3:30:25,  5.41s/it]
 62%|██████▏   | 3769/6100 [5:41:35<3:30:17,  5.41s/it]
 62%|██████▏   | 3770/6100 [5:41:40<3:30:10,  5.41s/it]
                                                       
{'loss': 0.071, 'learning_rate': 0.0001, 'epoch': 6.18}

 62%|██████▏   | 3770/6100 [5:41:40<3:30:10,  5.41s/it]
 62%|██████▏   | 3771/6100 [5:41:46<3:30:14,  5.42s/it]
 62%|██████▏   | 3772/6100 [5:41:51<3:29:57,  5.41s/it]
 62%|██████▏   | 3773/6100 [5:41:57<3:29:51,  5.41s/it]
 62%|██████▏   | 3774/6100 [5:42:02<3:29:51,  5.41s/it]
 62%|██████▏   | 3775/6100 [5:42:07<3:29:39,  5.41s/it]
                                                       
{'loss': 0.076, 'learning_rate': 0.0001, 'epoch': 6.19}

 62%|██████▏   | 3775/6100 [5:42:07<3:29:39,  5.41s/it]
 62%|██████▏   | 3776/6100 [5:42:13<3:29:31,  5.41s/it]
 62%|██████▏   | 3777/6100 [5:42:18<3:29:22,  5.41s/it]
 62%|██████▏   | 3778/6100 [5:42:24<3:29:26,  5.41s/it]
 62%|██████▏   | 3779/6100 [5:42:29<3:29:26,  5.41s/it]
 62%|██████▏   | 3780/6100 [5:42:35<3:29:28,  5.42s/it]
                                                       
{'loss': 0.071, 'learning_rate': 0.0001, 'epoch': 6.2}

 62%|██████▏   | 3780/6100 [5:42:35<3:29:28,  5.42s/it]
 62%|██████▏   | 3781/6100 [5:42:40<3:29:18,  5.42s/it]
 62%|██████▏   | 3782/6100 [5:42:45<3:29:23,  5.42s/it]
 62%|██████▏   | 3783/6100 [5:42:51<3:29:11,  5.42s/it]
 62%|██████▏   | 3784/6100 [5:42:56<3:29:00,  5.41s/it]
 62%|██████▏   | 3785/6100 [5:43:02<3:28:58,  5.42s/it]
                                                       
{'loss': 0.0758, 'learning_rate': 0.0001, 'epoch': 6.2}

 62%|██████▏   | 3785/6100 [5:43:02<3:28:58,  5.42s/it]
 62%|██████▏   | 3786/6100 [5:43:07<3:28:55,  5.42s/it]
 62%|██████▏   | 3787/6100 [5:43:12<3:28:43,  5.41s/it]
 62%|██████▏   | 3788/6100 [5:43:18<3:28:41,  5.42s/it]
 62%|██████▏   | 3789/6100 [5:43:23<3:28:41,  5.42s/it]
 62%|██████▏   | 3790/6100 [5:43:29<3:30:23,  5.46s/it]
                                                       
{'loss': 0.0707, 'learning_rate': 0.0001, 'epoch': 6.21}

 62%|██████▏   | 3790/6100 [5:43:29<3:30:23,  5.46s/it]
 62%|██████▏   | 3791/6100 [5:43:34<3:29:43,  5.45s/it]
 62%|██████▏   | 3792/6100 [5:43:40<3:29:08,  5.44s/it]
 62%|██████▏   | 3793/6100 [5:43:45<3:28:48,  5.43s/it]
 62%|██████▏   | 3794/6100 [5:43:51<3:28:38,  5.43s/it]
 62%|██████▏   | 3795/6100 [5:43:56<3:28:18,  5.42s/it]
                                                       
{'loss': 0.0702, 'learning_rate': 0.0001, 'epoch': 6.22}

 62%|██████▏   | 3795/6100 [5:43:56<3:28:18,  5.42s/it]
 62%|██████▏   | 3796/6100 [5:44:01<3:28:04,  5.42s/it]
 62%|██████▏   | 3797/6100 [5:44:07<3:28:47,  5.44s/it]
 62%|██████▏   | 3798/6100 [5:44:12<3:28:30,  5.43s/it]
 62%|██████▏   | 3799/6100 [5:44:18<3:28:06,  5.43s/it]
 62%|██████▏   | 3800/6100 [5:44:23<3:27:57,  5.42s/it]
                                                       
{'loss': 0.077, 'learning_rate': 0.0001, 'epoch': 6.23}

 62%|██████▏   | 3800/6100 [5:44:23<3:27:57,  5.42s/it]
 62%|██████▏   | 3801/6100 [5:44:28<3:27:50,  5.42s/it]
 62%|██████▏   | 3802/6100 [5:44:34<3:27:53,  5.43s/it]
 62%|██████▏   | 3803/6100 [5:44:39<3:27:52,  5.43s/it]
 62%|██████▏   | 3804/6100 [5:44:45<3:27:40,  5.43s/it]
 62%|██████▏   | 3805/6100 [5:44:50<3:28:29,  5.45s/it]
                                                       
{'loss': 0.0781, 'learning_rate': 0.0001, 'epoch': 6.24}

 62%|██████▏   | 3805/6100 [5:44:50<3:28:29,  5.45s/it]
 62%|██████▏   | 3806/6100 [5:44:56<3:27:54,  5.44s/it]
 62%|██████▏   | 3807/6100 [5:45:01<3:27:23,  5.43s/it]
 62%|██████▏   | 3808/6100 [5:45:06<3:26:57,  5.42s/it]
 62%|██████▏   | 3809/6100 [5:45:12<3:26:54,  5.42s/it]
 62%|██████▏   | 3810/6100 [5:45:17<3:26:36,  5.41s/it]
                                                       
{'loss': 0.0714, 'learning_rate': 0.0001, 'epoch': 6.25}

 62%|██████▏   | 3810/6100 [5:45:17<3:26:36,  5.41s/it]
 62%|██████▏   | 3811/6100 [5:45:23<3:26:28,  5.41s/it]
 62%|██████▏   | 3812/6100 [5:45:28<3:26:34,  5.42s/it]
 63%|██████▎   | 3813/6100 [5:45:34<3:26:23,  5.41s/it]
 63%|██████▎   | 3814/6100 [5:45:39<3:26:18,  5.42s/it]
 63%|██████▎   | 3815/6100 [5:45:44<3:26:15,  5.42s/it]
                                                       
{'loss': 0.0709, 'learning_rate': 0.0001, 'epoch': 6.25}

 63%|██████▎   | 3815/6100 [5:45:44<3:26:15,  5.42s/it]
 63%|██████▎   | 3816/6100 [5:45:50<3:26:09,  5.42s/it]
 63%|██████▎   | 3817/6100 [5:45:55<3:26:06,  5.42s/it]
 63%|██████▎   | 3818/6100 [5:46:01<3:26:10,  5.42s/it]
 63%|██████▎   | 3819/6100 [5:46:06<3:25:59,  5.42s/it]
 63%|██████▎   | 3820/6100 [5:46:11<3:25:42,  5.41s/it]
                                                       
{'loss': 0.0773, 'learning_rate': 0.0001, 'epoch': 6.26}

 63%|██████▎   | 3820/6100 [5:46:11<3:25:42,  5.41s/it]
 63%|██████▎   | 3821/6100 [5:46:17<3:25:50,  5.42s/it]
 63%|██████▎   | 3822/6100 [5:46:22<3:25:48,  5.42s/it]
 63%|██████▎   | 3823/6100 [5:46:28<3:25:45,  5.42s/it]
 63%|██████▎   | 3824/6100 [5:46:33<3:25:37,  5.42s/it]
 63%|██████▎   | 3825/6100 [5:46:39<3:25:13,  5.41s/it]
                                                       
{'loss': 0.069, 'learning_rate': 0.0001, 'epoch': 6.27}

 63%|██████▎   | 3825/6100 [5:46:39<3:25:13,  5.41s/it]
 63%|██████▎   | 3826/6100 [5:46:44<3:25:06,  5.41s/it]
 63%|██████▎   | 3827/6100 [5:46:49<3:24:57,  5.41s/it]
 63%|██████▎   | 3828/6100 [5:46:55<3:24:53,  5.41s/it]
 63%|██████▎   | 3829/6100 [5:47:00<3:27:04,  5.47s/it]
 63%|██████▎   | 3830/6100 [5:47:06<3:26:17,  5.45s/it]
                                                       
{'loss': 0.0816, 'learning_rate': 0.0001, 'epoch': 6.28}

 63%|██████▎   | 3830/6100 [5:47:06<3:26:17,  5.45s/it]
 63%|██████▎   | 3831/6100 [5:47:11<3:25:45,  5.44s/it]
 63%|██████▎   | 3832/6100 [5:47:17<3:25:22,  5.43s/it]
 63%|██████▎   | 3833/6100 [5:47:22<3:25:01,  5.43s/it]
 63%|██████▎   | 3834/6100 [5:47:27<3:24:40,  5.42s/it]
 63%|██████▎   | 3835/6100 [5:47:33<3:24:32,  5.42s/it]
                                                       
{'loss': 0.0677, 'learning_rate': 0.0001, 'epoch': 6.29}

 63%|██████▎   | 3835/6100 [5:47:33<3:24:32,  5.42s/it]
 63%|██████▎   | 3836/6100 [5:47:38<3:24:23,  5.42s/it]
 63%|██████▎   | 3837/6100 [5:47:44<3:24:05,  5.41s/it]
 63%|██████▎   | 3838/6100 [5:47:49<3:24:01,  5.41s/it]
 63%|██████▎   | 3839/6100 [5:47:55<3:23:50,  5.41s/it]
 63%|██████▎   | 3840/6100 [5:48:00<3:23:52,  5.41s/it]
                                                       
{'loss': 0.0735, 'learning_rate': 0.0001, 'epoch': 6.29}

 63%|██████▎   | 3840/6100 [5:48:00<3:23:52,  5.41s/it]
 63%|██████▎   | 3841/6100 [5:48:05<3:23:50,  5.41s/it]
 63%|██████▎   | 3842/6100 [5:48:11<3:23:44,  5.41s/it]
 63%|██████▎   | 3843/6100 [5:48:16<3:23:29,  5.41s/it]
 63%|██████▎   | 3844/6100 [5:48:22<3:23:21,  5.41s/it]
 63%|██████▎   | 3845/6100 [5:48:27<3:23:13,  5.41s/it]
                                                       
{'loss': 0.0712, 'learning_rate': 0.0001, 'epoch': 6.3}

 63%|██████▎   | 3845/6100 [5:48:27<3:23:13,  5.41s/it]
 63%|██████▎   | 3846/6100 [5:48:32<3:23:23,  5.41s/it]
 63%|██████▎   | 3847/6100 [5:48:38<3:23:16,  5.41s/it]
 63%|██████▎   | 3848/6100 [5:48:43<3:23:15,  5.42s/it]
 63%|██████▎   | 3849/6100 [5:48:49<3:23:11,  5.42s/it]
 63%|██████▎   | 3850/6100 [5:48:54<3:23:12,  5.42s/it]
                                                       
{'loss': 0.0706, 'learning_rate': 0.0001, 'epoch': 6.31}

 63%|██████▎   | 3850/6100 [5:48:54<3:23:12,  5.42s/it]
 63%|██████▎   | 3851/6100 [5:49:00<3:23:21,  5.43s/it]
 63%|██████▎   | 3852/6100 [5:49:05<3:23:07,  5.42s/it]
 63%|██████▎   | 3853/6100 [5:49:10<3:22:56,  5.42s/it]
 63%|██████▎   | 3854/6100 [5:49:16<3:22:48,  5.42s/it]
 63%|██████▎   | 3855/6100 [5:49:21<3:22:44,  5.42s/it]
                                                       
{'loss': 0.0705, 'learning_rate': 0.0001, 'epoch': 6.32}

 63%|██████▎   | 3855/6100 [5:49:21<3:22:44,  5.42s/it]
 63%|██████▎   | 3856/6100 [5:49:27<3:22:34,  5.42s/it]
 63%|██████▎   | 3857/6100 [5:49:32<3:22:21,  5.41s/it]
 63%|██████▎   | 3858/6100 [5:49:37<3:22:18,  5.41s/it]
 63%|██████▎   | 3859/6100 [5:49:43<3:22:14,  5.41s/it]
 63%|██████▎   | 3860/6100 [5:49:48<3:22:07,  5.41s/it]
                                                       
{'loss': 0.0739, 'learning_rate': 0.0001, 'epoch': 6.33}

 63%|██████▎   | 3860/6100 [5:49:48<3:22:07,  5.41s/it]
 63%|██████▎   | 3861/6100 [5:49:54<3:22:04,  5.41s/it]
 63%|██████▎   | 3862/6100 [5:49:59<3:21:57,  5.41s/it]
 63%|██████▎   | 3863/6100 [5:50:04<3:22:00,  5.42s/it]
 63%|██████▎   | 3864/6100 [5:50:10<3:21:48,  5.42s/it]
 63%|██████▎   | 3865/6100 [5:50:15<3:21:43,  5.42s/it]
                                                       
{'loss': 0.071, 'learning_rate': 0.0001, 'epoch': 6.34}

 63%|██████▎   | 3865/6100 [5:50:15<3:21:43,  5.42s/it]
 63%|██████▎   | 3866/6100 [5:50:21<3:21:38,  5.42s/it]
 63%|██████▎   | 3867/6100 [5:50:26<3:21:30,  5.41s/it]
 63%|██████▎   | 3868/6100 [5:50:32<3:21:22,  5.41s/it]
 63%|██████▎   | 3869/6100 [5:50:37<3:21:14,  5.41s/it]
 63%|██████▎   | 3870/6100 [5:50:42<3:21:10,  5.41s/it]
                                                       
{'loss': 0.0737, 'learning_rate': 0.0001, 'epoch': 6.34}

 63%|██████▎   | 3870/6100 [5:50:42<3:21:10,  5.41s/it]
 63%|██████▎   | 3871/6100 [5:50:48<3:21:02,  5.41s/it]
 63%|██████▎   | 3872/6100 [5:50:53<3:20:56,  5.41s/it]
 63%|██████▎   | 3873/6100 [5:50:59<3:20:53,  5.41s/it]
 64%|██████▎   | 3874/6100 [5:51:04<3:20:47,  5.41s/it]
 64%|██████▎   | 3875/6100 [5:51:09<3:20:38,  5.41s/it]
                                                       
{'loss': 0.0745, 'learning_rate': 0.0001, 'epoch': 6.35}

 64%|██████▎   | 3875/6100 [5:51:09<3:20:38,  5.41s/it]
 64%|██████▎   | 3876/6100 [5:51:15<3:20:45,  5.42s/it]
 64%|██████▎   | 3877/6100 [5:51:20<3:20:48,  5.42s/it]
 64%|██████▎   | 3878/6100 [5:51:26<3:20:35,  5.42s/it]
 64%|██████▎   | 3879/6100 [5:51:31<3:20:37,  5.42s/it]
 64%|██████▎   | 3880/6100 [5:51:37<3:20:53,  5.43s/it]
                                                       
{'loss': 0.0763, 'learning_rate': 0.0001, 'epoch': 6.36}

 64%|██████▎   | 3880/6100 [5:51:37<3:20:53,  5.43s/it]
 64%|██████▎   | 3881/6100 [5:51:42<3:20:36,  5.42s/it]
 64%|██████▎   | 3882/6100 [5:51:47<3:20:22,  5.42s/it]
 64%|██████▎   | 3883/6100 [5:51:53<3:20:11,  5.42s/it]
 64%|██████▎   | 3884/6100 [5:51:58<3:20:07,  5.42s/it]
 64%|██████▎   | 3885/6100 [5:52:04<3:20:02,  5.42s/it]
                                                       
{'loss': 0.0707, 'learning_rate': 0.0001, 'epoch': 6.37}

 64%|██████▎   | 3885/6100 [5:52:04<3:20:02,  5.42s/it]
 64%|██████▎   | 3886/6100 [5:52:09<3:19:55,  5.42s/it]
 64%|██████▎   | 3887/6100 [5:52:15<3:19:59,  5.42s/it]
 64%|██████▎   | 3888/6100 [5:52:20<3:19:57,  5.42s/it]
 64%|██████▍   | 3889/6100 [5:52:25<3:19:37,  5.42s/it]
 64%|██████▍   | 3890/6100 [5:52:31<3:19:33,  5.42s/it]
                                                       
{'loss': 0.0719, 'learning_rate': 0.0001, 'epoch': 6.38}

 64%|██████▍   | 3890/6100 [5:52:31<3:19:33,  5.42s/it]
 64%|██████▍   | 3891/6100 [5:52:36<3:19:25,  5.42s/it]
 64%|██████▍   | 3892/6100 [5:52:42<3:19:19,  5.42s/it]
 64%|██████▍   | 3893/6100 [5:52:47<3:19:13,  5.42s/it]
 64%|██████▍   | 3894/6100 [5:52:52<3:19:03,  5.41s/it]
 64%|██████▍   | 3895/6100 [5:52:58<3:18:55,  5.41s/it]
                                                       
{'loss': 0.0721, 'learning_rate': 0.0001, 'epoch': 6.38}

 64%|██████▍   | 3895/6100 [5:52:58<3:18:55,  5.41s/it]
 64%|██████▍   | 3896/6100 [5:53:03<3:18:57,  5.42s/it]
 64%|██████▍   | 3897/6100 [5:53:09<3:19:01,  5.42s/it]
 64%|██████▍   | 3898/6100 [5:53:14<3:18:52,  5.42s/it]
 64%|██████▍   | 3899/6100 [5:53:19<3:18:40,  5.42s/it]
 64%|██████▍   | 3900/6100 [5:53:25<3:18:36,  5.42s/it]
                                                       
{'loss': 0.0739, 'learning_rate': 0.0001, 'epoch': 6.39}

 64%|██████▍   | 3900/6100 [5:53:25<3:18:36,  5.42s/it]Saving model checkpoint to ./results/checkpoint-3900
Configuration saved in ./results/checkpoint-3900/config.json
Model weights saved in ./results/checkpoint-3900/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-3900/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-3900/special_tokens_map.json
[2023-02-22 01:45:41,098] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step3903 is begin to save!
[2023-02-22 01:45:41,101] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-3900/global_step3903/mp_rank_00_model_states.pt
[2023-02-22 01:45:41,101] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-3900/global_step3903/mp_rank_00_model_states.pt...
[2023-02-22 01:45:41,923] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-3900/global_step3903/mp_rank_00_model_states.pt.
[2023-02-22 01:45:41,925] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-3900/global_step3903/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 01:45:42,097] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-3900/global_step3903/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 01:45:42,098] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-3900/global_step3903/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 01:45:42,098] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3903 is ready now!
Deleting older checkpoint [results/checkpoint-3750] due to args.save_total_limit

 64%|██████▍   | 3901/6100 [5:53:32<3:42:20,  6.07s/it]
 64%|██████▍   | 3902/6100 [5:53:38<3:35:03,  5.87s/it]
 64%|██████▍   | 3903/6100 [5:53:43<3:29:46,  5.73s/it]
 64%|██████▍   | 3904/6100 [5:53:49<3:26:13,  5.63s/it]
 64%|██████▍   | 3905/6100 [5:53:54<3:23:49,  5.57s/it]
                                                       
{'loss': 0.0712, 'learning_rate': 0.0001, 'epoch': 6.4}

 64%|██████▍   | 3905/6100 [5:53:54<3:23:49,  5.57s/it]
 64%|██████▍   | 3906/6100 [5:54:00<3:21:59,  5.52s/it]
 64%|██████▍   | 3907/6100 [5:54:05<3:20:47,  5.49s/it]
 64%|██████▍   | 3908/6100 [5:54:10<3:19:49,  5.47s/it]
 64%|██████▍   | 3909/6100 [5:54:16<3:19:04,  5.45s/it]
 64%|██████▍   | 3910/6100 [5:54:21<3:18:37,  5.44s/it]
                                                       
{'loss': 0.072, 'learning_rate': 0.0001, 'epoch': 6.41}

 64%|██████▍   | 3910/6100 [5:54:21<3:18:37,  5.44s/it]
 64%|██████▍   | 3911/6100 [5:54:27<3:18:27,  5.44s/it]
 64%|██████▍   | 3912/6100 [5:54:32<3:18:05,  5.43s/it]
 64%|██████▍   | 3913/6100 [5:54:37<3:17:50,  5.43s/it]
 64%|██████▍   | 3914/6100 [5:54:43<3:17:32,  5.42s/it]
 64%|██████▍   | 3915/6100 [5:54:48<3:17:24,  5.42s/it]
                                                       
{'loss': 0.0714, 'learning_rate': 0.0001, 'epoch': 6.42}

 64%|██████▍   | 3915/6100 [5:54:48<3:17:24,  5.42s/it]
 64%|██████▍   | 3916/6100 [5:54:54<3:17:09,  5.42s/it]
 64%|██████▍   | 3917/6100 [5:54:59<3:16:58,  5.41s/it]
 64%|██████▍   | 3918/6100 [5:55:05<3:16:55,  5.42s/it]
 64%|██████▍   | 3919/6100 [5:55:10<3:16:55,  5.42s/it]
 64%|██████▍   | 3920/6100 [5:55:15<3:16:51,  5.42s/it]
                                                       
{'loss': 0.0721, 'learning_rate': 0.0001, 'epoch': 6.43}

 64%|██████▍   | 3920/6100 [5:55:15<3:16:51,  5.42s/it]
 64%|██████▍   | 3921/6100 [5:55:21<3:16:53,  5.42s/it]
 64%|██████▍   | 3922/6100 [5:55:26<3:16:48,  5.42s/it]
 64%|██████▍   | 3923/6100 [5:55:32<3:16:42,  5.42s/it]
 64%|██████▍   | 3924/6100 [5:55:37<3:16:39,  5.42s/it]
 64%|██████▍   | 3925/6100 [5:55:43<3:16:33,  5.42s/it]
                                                       
{'loss': 0.0701, 'learning_rate': 0.0001, 'epoch': 6.43}

 64%|██████▍   | 3925/6100 [5:55:43<3:16:33,  5.42s/it]
 64%|██████▍   | 3926/6100 [5:55:48<3:16:16,  5.42s/it]
 64%|██████▍   | 3927/6100 [5:55:53<3:16:08,  5.42s/it]
 64%|██████▍   | 3928/6100 [5:55:59<3:16:02,  5.42s/it]
 64%|██████▍   | 3929/6100 [5:56:04<3:16:00,  5.42s/it]
 64%|██████▍   | 3930/6100 [5:56:10<3:15:53,  5.42s/it]
                                                       
{'loss': 0.0696, 'learning_rate': 0.0001, 'epoch': 6.44}

 64%|██████▍   | 3930/6100 [5:56:10<3:15:53,  5.42s/it]
 64%|██████▍   | 3931/6100 [5:56:15<3:15:44,  5.41s/it]
 64%|██████▍   | 3932/6100 [5:56:20<3:15:40,  5.42s/it]
 64%|██████▍   | 3933/6100 [5:56:26<3:15:31,  5.41s/it]
 64%|██████▍   | 3934/6100 [5:56:31<3:15:40,  5.42s/it]
 65%|██████▍   | 3935/6100 [5:56:37<3:15:37,  5.42s/it]
                                                       
{'loss': 0.0736, 'learning_rate': 0.0001, 'epoch': 6.45}

 65%|██████▍   | 3935/6100 [5:56:37<3:15:37,  5.42s/it]
 65%|██████▍   | 3936/6100 [5:56:42<3:15:25,  5.42s/it]
 65%|██████▍   | 3937/6100 [5:56:48<3:15:16,  5.42s/it]
 65%|██████▍   | 3938/6100 [5:56:53<3:15:07,  5.41s/it]
 65%|██████▍   | 3939/6100 [5:56:58<3:15:01,  5.41s/it]
 65%|██████▍   | 3940/6100 [5:57:04<3:14:56,  5.41s/it]
                                                       
{'loss': 0.0692, 'learning_rate': 0.0001, 'epoch': 6.46}

 65%|██████▍   | 3940/6100 [5:57:04<3:14:56,  5.41s/it]
 65%|██████▍   | 3941/6100 [5:57:09<3:14:53,  5.42s/it]
 65%|██████▍   | 3942/6100 [5:57:15<3:14:45,  5.41s/it]
 65%|██████▍   | 3943/6100 [5:57:20<3:14:41,  5.42s/it]
 65%|██████▍   | 3944/6100 [5:57:25<3:14:45,  5.42s/it]
 65%|██████▍   | 3945/6100 [5:57:31<3:14:59,  5.43s/it]
                                                       
{'loss': 0.0725, 'learning_rate': 0.0001, 'epoch': 6.47}

 65%|██████▍   | 3945/6100 [5:57:31<3:14:59,  5.43s/it]
 65%|██████▍   | 3946/6100 [5:57:36<3:14:55,  5.43s/it]
 65%|██████▍   | 3947/6100 [5:57:42<3:14:36,  5.42s/it]
 65%|██████▍   | 3948/6100 [5:57:47<3:14:28,  5.42s/it]
 65%|██████▍   | 3949/6100 [5:57:53<3:14:23,  5.42s/it]
 65%|██████▍   | 3950/6100 [5:57:58<3:14:18,  5.42s/it]
                                                       
{'loss': 0.0724, 'learning_rate': 0.0001, 'epoch': 6.48}

 65%|██████▍   | 3950/6100 [5:57:58<3:14:18,  5.42s/it]
 65%|██████▍   | 3951/6100 [5:58:03<3:14:15,  5.42s/it]
 65%|██████▍   | 3952/6100 [5:58:09<3:14:09,  5.42s/it]
 65%|██████▍   | 3953/6100 [5:58:14<3:13:52,  5.42s/it]
 65%|██████▍   | 3954/6100 [5:58:20<3:13:58,  5.42s/it]
 65%|██████▍   | 3955/6100 [5:58:25<3:13:50,  5.42s/it]
                                                       
{'loss': 0.0744, 'learning_rate': 0.0001, 'epoch': 6.48}

 65%|██████▍   | 3955/6100 [5:58:25<3:13:50,  5.42s/it]
 65%|██████▍   | 3956/6100 [5:58:31<3:13:46,  5.42s/it]
 65%|██████▍   | 3957/6100 [5:58:36<3:13:41,  5.42s/it]
 65%|██████▍   | 3958/6100 [5:58:41<3:13:25,  5.42s/it]
 65%|██████▍   | 3959/6100 [5:58:47<3:13:20,  5.42s/it]
 65%|██████▍   | 3960/6100 [5:58:52<3:13:29,  5.42s/it]
                                                       
{'loss': 0.0738, 'learning_rate': 0.0001, 'epoch': 6.49}

 65%|██████▍   | 3960/6100 [5:58:52<3:13:29,  5.42s/it]
 65%|██████▍   | 3961/6100 [5:58:58<3:13:39,  5.43s/it]
 65%|██████▍   | 3962/6100 [5:59:03<3:13:28,  5.43s/it]
 65%|██████▍   | 3963/6100 [5:59:08<3:13:17,  5.43s/it]
 65%|██████▍   | 3964/6100 [5:59:14<3:13:01,  5.42s/it]
 65%|██████▌   | 3965/6100 [5:59:19<3:13:02,  5.42s/it]
                                                       
{'loss': 0.0765, 'learning_rate': 0.0001, 'epoch': 6.5}

 65%|██████▌   | 3965/6100 [5:59:19<3:13:02,  5.42s/it]
 65%|██████▌   | 3966/6100 [5:59:25<3:15:51,  5.51s/it]
 65%|██████▌   | 3967/6100 [5:59:30<3:15:14,  5.49s/it]
 65%|██████▌   | 3968/6100 [5:59:36<3:14:25,  5.47s/it]
 65%|██████▌   | 3969/6100 [5:59:41<3:13:41,  5.45s/it]
 65%|██████▌   | 3970/6100 [5:59:47<3:13:11,  5.44s/it]
                                                       
{'loss': 0.0735, 'learning_rate': 0.0001, 'epoch': 6.51}

 65%|██████▌   | 3970/6100 [5:59:47<3:13:11,  5.44s/it]
 65%|██████▌   | 3971/6100 [5:59:52<3:12:52,  5.44s/it]
 65%|██████▌   | 3972/6100 [5:59:58<3:12:42,  5.43s/it]
 65%|██████▌   | 3973/6100 [6:00:03<3:12:19,  5.43s/it]
 65%|██████▌   | 3974/6100 [6:00:08<3:12:05,  5.42s/it]
 65%|██████▌   | 3975/6100 [6:00:14<3:12:06,  5.42s/it]
                                                       
{'loss': 0.0727, 'learning_rate': 0.0001, 'epoch': 6.52}

 65%|██████▌   | 3975/6100 [6:00:14<3:12:06,  5.42s/it]
 65%|██████▌   | 3976/6100 [6:00:19<3:12:02,  5.43s/it]
 65%|██████▌   | 3977/6100 [6:00:25<3:11:47,  5.42s/it]
 65%|██████▌   | 3978/6100 [6:00:30<3:11:28,  5.41s/it]
 65%|██████▌   | 3979/6100 [6:00:35<3:11:18,  5.41s/it]
 65%|██████▌   | 3980/6100 [6:00:41<3:11:23,  5.42s/it]
                                                       
{'loss': 0.0783, 'learning_rate': 0.0001, 'epoch': 6.52}

 65%|██████▌   | 3980/6100 [6:00:41<3:11:23,  5.42s/it]
 65%|██████▌   | 3981/6100 [6:00:46<3:11:24,  5.42s/it]
 65%|██████▌   | 3982/6100 [6:00:52<3:11:17,  5.42s/it]
 65%|██████▌   | 3983/6100 [6:00:57<3:11:18,  5.42s/it]
 65%|██████▌   | 3984/6100 [6:01:03<3:11:09,  5.42s/it]
 65%|██████▌   | 3985/6100 [6:01:08<3:11:07,  5.42s/it]
                                                       
{'loss': 0.0717, 'learning_rate': 0.0001, 'epoch': 6.53}

 65%|██████▌   | 3985/6100 [6:01:08<3:11:07,  5.42s/it]
 65%|██████▌   | 3986/6100 [6:01:13<3:11:04,  5.42s/it]
 65%|██████▌   | 3987/6100 [6:01:19<3:11:02,  5.42s/it]
 65%|██████▌   | 3988/6100 [6:01:24<3:11:09,  5.43s/it]
 65%|██████▌   | 3989/6100 [6:01:30<3:10:55,  5.43s/it]
 65%|██████▌   | 3990/6100 [6:01:35<3:10:53,  5.43s/it]
                                                       
{'loss': 0.0748, 'learning_rate': 0.0001, 'epoch': 6.54}

 65%|██████▌   | 3990/6100 [6:01:35<3:10:53,  5.43s/it]
 65%|██████▌   | 3991/6100 [6:01:41<3:10:48,  5.43s/it]
 65%|██████▌   | 3992/6100 [6:01:46<3:10:48,  5.43s/it]
 65%|██████▌   | 3993/6100 [6:01:51<3:10:43,  5.43s/it]
 65%|██████▌   | 3994/6100 [6:01:57<3:10:28,  5.43s/it]
 65%|██████▌   | 3995/6100 [6:02:02<3:10:11,  5.42s/it]
                                                       
{'loss': 0.075, 'learning_rate': 0.0001, 'epoch': 6.55}

 65%|██████▌   | 3995/6100 [6:02:02<3:10:11,  5.42s/it]
 66%|██████▌   | 3996/6100 [6:02:08<3:10:09,  5.42s/it][2023-02-22 01:54:28,320] [INFO] [logging.py:75:log_dist] [Rank 0] step=4000, skipped=0, lr=[0.0001], mom=[[0.9, 0.999]]
[2023-02-22 01:54:28,387] [INFO] [timer.py:198:stop] epoch=0/micro_step=16000/global_step=4000, RunningAvgSamplesPerSec=35.874203992699876, CurrSamplesPerSec=35.90061977451843, MemAllocated=0.66GB, MaxMemAllocated=36.72GB

 66%|██████▌   | 3997/6100 [6:02:13<3:10:03,  5.42s/it]
 66%|██████▌   | 3998/6100 [6:02:19<3:09:48,  5.42s/it]
 66%|██████▌   | 3999/6100 [6:02:24<3:09:53,  5.42s/it]
 66%|██████▌   | 4000/6100 [6:02:29<3:09:52,  5.42s/it]
                                                       
{'loss': 0.0699, 'learning_rate': 0.0001, 'epoch': 6.56}

 66%|██████▌   | 4000/6100 [6:02:29<3:09:52,  5.42s/it]
 66%|██████▌   | 4001/6100 [6:02:35<3:09:44,  5.42s/it]
 66%|██████▌   | 4002/6100 [6:02:40<3:09:24,  5.42s/it]
 66%|██████▌   | 4003/6100 [6:02:46<3:09:23,  5.42s/it]
 66%|██████▌   | 4004/6100 [6:02:51<3:09:16,  5.42s/it]
 66%|██████▌   | 4005/6100 [6:02:57<3:09:12,  5.42s/it]
                                                       
{'loss': 0.0683, 'learning_rate': 0.0001, 'epoch': 6.57}

 66%|██████▌   | 4005/6100 [6:02:57<3:09:12,  5.42s/it]
 66%|██████▌   | 4006/6100 [6:03:02<3:09:15,  5.42s/it]
 66%|██████▌   | 4007/6100 [6:03:07<3:09:25,  5.43s/it]
 66%|██████▌   | 4008/6100 [6:03:13<3:09:07,  5.42s/it]
 66%|██████▌   | 4009/6100 [6:03:18<3:09:08,  5.43s/it]
 66%|██████▌   | 4010/6100 [6:03:24<3:10:05,  5.46s/it]
                                                       
{'loss': 0.0742, 'learning_rate': 0.0001, 'epoch': 6.57}

 66%|██████▌   | 4010/6100 [6:03:24<3:10:05,  5.46s/it]
 66%|██████▌   | 4011/6100 [6:03:29<3:09:32,  5.44s/it]
 66%|██████▌   | 4012/6100 [6:03:35<3:09:15,  5.44s/it]
 66%|██████▌   | 4013/6100 [6:03:40<3:09:00,  5.43s/it]
 66%|██████▌   | 4014/6100 [6:03:45<3:08:48,  5.43s/it]
 66%|██████▌   | 4015/6100 [6:03:51<3:08:45,  5.43s/it]
                                                       
{'loss': 0.0726, 'learning_rate': 0.0001, 'epoch': 6.58}

 66%|██████▌   | 4015/6100 [6:03:51<3:08:45,  5.43s/it]
 66%|██████▌   | 4016/6100 [6:03:56<3:08:36,  5.43s/it]
 66%|██████▌   | 4017/6100 [6:04:02<3:08:21,  5.43s/it]
 66%|██████▌   | 4018/6100 [6:04:07<3:08:04,  5.42s/it]
 66%|██████▌   | 4019/6100 [6:04:13<3:08:59,  5.45s/it]
 66%|██████▌   | 4020/6100 [6:04:18<3:08:31,  5.44s/it]
                                                       
{'loss': 0.0719, 'learning_rate': 0.0001, 'epoch': 6.59}

 66%|██████▌   | 4020/6100 [6:04:18<3:08:31,  5.44s/it]
 66%|██████▌   | 4021/6100 [6:04:23<3:08:13,  5.43s/it]
 66%|██████▌   | 4022/6100 [6:04:29<3:08:13,  5.43s/it]
 66%|██████▌   | 4023/6100 [6:04:34<3:08:01,  5.43s/it]
 66%|██████▌   | 4024/6100 [6:04:40<3:07:51,  5.43s/it]
 66%|██████▌   | 4025/6100 [6:04:45<3:08:03,  5.44s/it]
                                                       
{'loss': 0.0714, 'learning_rate': 0.0001, 'epoch': 6.6}

 66%|██████▌   | 4025/6100 [6:04:45<3:08:03,  5.44s/it]
 66%|██████▌   | 4026/6100 [6:04:51<3:07:53,  5.44s/it]
 66%|██████▌   | 4027/6100 [6:04:56<3:08:14,  5.45s/it]
 66%|██████▌   | 4028/6100 [6:05:02<3:07:54,  5.44s/it]
 66%|██████▌   | 4029/6100 [6:05:07<3:07:40,  5.44s/it]
 66%|██████▌   | 4030/6100 [6:05:12<3:07:38,  5.44s/it]
                                                       
{'loss': 0.0715, 'learning_rate': 0.0001, 'epoch': 6.61}

 66%|██████▌   | 4030/6100 [6:05:12<3:07:38,  5.44s/it]
 66%|██████▌   | 4031/6100 [6:05:18<3:07:22,  5.43s/it]
 66%|██████▌   | 4032/6100 [6:05:23<3:07:15,  5.43s/it]
 66%|██████▌   | 4033/6100 [6:05:29<3:07:00,  5.43s/it]
 66%|██████▌   | 4034/6100 [6:05:34<3:06:49,  5.43s/it]
 66%|██████▌   | 4035/6100 [6:05:40<3:06:40,  5.42s/it]
                                                       
{'loss': 0.074, 'learning_rate': 0.0001, 'epoch': 6.61}

 66%|██████▌   | 4035/6100 [6:05:40<3:06:40,  5.42s/it]
 66%|██████▌   | 4036/6100 [6:05:45<3:06:46,  5.43s/it]
 66%|██████▌   | 4037/6100 [6:05:50<3:06:28,  5.42s/it]
 66%|██████▌   | 4038/6100 [6:05:56<3:06:18,  5.42s/it]
 66%|██████▌   | 4039/6100 [6:06:01<3:06:15,  5.42s/it]
 66%|██████▌   | 4040/6100 [6:06:07<3:06:06,  5.42s/it]
                                                       
{'loss': 0.0691, 'learning_rate': 0.0001, 'epoch': 6.62}

 66%|██████▌   | 4040/6100 [6:06:07<3:06:06,  5.42s/it]
 66%|██████▌   | 4041/6100 [6:06:12<3:06:07,  5.42s/it]
 66%|██████▋   | 4042/6100 [6:06:18<3:06:06,  5.43s/it]
 66%|██████▋   | 4043/6100 [6:06:23<3:06:01,  5.43s/it]
 66%|██████▋   | 4044/6100 [6:06:28<3:05:47,  5.42s/it]
 66%|██████▋   | 4045/6100 [6:06:34<3:05:41,  5.42s/it]
                                                       
{'loss': 0.073, 'learning_rate': 0.0001, 'epoch': 6.63}

 66%|██████▋   | 4045/6100 [6:06:34<3:05:41,  5.42s/it]
 66%|██████▋   | 4046/6100 [6:06:39<3:05:34,  5.42s/it]
 66%|██████▋   | 4047/6100 [6:06:45<3:05:31,  5.42s/it]
 66%|██████▋   | 4048/6100 [6:06:50<3:05:20,  5.42s/it]
 66%|██████▋   | 4049/6100 [6:06:55<3:05:22,  5.42s/it]
 66%|██████▋   | 4050/6100 [6:07:01<3:05:22,  5.43s/it]
                                                       
{'loss': 0.067, 'learning_rate': 0.0001, 'epoch': 6.64}

 66%|██████▋   | 4050/6100 [6:07:01<3:05:22,  5.43s/it]Saving model checkpoint to ./results/checkpoint-4050
Configuration saved in ./results/checkpoint-4050/config.json
Model weights saved in ./results/checkpoint-4050/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-4050/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-4050/special_tokens_map.json
[2023-02-22 01:59:17,065] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step4053 is begin to save!
[2023-02-22 01:59:17,070] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-4050/global_step4053/mp_rank_00_model_states.pt
[2023-02-22 01:59:17,070] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-4050/global_step4053/mp_rank_00_model_states.pt...
[2023-02-22 01:59:17,891] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-4050/global_step4053/mp_rank_00_model_states.pt.
[2023-02-22 01:59:17,893] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-4050/global_step4053/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 01:59:18,066] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-4050/global_step4053/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 01:59:18,067] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-4050/global_step4053/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 01:59:18,067] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4053 is ready now!
Deleting older checkpoint [results/checkpoint-3900] due to args.save_total_limit

 66%|██████▋   | 4051/6100 [6:07:08<3:27:27,  6.07s/it]
 66%|██████▋   | 4052/6100 [6:07:14<3:20:38,  5.88s/it]
 66%|██████▋   | 4053/6100 [6:07:19<3:15:58,  5.74s/it]
 66%|██████▋   | 4054/6100 [6:07:25<3:12:34,  5.65s/it]
 66%|██████▋   | 4055/6100 [6:07:30<3:10:13,  5.58s/it]
                                                       
{'loss': 0.0706, 'learning_rate': 0.0001, 'epoch': 6.65}

 66%|██████▋   | 4055/6100 [6:07:30<3:10:13,  5.58s/it]
 66%|██████▋   | 4056/6100 [6:07:36<3:08:29,  5.53s/it]
 67%|██████▋   | 4057/6100 [6:07:41<3:07:22,  5.50s/it]
 67%|██████▋   | 4058/6100 [6:07:46<3:06:21,  5.48s/it]
 67%|██████▋   | 4059/6100 [6:07:52<3:05:42,  5.46s/it]
 67%|██████▋   | 4060/6100 [6:07:57<3:05:19,  5.45s/it]
                                                       
{'loss': 0.0687, 'learning_rate': 0.0001, 'epoch': 6.66}

 67%|██████▋   | 4060/6100 [6:07:57<3:05:19,  5.45s/it]
 67%|██████▋   | 4061/6100 [6:08:03<3:04:56,  5.44s/it]
 67%|██████▋   | 4062/6100 [6:08:08<3:04:32,  5.43s/it]
 67%|██████▋   | 4063/6100 [6:08:14<3:04:21,  5.43s/it]
 67%|██████▋   | 4064/6100 [6:08:19<3:04:23,  5.43s/it]
 67%|██████▋   | 4065/6100 [6:08:24<3:04:20,  5.43s/it]
                                                       
{'loss': 0.0726, 'learning_rate': 0.0001, 'epoch': 6.66}

 67%|██████▋   | 4065/6100 [6:08:24<3:04:20,  5.43s/it]
 67%|██████▋   | 4066/6100 [6:08:30<3:03:58,  5.43s/it]
 67%|██████▋   | 4067/6100 [6:08:35<3:03:51,  5.43s/it]
 67%|██████▋   | 4068/6100 [6:08:41<3:03:41,  5.42s/it]
 67%|██████▋   | 4069/6100 [6:08:46<3:03:56,  5.43s/it]
 67%|██████▋   | 4070/6100 [6:08:52<3:03:42,  5.43s/it]
                                                       
{'loss': 0.0689, 'learning_rate': 0.0001, 'epoch': 6.67}

 67%|██████▋   | 4070/6100 [6:08:52<3:03:42,  5.43s/it]
 67%|██████▋   | 4071/6100 [6:08:57<3:03:38,  5.43s/it]
 67%|██████▋   | 4072/6100 [6:09:02<3:03:21,  5.42s/it]
 67%|██████▋   | 4073/6100 [6:09:08<3:03:19,  5.43s/it]
 67%|██████▋   | 4074/6100 [6:09:13<3:03:08,  5.42s/it]
 67%|██████▋   | 4075/6100 [6:09:19<3:02:52,  5.42s/it]
                                                       
{'loss': 0.0718, 'learning_rate': 0.0001, 'epoch': 6.68}

 67%|██████▋   | 4075/6100 [6:09:19<3:02:52,  5.42s/it]
 67%|██████▋   | 4076/6100 [6:09:25<3:07:41,  5.56s/it]
 67%|██████▋   | 4077/6100 [6:09:30<3:06:13,  5.52s/it]
 67%|██████▋   | 4078/6100 [6:09:35<3:05:04,  5.49s/it]
 67%|██████▋   | 4079/6100 [6:09:41<3:04:22,  5.47s/it]
 67%|██████▋   | 4080/6100 [6:09:46<3:03:47,  5.46s/it]
                                                       
{'loss': 0.0713, 'learning_rate': 0.0001, 'epoch': 6.69}

 67%|██████▋   | 4080/6100 [6:09:46<3:03:47,  5.46s/it]
 67%|██████▋   | 4081/6100 [6:09:52<3:03:28,  5.45s/it]
 67%|██████▋   | 4082/6100 [6:09:57<3:03:09,  5.45s/it]
 67%|██████▋   | 4083/6100 [6:10:03<3:02:48,  5.44s/it]
 67%|██████▋   | 4084/6100 [6:10:08<3:02:27,  5.43s/it]
 67%|██████▋   | 4085/6100 [6:10:13<3:02:04,  5.42s/it]
                                                       
{'loss': 0.0759, 'learning_rate': 0.0001, 'epoch': 6.7}

 67%|██████▋   | 4085/6100 [6:10:13<3:02:04,  5.42s/it]
 67%|██████▋   | 4086/6100 [6:10:19<3:01:56,  5.42s/it]
 67%|██████▋   | 4087/6100 [6:10:24<3:02:52,  5.45s/it]
 67%|██████▋   | 4088/6100 [6:10:30<3:02:28,  5.44s/it]
 67%|██████▋   | 4089/6100 [6:10:35<3:02:17,  5.44s/it]
 67%|██████▋   | 4090/6100 [6:10:41<3:02:02,  5.43s/it]
                                                       
{'loss': 0.076, 'learning_rate': 0.0001, 'epoch': 6.7}

 67%|██████▋   | 4090/6100 [6:10:41<3:02:02,  5.43s/it]
 67%|██████▋   | 4091/6100 [6:10:46<3:01:44,  5.43s/it]
 67%|██████▋   | 4092/6100 [6:10:51<3:01:49,  5.43s/it]
 67%|██████▋   | 4093/6100 [6:10:57<3:01:44,  5.43s/it]
 67%|██████▋   | 4094/6100 [6:11:02<3:01:31,  5.43s/it]
 67%|██████▋   | 4095/6100 [6:11:08<3:01:21,  5.43s/it]
                                                       
{'loss': 0.0683, 'learning_rate': 0.0001, 'epoch': 6.71}

 67%|██████▋   | 4095/6100 [6:11:08<3:01:21,  5.43s/it]
 67%|██████▋   | 4096/6100 [6:11:13<3:01:11,  5.42s/it]
 67%|██████▋   | 4097/6100 [6:11:19<3:01:00,  5.42s/it]
 67%|██████▋   | 4098/6100 [6:11:24<3:01:02,  5.43s/it]
 67%|██████▋   | 4099/6100 [6:11:29<3:00:59,  5.43s/it]
 67%|██████▋   | 4100/6100 [6:11:35<3:00:49,  5.42s/it]
                                                       
{'loss': 0.0766, 'learning_rate': 0.0001, 'epoch': 6.72}

 67%|██████▋   | 4100/6100 [6:11:35<3:00:49,  5.42s/it]
 67%|██████▋   | 4101/6100 [6:11:40<3:00:42,  5.42s/it]
 67%|██████▋   | 4102/6100 [6:11:46<3:00:36,  5.42s/it]
 67%|██████▋   | 4103/6100 [6:11:51<3:00:28,  5.42s/it]
 67%|██████▋   | 4104/6100 [6:11:57<3:00:24,  5.42s/it]
 67%|██████▋   | 4105/6100 [6:12:02<3:00:18,  5.42s/it]
                                                       
{'loss': 0.072, 'learning_rate': 0.0001, 'epoch': 6.73}

 67%|██████▋   | 4105/6100 [6:12:02<3:00:18,  5.42s/it]
 67%|██████▋   | 4106/6100 [6:12:07<3:00:25,  5.43s/it]
 67%|██████▋   | 4107/6100 [6:12:13<3:00:09,  5.42s/it]
 67%|██████▋   | 4108/6100 [6:12:18<3:00:06,  5.43s/it]
 67%|██████▋   | 4109/6100 [6:12:24<3:06:18,  5.61s/it]
 67%|██████▋   | 4110/6100 [6:12:30<3:04:16,  5.56s/it]
                                                       
{'loss': 0.0762, 'learning_rate': 0.0001, 'epoch': 6.74}

 67%|██████▋   | 4110/6100 [6:12:30<3:04:16,  5.56s/it]
 67%|██████▋   | 4111/6100 [6:12:35<3:02:57,  5.52s/it]
 67%|██████▋   | 4112/6100 [6:12:41<3:01:55,  5.49s/it]
 67%|██████▋   | 4113/6100 [6:12:46<3:01:00,  5.47s/it]
 67%|██████▋   | 4114/6100 [6:12:51<3:00:31,  5.45s/it]
 67%|██████▋   | 4115/6100 [6:12:57<3:00:07,  5.44s/it]
                                                       
{'loss': 0.0687, 'learning_rate': 0.0001, 'epoch': 6.75}

 67%|██████▋   | 4115/6100 [6:12:57<3:00:07,  5.44s/it]
 67%|██████▋   | 4116/6100 [6:13:02<2:59:50,  5.44s/it]
 67%|██████▋   | 4117/6100 [6:13:08<2:59:27,  5.43s/it]
 68%|██████▊   | 4118/6100 [6:13:13<2:59:14,  5.43s/it]
 68%|██████▊   | 4119/6100 [6:13:18<2:58:59,  5.42s/it]
 68%|██████▊   | 4120/6100 [6:13:24<2:58:58,  5.42s/it]
                                                       
{'loss': 0.0655, 'learning_rate': 0.0001, 'epoch': 6.75}

 68%|██████▊   | 4120/6100 [6:13:24<2:58:58,  5.42s/it]
 68%|██████▊   | 4121/6100 [6:13:29<2:59:09,  5.43s/it]
 68%|██████▊   | 4122/6100 [6:13:35<2:58:58,  5.43s/it]
 68%|██████▊   | 4123/6100 [6:13:40<2:58:47,  5.43s/it]
 68%|██████▊   | 4124/6100 [6:13:46<2:58:38,  5.42s/it]
 68%|██████▊   | 4125/6100 [6:13:51<2:58:29,  5.42s/it]
                                                       
{'loss': 0.0683, 'learning_rate': 0.0001, 'epoch': 6.76}

 68%|██████▊   | 4125/6100 [6:13:51<2:58:29,  5.42s/it]
 68%|██████▊   | 4126/6100 [6:13:56<2:58:22,  5.42s/it]
 68%|██████▊   | 4127/6100 [6:14:02<2:58:12,  5.42s/it]
 68%|██████▊   | 4128/6100 [6:14:07<2:58:06,  5.42s/it]
 68%|██████▊   | 4129/6100 [6:14:13<2:58:00,  5.42s/it]
 68%|██████▊   | 4130/6100 [6:14:18<2:57:55,  5.42s/it]
                                                       
{'loss': 0.0765, 'learning_rate': 0.0001, 'epoch': 6.77}

 68%|██████▊   | 4130/6100 [6:14:18<2:57:55,  5.42s/it]
 68%|██████▊   | 4131/6100 [6:14:24<2:57:58,  5.42s/it]
 68%|██████▊   | 4132/6100 [6:14:29<2:58:23,  5.44s/it]
 68%|██████▊   | 4133/6100 [6:14:34<2:58:07,  5.43s/it]
 68%|██████▊   | 4134/6100 [6:14:40<2:57:54,  5.43s/it]
 68%|██████▊   | 4135/6100 [6:14:45<2:57:42,  5.43s/it]
                                                       
{'loss': 0.0686, 'learning_rate': 0.0001, 'epoch': 6.78}

 68%|██████▊   | 4135/6100 [6:14:45<2:57:42,  5.43s/it]
 68%|██████▊   | 4136/6100 [6:14:51<2:57:38,  5.43s/it]
 68%|██████▊   | 4137/6100 [6:14:56<2:57:26,  5.42s/it]
 68%|██████▊   | 4138/6100 [6:15:02<2:57:17,  5.42s/it]
 68%|██████▊   | 4139/6100 [6:15:07<2:57:18,  5.43s/it]
 68%|██████▊   | 4140/6100 [6:15:12<2:57:13,  5.43s/it]
                                                       
{'loss': 0.0698, 'learning_rate': 0.0001, 'epoch': 6.79}

 68%|██████▊   | 4140/6100 [6:15:12<2:57:13,  5.43s/it]
 68%|██████▊   | 4141/6100 [6:15:18<2:57:09,  5.43s/it]
 68%|██████▊   | 4142/6100 [6:15:23<2:57:16,  5.43s/it]
 68%|██████▊   | 4143/6100 [6:15:29<2:57:18,  5.44s/it]
 68%|██████▊   | 4144/6100 [6:15:34<2:57:06,  5.43s/it]
 68%|██████▊   | 4145/6100 [6:15:40<2:57:11,  5.44s/it]
                                                       
{'loss': 0.0673, 'learning_rate': 0.0001, 'epoch': 6.79}

 68%|██████▊   | 4145/6100 [6:15:40<2:57:11,  5.44s/it]
 68%|██████▊   | 4146/6100 [6:15:45<2:57:03,  5.44s/it]
 68%|██████▊   | 4147/6100 [6:15:50<2:56:47,  5.43s/it]
 68%|██████▊   | 4148/6100 [6:15:56<2:56:34,  5.43s/it]
 68%|██████▊   | 4149/6100 [6:16:01<2:56:28,  5.43s/it]
 68%|██████▊   | 4150/6100 [6:16:07<2:57:06,  5.45s/it]
                                                       
{'loss': 0.073, 'learning_rate': 0.0001, 'epoch': 6.8}

 68%|██████▊   | 4150/6100 [6:16:07<2:57:06,  5.45s/it]
 68%|██████▊   | 4151/6100 [6:16:12<2:56:49,  5.44s/it]
 68%|██████▊   | 4152/6100 [6:16:18<2:56:32,  5.44s/it]
 68%|██████▊   | 4153/6100 [6:16:23<2:56:22,  5.44s/it]
 68%|██████▊   | 4154/6100 [6:16:29<2:56:04,  5.43s/it]
 68%|██████▊   | 4155/6100 [6:16:34<2:55:57,  5.43s/it]
                                                       
{'loss': 0.0759, 'learning_rate': 0.0001, 'epoch': 6.81}

 68%|██████▊   | 4155/6100 [6:16:34<2:55:57,  5.43s/it]
 68%|██████▊   | 4156/6100 [6:16:39<2:56:06,  5.44s/it]
 68%|██████▊   | 4157/6100 [6:16:45<2:55:58,  5.43s/it]
 68%|██████▊   | 4158/6100 [6:16:50<2:55:52,  5.43s/it]
 68%|██████▊   | 4159/6100 [6:16:56<2:55:39,  5.43s/it]
 68%|██████▊   | 4160/6100 [6:17:01<2:55:29,  5.43s/it]
                                                       
{'loss': 0.0736, 'learning_rate': 0.0001, 'epoch': 6.82}

 68%|██████▊   | 4160/6100 [6:17:01<2:55:29,  5.43s/it]
 68%|██████▊   | 4161/6100 [6:17:07<2:55:21,  5.43s/it]
 68%|██████▊   | 4162/6100 [6:17:12<2:55:09,  5.42s/it]
 68%|██████▊   | 4163/6100 [6:17:17<2:55:01,  5.42s/it]
 68%|██████▊   | 4164/6100 [6:17:23<2:54:54,  5.42s/it]
 68%|██████▊   | 4165/6100 [6:17:28<2:54:50,  5.42s/it]
                                                       
{'loss': 0.0684, 'learning_rate': 0.0001, 'epoch': 6.83}

 68%|██████▊   | 4165/6100 [6:17:28<2:54:50,  5.42s/it]
 68%|██████▊   | 4166/6100 [6:17:34<2:54:50,  5.42s/it]
 68%|██████▊   | 4167/6100 [6:17:39<2:54:43,  5.42s/it]
 68%|██████▊   | 4168/6100 [6:17:44<2:54:33,  5.42s/it]
 68%|██████▊   | 4169/6100 [6:17:50<2:54:39,  5.43s/it]
 68%|██████▊   | 4170/6100 [6:17:55<2:54:42,  5.43s/it]
                                                       
{'loss': 0.0761, 'learning_rate': 0.0001, 'epoch': 6.84}

 68%|██████▊   | 4170/6100 [6:17:55<2:54:42,  5.43s/it]
 68%|██████▊   | 4171/6100 [6:18:01<2:54:58,  5.44s/it]
 68%|██████▊   | 4172/6100 [6:18:06<2:54:39,  5.44s/it]
 68%|██████▊   | 4173/6100 [6:18:12<2:54:29,  5.43s/it]
 68%|██████▊   | 4174/6100 [6:18:17<2:54:16,  5.43s/it]
 68%|██████▊   | 4175/6100 [6:18:22<2:54:01,  5.42s/it]
                                                       
{'loss': 0.0725, 'learning_rate': 0.0001, 'epoch': 6.84}

 68%|██████▊   | 4175/6100 [6:18:22<2:54:01,  5.42s/it]
 68%|██████▊   | 4176/6100 [6:18:28<2:54:01,  5.43s/it]
 68%|██████▊   | 4177/6100 [6:18:33<2:53:49,  5.42s/it]
 68%|██████▊   | 4178/6100 [6:18:39<2:53:48,  5.43s/it]
 69%|██████▊   | 4179/6100 [6:18:44<2:53:41,  5.43s/it]
 69%|██████▊   | 4180/6100 [6:18:50<2:53:31,  5.42s/it]
                                                       
{'loss': 0.0744, 'learning_rate': 0.0001, 'epoch': 6.85}

 69%|██████▊   | 4180/6100 [6:18:50<2:53:31,  5.42s/it]
 69%|██████▊   | 4181/6100 [6:18:55<2:53:29,  5.42s/it]
 69%|██████▊   | 4182/6100 [6:19:00<2:53:23,  5.42s/it]
 69%|██████▊   | 4183/6100 [6:19:06<2:53:08,  5.42s/it]
 69%|██████▊   | 4184/6100 [6:19:11<2:53:06,  5.42s/it]
 69%|██████▊   | 4185/6100 [6:19:17<2:53:07,  5.42s/it]
                                                       
{'loss': 0.0711, 'learning_rate': 0.0001, 'epoch': 6.86}

 69%|██████▊   | 4185/6100 [6:19:17<2:53:07,  5.42s/it]
 69%|██████▊   | 4186/6100 [6:19:22<2:53:08,  5.43s/it]
 69%|██████▊   | 4187/6100 [6:19:28<2:52:58,  5.43s/it]
 69%|██████▊   | 4188/6100 [6:19:33<2:52:58,  5.43s/it]
 69%|██████▊   | 4189/6100 [6:19:38<2:52:50,  5.43s/it]
 69%|██████▊   | 4190/6100 [6:19:44<2:52:42,  5.43s/it]
                                                       
{'loss': 0.0698, 'learning_rate': 0.0001, 'epoch': 6.87}

 69%|██████▊   | 4190/6100 [6:19:44<2:52:42,  5.43s/it]
 69%|██████▊   | 4191/6100 [6:19:49<2:52:44,  5.43s/it]
 69%|██████▊   | 4192/6100 [6:19:55<2:52:41,  5.43s/it]
 69%|██████▊   | 4193/6100 [6:20:00<2:52:21,  5.42s/it]
 69%|██████▉   | 4194/6100 [6:20:06<2:52:15,  5.42s/it]
 69%|██████▉   | 4195/6100 [6:20:11<2:52:18,  5.43s/it]
                                                       
{'loss': 0.0765, 'learning_rate': 0.0001, 'epoch': 6.88}

 69%|██████▉   | 4195/6100 [6:20:11<2:52:18,  5.43s/it]
 69%|██████▉   | 4196/6100 [6:20:16<2:52:14,  5.43s/it]
 69%|██████▉   | 4197/6100 [6:20:22<2:51:58,  5.42s/it]
 69%|██████▉   | 4198/6100 [6:20:27<2:51:46,  5.42s/it]
 69%|██████▉   | 4199/6100 [6:20:33<2:51:42,  5.42s/it]
 69%|██████▉   | 4200/6100 [6:20:38<2:51:45,  5.42s/it]
                                                       
{'loss': 0.0746, 'learning_rate': 0.0001, 'epoch': 6.88}

 69%|██████▉   | 4200/6100 [6:20:38<2:51:45,  5.42s/it]Saving model checkpoint to ./results/checkpoint-4200
Configuration saved in ./results/checkpoint-4200/config.json
Model weights saved in ./results/checkpoint-4200/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-4200/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-4200/special_tokens_map.json
[2023-02-22 02:12:54,287] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step4203 is begin to save!
[2023-02-22 02:12:54,290] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-4200/global_step4203/mp_rank_00_model_states.pt
[2023-02-22 02:12:54,290] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-4200/global_step4203/mp_rank_00_model_states.pt...
[2023-02-22 02:12:55,113] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-4200/global_step4203/mp_rank_00_model_states.pt.
[2023-02-22 02:12:55,115] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-4200/global_step4203/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 02:12:55,289] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-4200/global_step4203/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 02:12:55,290] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-4200/global_step4203/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 02:12:55,290] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4203 is ready now!
Deleting older checkpoint [results/checkpoint-4050] due to args.save_total_limit

 69%|██████▉   | 4201/6100 [6:20:46<3:12:29,  6.08s/it]
 69%|██████▉   | 4202/6100 [6:20:51<3:06:05,  5.88s/it]
 69%|██████▉   | 4203/6100 [6:20:57<3:01:37,  5.74s/it]
 69%|██████▉   | 4204/6100 [6:21:03<3:05:26,  5.87s/it]
 69%|██████▉   | 4205/6100 [6:21:08<3:01:02,  5.73s/it]
                                                       
{'loss': 0.0786, 'learning_rate': 0.0001, 'epoch': 6.89}

 69%|██████▉   | 4205/6100 [6:21:08<3:01:02,  5.73s/it]
 69%|██████▉   | 4206/6100 [6:21:14<2:57:57,  5.64s/it]
 69%|██████▉   | 4207/6100 [6:21:19<2:55:43,  5.57s/it]
 69%|██████▉   | 4208/6100 [6:21:24<2:54:31,  5.53s/it]
 69%|██████▉   | 4209/6100 [6:21:30<2:53:22,  5.50s/it]
 69%|██████▉   | 4210/6100 [6:21:35<2:52:39,  5.48s/it]
                                                       
{'loss': 0.0734, 'learning_rate': 0.0001, 'epoch': 6.9}

 69%|██████▉   | 4210/6100 [6:21:35<2:52:39,  5.48s/it]
 69%|██████▉   | 4211/6100 [6:21:41<2:52:07,  5.47s/it]
 69%|██████▉   | 4212/6100 [6:21:46<2:51:31,  5.45s/it]
 69%|██████▉   | 4213/6100 [6:21:52<2:51:05,  5.44s/it]
 69%|██████▉   | 4214/6100 [6:21:57<2:51:09,  5.45s/it]
 69%|██████▉   | 4215/6100 [6:22:02<2:50:59,  5.44s/it]
                                                       
{'loss': 0.076, 'learning_rate': 0.0001, 'epoch': 6.91}

 69%|██████▉   | 4215/6100 [6:22:02<2:50:59,  5.44s/it]
 69%|██████▉   | 4216/6100 [6:22:08<2:50:51,  5.44s/it]
 69%|██████▉   | 4217/6100 [6:22:13<2:50:32,  5.43s/it]
 69%|██████▉   | 4218/6100 [6:22:19<2:50:21,  5.43s/it]
 69%|██████▉   | 4219/6100 [6:22:24<2:50:12,  5.43s/it]
 69%|██████▉   | 4220/6100 [6:22:30<2:50:08,  5.43s/it]
                                                       
{'loss': 0.0757, 'learning_rate': 0.0001, 'epoch': 6.92}

 69%|██████▉   | 4220/6100 [6:22:30<2:50:08,  5.43s/it]
 69%|██████▉   | 4221/6100 [6:22:35<2:50:00,  5.43s/it]
 69%|██████▉   | 4222/6100 [6:22:40<2:49:50,  5.43s/it]
 69%|██████▉   | 4223/6100 [6:22:46<2:49:47,  5.43s/it]
 69%|██████▉   | 4224/6100 [6:22:51<2:49:30,  5.42s/it]
 69%|██████▉   | 4225/6100 [6:22:57<2:49:22,  5.42s/it]
                                                       
{'loss': 0.0716, 'learning_rate': 0.0001, 'epoch': 6.93}

 69%|██████▉   | 4225/6100 [6:22:57<2:49:22,  5.42s/it]
 69%|██████▉   | 4226/6100 [6:23:02<2:49:21,  5.42s/it]
 69%|██████▉   | 4227/6100 [6:23:08<2:49:28,  5.43s/it]
 69%|██████▉   | 4228/6100 [6:23:13<2:49:19,  5.43s/it]
 69%|██████▉   | 4229/6100 [6:23:18<2:49:13,  5.43s/it]
 69%|██████▉   | 4230/6100 [6:23:24<2:49:01,  5.42s/it]
                                                       
{'loss': 0.0736, 'learning_rate': 0.0001, 'epoch': 6.93}

 69%|██████▉   | 4230/6100 [6:23:24<2:49:01,  5.42s/it]
 69%|██████▉   | 4231/6100 [6:23:29<2:48:56,  5.42s/it]
 69%|██████▉   | 4232/6100 [6:23:35<2:48:58,  5.43s/it]
 69%|██████▉   | 4233/6100 [6:23:40<2:48:45,  5.42s/it]
 69%|██████▉   | 4234/6100 [6:23:45<2:48:29,  5.42s/it]
 69%|██████▉   | 4235/6100 [6:23:51<2:48:28,  5.42s/it]
                                                       
{'loss': 0.0729, 'learning_rate': 0.0001, 'epoch': 6.94}

 69%|██████▉   | 4235/6100 [6:23:51<2:48:28,  5.42s/it]
 69%|██████▉   | 4236/6100 [6:23:56<2:48:22,  5.42s/it]
 69%|██████▉   | 4237/6100 [6:24:02<2:48:15,  5.42s/it]
 69%|██████▉   | 4238/6100 [6:24:07<2:48:07,  5.42s/it]
 69%|██████▉   | 4239/6100 [6:24:13<2:48:00,  5.42s/it]
 70%|██████▉   | 4240/6100 [6:24:18<2:48:02,  5.42s/it]
                                                       
{'loss': 0.0674, 'learning_rate': 0.0001, 'epoch': 6.95}

 70%|██████▉   | 4240/6100 [6:24:18<2:48:02,  5.42s/it]
 70%|██████▉   | 4241/6100 [6:24:23<2:48:01,  5.42s/it]
 70%|██████▉   | 4242/6100 [6:24:29<2:48:04,  5.43s/it]
 70%|██████▉   | 4243/6100 [6:24:34<2:47:56,  5.43s/it]
 70%|██████▉   | 4244/6100 [6:24:40<2:47:54,  5.43s/it]
 70%|██████▉   | 4245/6100 [6:24:45<2:47:52,  5.43s/it]
                                                       
{'loss': 0.071, 'learning_rate': 0.0001, 'epoch': 6.96}

 70%|██████▉   | 4245/6100 [6:24:45<2:47:52,  5.43s/it]
 70%|██████▉   | 4246/6100 [6:24:51<2:47:57,  5.44s/it]
 70%|██████▉   | 4247/6100 [6:24:56<2:47:49,  5.43s/it]
 70%|██████▉   | 4248/6100 [6:25:01<2:47:33,  5.43s/it]
 70%|██████▉   | 4249/6100 [6:25:07<2:47:41,  5.44s/it]
 70%|██████▉   | 4250/6100 [6:25:12<2:47:37,  5.44s/it]
                                                       
{'loss': 0.0717, 'learning_rate': 0.0001, 'epoch': 6.97}

 70%|██████▉   | 4250/6100 [6:25:12<2:47:37,  5.44s/it]
 70%|██████▉   | 4251/6100 [6:25:18<2:47:24,  5.43s/it]
 70%|██████▉   | 4252/6100 [6:25:23<2:47:15,  5.43s/it]
 70%|██████▉   | 4253/6100 [6:25:29<2:47:03,  5.43s/it]
 70%|██████▉   | 4254/6100 [6:25:34<2:46:58,  5.43s/it]
 70%|██████▉   | 4255/6100 [6:25:39<2:46:51,  5.43s/it]
                                                       
{'loss': 0.0736, 'learning_rate': 0.0001, 'epoch': 6.97}

 70%|██████▉   | 4255/6100 [6:25:39<2:46:51,  5.43s/it]
 70%|██████▉   | 4256/6100 [6:25:45<2:46:47,  5.43s/it]
 70%|██████▉   | 4257/6100 [6:25:50<2:46:37,  5.42s/it]
 70%|██████▉   | 4258/6100 [6:25:56<2:46:33,  5.43s/it]
 70%|██████▉   | 4259/6100 [6:26:01<2:46:30,  5.43s/it]
 70%|██████▉   | 4260/6100 [6:26:07<2:46:21,  5.42s/it]
                                                       
{'loss': 0.0716, 'learning_rate': 0.0001, 'epoch': 6.98}

 70%|██████▉   | 4260/6100 [6:26:07<2:46:21,  5.42s/it]
 70%|██████▉   | 4261/6100 [6:26:12<2:46:16,  5.42s/it]
 70%|██████▉   | 4262/6100 [6:26:17<2:46:06,  5.42s/it]
 70%|██████▉   | 4263/6100 [6:26:23<2:46:02,  5.42s/it]
 70%|██████▉   | 4264/6100 [6:26:28<2:45:58,  5.42s/it]
 70%|██████▉   | 4265/6100 [6:26:34<2:45:46,  5.42s/it]
                                                       
{'loss': 0.0729, 'learning_rate': 0.0001, 'epoch': 6.99}

 70%|██████▉   | 4265/6100 [6:26:34<2:45:46,  5.42s/it]
 70%|██████▉   | 4266/6100 [6:26:39<2:45:38,  5.42s/it]
 70%|██████▉   | 4267/6100 [6:26:45<2:45:40,  5.42s/it]
 70%|██████▉   | 4268/6100 [6:26:50<2:45:33,  5.42s/it]
 70%|██████▉   | 4269/6100 [6:26:55<2:45:25,  5.42s/it]
 70%|███████   | 4270/6100 [6:27:01<2:45:18,  5.42s/it]
                                                       
{'loss': 0.0719, 'learning_rate': 0.0001, 'epoch': 7.0}

 70%|███████   | 4270/6100 [6:27:01<2:45:18,  5.42s/it]
 70%|███████   | 4271/6100 [6:27:09<3:09:06,  6.20s/it]
 70%|███████   | 4272/6100 [6:27:14<3:01:45,  5.97s/it]
 70%|███████   | 4273/6100 [6:27:20<2:56:42,  5.80s/it]
 70%|███████   | 4274/6100 [6:27:25<2:53:03,  5.69s/it]
 70%|███████   | 4275/6100 [6:27:30<2:50:31,  5.61s/it]
                                                       
{'loss': 0.083, 'learning_rate': 0.0001, 'epoch': 7.01}

 70%|███████   | 4275/6100 [6:27:30<2:50:31,  5.61s/it]
 70%|███████   | 4276/6100 [6:27:36<2:48:46,  5.55s/it]
 70%|███████   | 4277/6100 [6:27:41<2:47:34,  5.52s/it]
 70%|███████   | 4278/6100 [6:27:47<2:46:45,  5.49s/it]
 70%|███████   | 4279/6100 [6:27:52<2:46:02,  5.47s/it]
 70%|███████   | 4280/6100 [6:27:58<2:45:23,  5.45s/it]
                                                       
{'loss': 0.0683, 'learning_rate': 0.0001, 'epoch': 7.02}

 70%|███████   | 4280/6100 [6:27:58<2:45:23,  5.45s/it]
 70%|███████   | 4281/6100 [6:28:03<2:45:11,  5.45s/it]
 70%|███████   | 4282/6100 [6:28:08<2:44:47,  5.44s/it]
 70%|███████   | 4283/6100 [6:28:14<2:44:39,  5.44s/it]
 70%|███████   | 4284/6100 [6:28:19<2:44:24,  5.43s/it]
 70%|███████   | 4285/6100 [6:28:25<2:44:14,  5.43s/it]
                                                       
{'loss': 0.0731, 'learning_rate': 0.0001, 'epoch': 7.02}

 70%|███████   | 4285/6100 [6:28:25<2:44:14,  5.43s/it]
 70%|███████   | 4286/6100 [6:28:30<2:44:06,  5.43s/it]
 70%|███████   | 4287/6100 [6:28:36<2:43:56,  5.43s/it]
 70%|███████   | 4288/6100 [6:28:41<2:43:47,  5.42s/it]
 70%|███████   | 4289/6100 [6:28:46<2:43:34,  5.42s/it]
 70%|███████   | 4290/6100 [6:28:52<2:43:30,  5.42s/it]
                                                       
{'loss': 0.0674, 'learning_rate': 0.0001, 'epoch': 7.03}

 70%|███████   | 4290/6100 [6:28:52<2:43:30,  5.42s/it]
 70%|███████   | 4291/6100 [6:28:57<2:43:24,  5.42s/it]
 70%|███████   | 4292/6100 [6:29:03<2:43:30,  5.43s/it]
 70%|███████   | 4293/6100 [6:29:08<2:43:23,  5.43s/it]
 70%|███████   | 4294/6100 [6:29:14<2:43:14,  5.42s/it]
 70%|███████   | 4295/6100 [6:29:19<2:43:09,  5.42s/it]
                                                       
{'loss': 0.0689, 'learning_rate': 0.0001, 'epoch': 7.04}

 70%|███████   | 4295/6100 [6:29:19<2:43:09,  5.42s/it]
 70%|███████   | 4296/6100 [6:29:24<2:43:06,  5.42s/it]
 70%|███████   | 4297/6100 [6:29:30<2:42:56,  5.42s/it]
 70%|███████   | 4298/6100 [6:29:35<2:42:44,  5.42s/it]
 70%|███████   | 4299/6100 [6:29:41<2:42:38,  5.42s/it]
 70%|███████   | 4300/6100 [6:29:46<2:42:31,  5.42s/it]
                                                       
{'loss': 0.0717, 'learning_rate': 0.0001, 'epoch': 7.05}

 70%|███████   | 4300/6100 [6:29:46<2:42:31,  5.42s/it]
 71%|███████   | 4301/6100 [6:29:51<2:42:36,  5.42s/it]
 71%|███████   | 4302/6100 [6:29:57<2:42:31,  5.42s/it]
 71%|███████   | 4303/6100 [6:30:02<2:42:24,  5.42s/it]
 71%|███████   | 4304/6100 [6:30:08<2:42:18,  5.42s/it]
 71%|███████   | 4305/6100 [6:30:13<2:42:17,  5.42s/it]
                                                       
{'loss': 0.0727, 'learning_rate': 0.0001, 'epoch': 7.06}

 71%|███████   | 4305/6100 [6:30:13<2:42:17,  5.42s/it]
 71%|███████   | 4306/6100 [6:30:19<2:42:08,  5.42s/it]
 71%|███████   | 4307/6100 [6:30:24<2:42:03,  5.42s/it]
 71%|███████   | 4308/6100 [6:30:29<2:41:59,  5.42s/it]
 71%|███████   | 4309/6100 [6:30:35<2:41:55,  5.42s/it]
 71%|███████   | 4310/6100 [6:30:40<2:41:42,  5.42s/it]
                                                       
{'loss': 0.0676, 'learning_rate': 0.0001, 'epoch': 7.07}

 71%|███████   | 4310/6100 [6:30:40<2:41:42,  5.42s/it]
 71%|███████   | 4311/6100 [6:30:46<2:41:52,  5.43s/it]
 71%|███████   | 4312/6100 [6:30:51<2:41:56,  5.43s/it]
 71%|███████   | 4313/6100 [6:30:57<2:41:41,  5.43s/it]
 71%|███████   | 4314/6100 [6:31:02<2:41:32,  5.43s/it]
 71%|███████   | 4315/6100 [6:31:07<2:41:25,  5.43s/it]
                                                       
{'loss': 0.0681, 'learning_rate': 0.0001, 'epoch': 7.07}

 71%|███████   | 4315/6100 [6:31:07<2:41:25,  5.43s/it]
 71%|███████   | 4316/6100 [6:31:13<2:41:19,  5.43s/it]
 71%|███████   | 4317/6100 [6:31:18<2:41:08,  5.42s/it]
 71%|███████   | 4318/6100 [6:31:24<2:40:59,  5.42s/it]
 71%|███████   | 4319/6100 [6:31:29<2:40:54,  5.42s/it]
 71%|███████   | 4320/6100 [6:31:35<2:40:50,  5.42s/it]
                                                       
{'loss': 0.0665, 'learning_rate': 0.0001, 'epoch': 7.08}

 71%|███████   | 4320/6100 [6:31:35<2:40:50,  5.42s/it]
 71%|███████   | 4321/6100 [6:31:40<2:40:50,  5.42s/it]
 71%|███████   | 4322/6100 [6:31:45<2:40:45,  5.43s/it]
 71%|███████   | 4323/6100 [6:31:51<2:40:42,  5.43s/it]
 71%|███████   | 4324/6100 [6:31:56<2:40:35,  5.43s/it]
 71%|███████   | 4325/6100 [6:32:02<2:40:30,  5.43s/it]
                                                       
{'loss': 0.0696, 'learning_rate': 0.0001, 'epoch': 7.09}

 71%|███████   | 4325/6100 [6:32:02<2:40:30,  5.43s/it]
 71%|███████   | 4326/6100 [6:32:07<2:40:29,  5.43s/it]
 71%|███████   | 4327/6100 [6:32:13<2:40:22,  5.43s/it]
 71%|███████   | 4328/6100 [6:32:18<2:40:13,  5.43s/it]
 71%|███████   | 4329/6100 [6:32:23<2:40:01,  5.42s/it]
 71%|███████   | 4330/6100 [6:32:29<2:39:51,  5.42s/it]
                                                       
{'loss': 0.0707, 'learning_rate': 0.0001, 'epoch': 7.1}

 71%|███████   | 4330/6100 [6:32:29<2:39:51,  5.42s/it]
 71%|███████   | 4331/6100 [6:32:34<2:39:44,  5.42s/it]
 71%|███████   | 4332/6100 [6:32:40<2:40:34,  5.45s/it]
 71%|███████   | 4333/6100 [6:32:45<2:40:08,  5.44s/it]
 71%|███████   | 4334/6100 [6:32:51<2:39:52,  5.43s/it]
 71%|███████   | 4335/6100 [6:32:56<2:39:46,  5.43s/it]
                                                       
{'loss': 0.0677, 'learning_rate': 0.0001, 'epoch': 7.11}

 71%|███████   | 4335/6100 [6:32:56<2:39:46,  5.43s/it]
 71%|███████   | 4336/6100 [6:33:01<2:39:39,  5.43s/it]
 71%|███████   | 4337/6100 [6:33:07<2:39:34,  5.43s/it]
 71%|███████   | 4338/6100 [6:33:12<2:40:55,  5.48s/it]
 71%|███████   | 4339/6100 [6:33:18<2:40:16,  5.46s/it]
 71%|███████   | 4340/6100 [6:33:23<2:39:49,  5.45s/it]
                                                       
{'loss': 0.0664, 'learning_rate': 0.0001, 'epoch': 7.11}

 71%|███████   | 4340/6100 [6:33:23<2:39:49,  5.45s/it]
 71%|███████   | 4341/6100 [6:33:29<2:39:33,  5.44s/it]
 71%|███████   | 4342/6100 [6:33:34<2:39:12,  5.43s/it]
 71%|███████   | 4343/6100 [6:33:40<2:38:57,  5.43s/it]
 71%|███████   | 4344/6100 [6:33:45<2:38:46,  5.42s/it]
 71%|███████   | 4345/6100 [6:33:50<2:38:36,  5.42s/it]
                                                       
{'loss': 0.0707, 'learning_rate': 0.0001, 'epoch': 7.12}

 71%|███████   | 4345/6100 [6:33:50<2:38:36,  5.42s/it]
 71%|███████   | 4346/6100 [6:33:57<2:46:01,  5.68s/it]
 71%|███████▏  | 4347/6100 [6:34:02<2:43:34,  5.60s/it]
 71%|███████▏  | 4348/6100 [6:34:07<2:41:54,  5.54s/it]
 71%|███████▏  | 4349/6100 [6:34:13<2:40:41,  5.51s/it]
 71%|███████▏  | 4350/6100 [6:34:18<2:39:51,  5.48s/it]
                                                       
{'loss': 0.0703, 'learning_rate': 0.0001, 'epoch': 7.13}

 71%|███████▏  | 4350/6100 [6:34:18<2:39:51,  5.48s/it]Saving model checkpoint to ./results/checkpoint-4350
Configuration saved in ./results/checkpoint-4350/config.json
Model weights saved in ./results/checkpoint-4350/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-4350/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-4350/special_tokens_map.json
[2023-02-22 02:26:34,494] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step4353 is begin to save!
[2023-02-22 02:26:34,497] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-4350/global_step4353/mp_rank_00_model_states.pt
[2023-02-22 02:26:34,497] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-4350/global_step4353/mp_rank_00_model_states.pt...
[2023-02-22 02:26:35,321] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-4350/global_step4353/mp_rank_00_model_states.pt.
[2023-02-22 02:26:35,323] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-4350/global_step4353/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 02:26:35,496] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-4350/global_step4353/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 02:26:35,496] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-4350/global_step4353/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 02:26:35,497] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4353 is ready now!
Deleting older checkpoint [results/checkpoint-4200] due to args.save_total_limit

 71%|███████▏  | 4351/6100 [6:34:26<2:58:05,  6.11s/it]
 71%|███████▏  | 4352/6100 [6:34:31<2:51:52,  5.90s/it]
 71%|███████▏  | 4353/6100 [6:34:37<2:47:38,  5.76s/it]
 71%|███████▏  | 4354/6100 [6:34:42<2:44:32,  5.65s/it]
 71%|███████▏  | 4355/6100 [6:34:48<2:42:19,  5.58s/it]
                                                       
{'loss': 0.0692, 'learning_rate': 0.0001, 'epoch': 7.14}

 71%|███████▏  | 4355/6100 [6:34:48<2:42:19,  5.58s/it]
 71%|███████▏  | 4356/6100 [6:34:53<2:40:43,  5.53s/it]
 71%|███████▏  | 4357/6100 [6:34:58<2:39:33,  5.49s/it]
 71%|███████▏  | 4358/6100 [6:35:04<2:38:44,  5.47s/it]
 71%|███████▏  | 4359/6100 [6:35:09<2:38:18,  5.46s/it]
 71%|███████▏  | 4360/6100 [6:35:15<2:37:46,  5.44s/it]
                                                       
{'loss': 0.0696, 'learning_rate': 0.0001, 'epoch': 7.15}

 71%|███████▏  | 4360/6100 [6:35:15<2:37:46,  5.44s/it]
 71%|███████▏  | 4361/6100 [6:35:20<2:37:24,  5.43s/it]
 72%|███████▏  | 4362/6100 [6:35:25<2:37:11,  5.43s/it]
 72%|███████▏  | 4363/6100 [6:35:31<2:37:01,  5.42s/it]
 72%|███████▏  | 4364/6100 [6:35:36<2:37:04,  5.43s/it]
 72%|███████▏  | 4365/6100 [6:35:42<2:36:50,  5.42s/it]
                                                       
{'loss': 0.0709, 'learning_rate': 0.0001, 'epoch': 7.16}

 72%|███████▏  | 4365/6100 [6:35:42<2:36:50,  5.42s/it]
 72%|███████▏  | 4366/6100 [6:35:47<2:36:41,  5.42s/it]
 72%|███████▏  | 4367/6100 [6:35:53<2:36:26,  5.42s/it]
 72%|███████▏  | 4368/6100 [6:35:58<2:36:15,  5.41s/it]
 72%|███████▏  | 4369/6100 [6:36:03<2:36:14,  5.42s/it]
 72%|███████▏  | 4370/6100 [6:36:09<2:36:09,  5.42s/it]
                                                       
{'loss': 0.074, 'learning_rate': 0.0001, 'epoch': 7.16}

 72%|███████▏  | 4370/6100 [6:36:09<2:36:09,  5.42s/it]
 72%|███████▏  | 4371/6100 [6:36:14<2:36:58,  5.45s/it]
 72%|███████▏  | 4372/6100 [6:36:20<2:36:32,  5.44s/it]
 72%|███████▏  | 4373/6100 [6:36:25<2:36:19,  5.43s/it]
 72%|███████▏  | 4374/6100 [6:36:31<2:36:12,  5.43s/it]
 72%|███████▏  | 4375/6100 [6:36:36<2:36:02,  5.43s/it]
                                                       
{'loss': 0.0712, 'learning_rate': 0.0001, 'epoch': 7.17}

 72%|███████▏  | 4375/6100 [6:36:36<2:36:02,  5.43s/it]
 72%|███████▏  | 4376/6100 [6:36:41<2:35:52,  5.42s/it]
 72%|███████▏  | 4377/6100 [6:36:47<2:35:43,  5.42s/it]
 72%|███████▏  | 4378/6100 [6:36:52<2:35:37,  5.42s/it]
 72%|███████▏  | 4379/6100 [6:36:58<2:35:25,  5.42s/it]
 72%|███████▏  | 4380/6100 [6:37:03<2:35:20,  5.42s/it]
                                                       
{'loss': 0.0704, 'learning_rate': 0.0001, 'epoch': 7.18}

 72%|███████▏  | 4380/6100 [6:37:03<2:35:20,  5.42s/it]
 72%|███████▏  | 4381/6100 [6:37:09<2:35:30,  5.43s/it]
 72%|███████▏  | 4382/6100 [6:37:14<2:35:29,  5.43s/it]
 72%|███████▏  | 4383/6100 [6:37:19<2:35:26,  5.43s/it]
 72%|███████▏  | 4384/6100 [6:37:25<2:35:10,  5.43s/it]
 72%|███████▏  | 4385/6100 [6:37:30<2:34:59,  5.42s/it]
                                                       
{'loss': 0.0751, 'learning_rate': 0.0001, 'epoch': 7.19}

 72%|███████▏  | 4385/6100 [6:37:30<2:34:59,  5.42s/it]
 72%|███████▏  | 4386/6100 [6:37:36<2:34:56,  5.42s/it]
 72%|███████▏  | 4387/6100 [6:37:41<2:34:45,  5.42s/it]
 72%|███████▏  | 4388/6100 [6:37:46<2:34:48,  5.43s/it]
 72%|███████▏  | 4389/6100 [6:37:52<2:34:43,  5.43s/it]
 72%|███████▏  | 4390/6100 [6:37:57<2:34:39,  5.43s/it]
                                                       
{'loss': 0.0677, 'learning_rate': 0.0001, 'epoch': 7.2}

 72%|███████▏  | 4390/6100 [6:37:57<2:34:39,  5.43s/it]
 72%|███████▏  | 4391/6100 [6:38:03<2:34:26,  5.42s/it]
 72%|███████▏  | 4392/6100 [6:38:08<2:34:18,  5.42s/it]
 72%|███████▏  | 4393/6100 [6:38:14<2:34:15,  5.42s/it]
 72%|███████▏  | 4394/6100 [6:38:19<2:34:10,  5.42s/it]
 72%|███████▏  | 4395/6100 [6:38:24<2:34:08,  5.42s/it]
                                                       
{'loss': 0.0692, 'learning_rate': 0.0001, 'epoch': 7.2}

 72%|███████▏  | 4395/6100 [6:38:24<2:34:08,  5.42s/it]
 72%|███████▏  | 4396/6100 [6:38:30<2:34:11,  5.43s/it]
 72%|███████▏  | 4397/6100 [6:38:35<2:34:01,  5.43s/it]
 72%|███████▏  | 4398/6100 [6:38:41<2:33:57,  5.43s/it]
 72%|███████▏  | 4399/6100 [6:38:46<2:33:53,  5.43s/it]
 72%|███████▏  | 4400/6100 [6:38:52<2:33:36,  5.42s/it]
                                                       
{'loss': 0.0731, 'learning_rate': 0.0001, 'epoch': 7.21}

 72%|███████▏  | 4400/6100 [6:38:52<2:33:36,  5.42s/it]
 72%|███████▏  | 4401/6100 [6:38:57<2:33:35,  5.42s/it]
 72%|███████▏  | 4402/6100 [6:39:02<2:33:33,  5.43s/it]
 72%|███████▏  | 4403/6100 [6:39:08<2:33:33,  5.43s/it]
 72%|███████▏  | 4404/6100 [6:39:13<2:33:28,  5.43s/it]
 72%|███████▏  | 4405/6100 [6:39:19<2:33:18,  5.43s/it]
                                                       
{'loss': 0.072, 'learning_rate': 0.0001, 'epoch': 7.22}

 72%|███████▏  | 4405/6100 [6:39:19<2:33:18,  5.43s/it]
 72%|███████▏  | 4406/6100 [6:39:24<2:33:15,  5.43s/it]
 72%|███████▏  | 4407/6100 [6:39:30<2:33:11,  5.43s/it]
 72%|███████▏  | 4408/6100 [6:39:35<2:32:57,  5.42s/it]
 72%|███████▏  | 4409/6100 [6:39:40<2:32:47,  5.42s/it]
 72%|███████▏  | 4410/6100 [6:39:46<2:32:40,  5.42s/it]
                                                       
{'loss': 0.074, 'learning_rate': 0.0001, 'epoch': 7.23}

 72%|███████▏  | 4410/6100 [6:39:46<2:32:40,  5.42s/it]
 72%|███████▏  | 4411/6100 [6:39:51<2:32:47,  5.43s/it]
 72%|███████▏  | 4412/6100 [6:39:57<2:32:43,  5.43s/it]
 72%|███████▏  | 4413/6100 [6:40:02<2:32:30,  5.42s/it]
 72%|███████▏  | 4414/6100 [6:40:08<2:32:18,  5.42s/it]
 72%|███████▏  | 4415/6100 [6:40:13<2:32:13,  5.42s/it]
                                                       
{'loss': 0.0678, 'learning_rate': 0.0001, 'epoch': 7.24}

 72%|███████▏  | 4415/6100 [6:40:13<2:32:13,  5.42s/it]
 72%|███████▏  | 4416/6100 [6:40:18<2:32:14,  5.42s/it]
 72%|███████▏  | 4417/6100 [6:40:24<2:32:06,  5.42s/it]
 72%|███████▏  | 4418/6100 [6:40:29<2:32:02,  5.42s/it]
 72%|███████▏  | 4419/6100 [6:40:35<2:31:57,  5.42s/it]
 72%|███████▏  | 4420/6100 [6:40:40<2:31:45,  5.42s/it]
                                                       
{'loss': 0.071, 'learning_rate': 0.0001, 'epoch': 7.25}

 72%|███████▏  | 4420/6100 [6:40:40<2:31:45,  5.42s/it]
 72%|███████▏  | 4421/6100 [6:40:45<2:31:41,  5.42s/it]
 72%|███████▏  | 4422/6100 [6:40:51<2:31:32,  5.42s/it]
 73%|███████▎  | 4423/6100 [6:40:56<2:31:24,  5.42s/it]
 73%|███████▎  | 4424/6100 [6:41:02<2:31:18,  5.42s/it]
 73%|███████▎  | 4425/6100 [6:41:07<2:31:22,  5.42s/it]
                                                       
{'loss': 0.0701, 'learning_rate': 0.0001, 'epoch': 7.25}

 73%|███████▎  | 4425/6100 [6:41:07<2:31:22,  5.42s/it]
 73%|███████▎  | 4426/6100 [6:41:13<2:31:10,  5.42s/it]
 73%|███████▎  | 4427/6100 [6:41:18<2:31:04,  5.42s/it]
 73%|███████▎  | 4428/6100 [6:41:23<2:31:00,  5.42s/it]
 73%|███████▎  | 4429/6100 [6:41:29<2:30:49,  5.42s/it]
 73%|███████▎  | 4430/6100 [6:41:34<2:30:52,  5.42s/it]
                                                       
{'loss': 0.0709, 'learning_rate': 0.0001, 'epoch': 7.26}

 73%|███████▎  | 4430/6100 [6:41:34<2:30:52,  5.42s/it]
 73%|███████▎  | 4431/6100 [6:41:40<2:30:45,  5.42s/it]
 73%|███████▎  | 4432/6100 [6:41:45<2:30:38,  5.42s/it]
 73%|███████▎  | 4433/6100 [6:41:51<2:30:37,  5.42s/it]
 73%|███████▎  | 4434/6100 [6:41:56<2:30:29,  5.42s/it]
 73%|███████▎  | 4435/6100 [6:42:01<2:30:27,  5.42s/it]
                                                       
{'loss': 0.0678, 'learning_rate': 0.0001, 'epoch': 7.27}

 73%|███████▎  | 4435/6100 [6:42:01<2:30:27,  5.42s/it]
 73%|███████▎  | 4436/6100 [6:42:07<2:30:20,  5.42s/it]
 73%|███████▎  | 4437/6100 [6:42:12<2:30:16,  5.42s/it]
 73%|███████▎  | 4438/6100 [6:42:18<2:30:10,  5.42s/it]
 73%|███████▎  | 4439/6100 [6:42:23<2:30:00,  5.42s/it]
 73%|███████▎  | 4440/6100 [6:42:28<2:30:01,  5.42s/it]
                                                       
{'loss': 0.0706, 'learning_rate': 0.0001, 'epoch': 7.28}

 73%|███████▎  | 4440/6100 [6:42:28<2:30:01,  5.42s/it]
 73%|███████▎  | 4441/6100 [6:42:34<2:29:56,  5.42s/it]
 73%|███████▎  | 4442/6100 [6:42:39<2:30:01,  5.43s/it]
 73%|███████▎  | 4443/6100 [6:42:45<2:29:47,  5.42s/it]
 73%|███████▎  | 4444/6100 [6:42:50<2:29:46,  5.43s/it]
 73%|███████▎  | 4445/6100 [6:42:56<2:29:41,  5.43s/it]
                                                       
{'loss': 0.0687, 'learning_rate': 0.0001, 'epoch': 7.29}

 73%|███████▎  | 4445/6100 [6:42:56<2:29:41,  5.43s/it]
 73%|███████▎  | 4446/6100 [6:43:01<2:29:32,  5.42s/it]
 73%|███████▎  | 4447/6100 [6:43:06<2:29:20,  5.42s/it]
 73%|███████▎  | 4448/6100 [6:43:12<2:29:22,  5.43s/it]
 73%|███████▎  | 4449/6100 [6:43:17<2:29:12,  5.42s/it]
 73%|███████▎  | 4450/6100 [6:43:23<2:29:20,  5.43s/it]
                                                       
{'loss': 0.0712, 'learning_rate': 0.0001, 'epoch': 7.29}

 73%|███████▎  | 4450/6100 [6:43:23<2:29:20,  5.43s/it]
 73%|███████▎  | 4451/6100 [6:43:28<2:29:13,  5.43s/it]
 73%|███████▎  | 4452/6100 [6:43:34<2:29:07,  5.43s/it]
 73%|███████▎  | 4453/6100 [6:43:39<2:28:52,  5.42s/it]
 73%|███████▎  | 4454/6100 [6:43:44<2:28:45,  5.42s/it]
 73%|███████▎  | 4455/6100 [6:43:50<2:28:36,  5.42s/it]
                                                       
{'loss': 0.0684, 'learning_rate': 0.0001, 'epoch': 7.3}

 73%|███████▎  | 4455/6100 [6:43:50<2:28:36,  5.42s/it]
 73%|███████▎  | 4456/6100 [6:43:55<2:28:42,  5.43s/it]
 73%|███████▎  | 4457/6100 [6:44:01<2:28:35,  5.43s/it]
 73%|███████▎  | 4458/6100 [6:44:06<2:28:26,  5.42s/it]
 73%|███████▎  | 4459/6100 [6:44:12<2:28:25,  5.43s/it]
 73%|███████▎  | 4460/6100 [6:44:17<2:28:13,  5.42s/it]
                                                       
{'loss': 0.0695, 'learning_rate': 0.0001, 'epoch': 7.31}

 73%|███████▎  | 4460/6100 [6:44:17<2:28:13,  5.42s/it]
 73%|███████▎  | 4461/6100 [6:44:22<2:28:09,  5.42s/it]
 73%|███████▎  | 4462/6100 [6:44:28<2:28:02,  5.42s/it]
 73%|███████▎  | 4463/6100 [6:44:33<2:27:53,  5.42s/it]
 73%|███████▎  | 4464/6100 [6:44:39<2:27:55,  5.42s/it]
 73%|███████▎  | 4465/6100 [6:44:44<2:27:48,  5.42s/it]
                                                       
{'loss': 0.0716, 'learning_rate': 0.0001, 'epoch': 7.32}

 73%|███████▎  | 4465/6100 [6:44:44<2:27:48,  5.42s/it]
 73%|███████▎  | 4466/6100 [6:44:50<2:27:48,  5.43s/it]
 73%|███████▎  | 4467/6100 [6:44:55<2:27:39,  5.43s/it]
 73%|███████▎  | 4468/6100 [6:45:00<2:27:28,  5.42s/it]
 73%|███████▎  | 4469/6100 [6:45:06<2:27:20,  5.42s/it]
 73%|███████▎  | 4470/6100 [6:45:11<2:27:45,  5.44s/it]
                                                       
{'loss': 0.0711, 'learning_rate': 0.0001, 'epoch': 7.33}

 73%|███████▎  | 4470/6100 [6:45:11<2:27:45,  5.44s/it]
 73%|███████▎  | 4471/6100 [6:45:17<2:27:31,  5.43s/it]
 73%|███████▎  | 4472/6100 [6:45:22<2:27:16,  5.43s/it]
 73%|███████▎  | 4473/6100 [6:45:28<2:27:05,  5.42s/it]
 73%|███████▎  | 4474/6100 [6:45:33<2:26:58,  5.42s/it]
 73%|███████▎  | 4475/6100 [6:45:38<2:26:52,  5.42s/it]
                                                       
{'loss': 0.0706, 'learning_rate': 0.0001, 'epoch': 7.34}

 73%|███████▎  | 4475/6100 [6:45:38<2:26:52,  5.42s/it]
 73%|███████▎  | 4476/6100 [6:45:44<2:26:45,  5.42s/it]
 73%|███████▎  | 4477/6100 [6:45:49<2:26:35,  5.42s/it]
 73%|███████▎  | 4478/6100 [6:45:55<2:26:29,  5.42s/it]
 73%|███████▎  | 4479/6100 [6:46:00<2:26:22,  5.42s/it]
 73%|███████▎  | 4480/6100 [6:46:05<2:26:18,  5.42s/it]
                                                       
{'loss': 0.0703, 'learning_rate': 0.0001, 'epoch': 7.34}

 73%|███████▎  | 4480/6100 [6:46:05<2:26:18,  5.42s/it]
 73%|███████▎  | 4481/6100 [6:46:11<2:26:10,  5.42s/it]
 73%|███████▎  | 4482/6100 [6:46:16<2:26:07,  5.42s/it]
 73%|███████▎  | 4483/6100 [6:46:22<2:26:07,  5.42s/it]
 74%|███████▎  | 4484/6100 [6:46:27<2:26:15,  5.43s/it]
 74%|███████▎  | 4485/6100 [6:46:33<2:26:06,  5.43s/it]
                                                       
{'loss': 0.0666, 'learning_rate': 0.0001, 'epoch': 7.35}

 74%|███████▎  | 4485/6100 [6:46:33<2:26:06,  5.43s/it]
 74%|███████▎  | 4486/6100 [6:46:38<2:25:58,  5.43s/it]
 74%|███████▎  | 4487/6100 [6:46:43<2:25:48,  5.42s/it]
 74%|███████▎  | 4488/6100 [6:46:49<2:25:46,  5.43s/it]
 74%|███████▎  | 4489/6100 [6:46:54<2:25:56,  5.44s/it]
 74%|███████▎  | 4490/6100 [6:47:00<2:25:53,  5.44s/it]
                                                       
{'loss': 0.0679, 'learning_rate': 0.0001, 'epoch': 7.36}

 74%|███████▎  | 4490/6100 [6:47:00<2:25:53,  5.44s/it]
 74%|███████▎  | 4491/6100 [6:47:05<2:25:36,  5.43s/it]
 74%|███████▎  | 4492/6100 [6:47:11<2:25:25,  5.43s/it]
 74%|███████▎  | 4493/6100 [6:47:16<2:25:21,  5.43s/it]
 74%|███████▎  | 4494/6100 [6:47:21<2:25:16,  5.43s/it]
 74%|███████▎  | 4495/6100 [6:47:27<2:25:11,  5.43s/it]
                                                       
{'loss': 0.0663, 'learning_rate': 0.0001, 'epoch': 7.37}

 74%|███████▎  | 4495/6100 [6:47:27<2:25:11,  5.43s/it]
 74%|███████▎  | 4496/6100 [6:47:32<2:25:01,  5.42s/it]
 74%|███████▎  | 4497/6100 [6:47:38<2:24:49,  5.42s/it]
 74%|███████▎  | 4498/6100 [6:47:43<2:24:47,  5.42s/it]
 74%|███████▍  | 4499/6100 [6:47:49<2:24:44,  5.42s/it]
 74%|███████▍  | 4500/6100 [6:47:54<2:24:42,  5.43s/it]
                                                       
{'loss': 0.0716, 'learning_rate': 0.0001, 'epoch': 7.38}

 74%|███████▍  | 4500/6100 [6:47:54<2:24:42,  5.43s/it]Saving model checkpoint to ./results/checkpoint-4500
Configuration saved in ./results/checkpoint-4500/config.json
Model weights saved in ./results/checkpoint-4500/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-4500/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-4500/special_tokens_map.json
[2023-02-22 02:40:10,153] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step4503 is begin to save!
[2023-02-22 02:40:10,156] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-4500/global_step4503/mp_rank_00_model_states.pt
[2023-02-22 02:40:10,156] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-4500/global_step4503/mp_rank_00_model_states.pt...
[2023-02-22 02:40:10,967] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-4500/global_step4503/mp_rank_00_model_states.pt.
[2023-02-22 02:40:10,969] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-4500/global_step4503/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 02:40:11,153] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-4500/global_step4503/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 02:40:11,154] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-4500/global_step4503/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 02:40:11,154] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4503 is ready now!
Deleting older checkpoint [results/checkpoint-4350] due to args.save_total_limit

 74%|███████▍  | 4501/6100 [6:48:02<2:41:40,  6.07s/it]
 74%|███████▍  | 4502/6100 [6:48:07<2:36:23,  5.87s/it]
 74%|███████▍  | 4503/6100 [6:48:12<2:32:41,  5.74s/it]
 74%|███████▍  | 4504/6100 [6:48:18<2:30:04,  5.64s/it]
 74%|███████▍  | 4505/6100 [6:48:23<2:28:12,  5.58s/it]
                                                       
{'loss': 0.0702, 'learning_rate': 0.0001, 'epoch': 7.38}

 74%|███████▍  | 4505/6100 [6:48:23<2:28:12,  5.58s/it]
 74%|███████▍  | 4506/6100 [6:48:29<2:26:56,  5.53s/it]
 74%|███████▍  | 4507/6100 [6:48:34<2:25:55,  5.50s/it]
 74%|███████▍  | 4508/6100 [6:48:39<2:25:08,  5.47s/it]
 74%|███████▍  | 4509/6100 [6:48:45<2:24:36,  5.45s/it]
 74%|███████▍  | 4510/6100 [6:48:50<2:24:18,  5.45s/it]
                                                       
{'loss': 0.0692, 'learning_rate': 0.0001, 'epoch': 7.39}

 74%|███████▍  | 4510/6100 [6:48:50<2:24:18,  5.45s/it]
 74%|███████▍  | 4511/6100 [6:48:56<2:24:03,  5.44s/it]
 74%|███████▍  | 4512/6100 [6:49:01<2:23:46,  5.43s/it]
 74%|███████▍  | 4513/6100 [6:49:07<2:23:33,  5.43s/it]
 74%|███████▍  | 4514/6100 [6:49:12<2:23:25,  5.43s/it]
 74%|███████▍  | 4515/6100 [6:49:17<2:23:15,  5.42s/it]
                                                       
{'loss': 0.072, 'learning_rate': 0.0001, 'epoch': 7.4}

 74%|███████▍  | 4515/6100 [6:49:17<2:23:15,  5.42s/it]
 74%|███████▍  | 4516/6100 [6:49:23<2:23:10,  5.42s/it]
 74%|███████▍  | 4517/6100 [6:49:28<2:23:05,  5.42s/it]
 74%|███████▍  | 4518/6100 [6:49:34<2:22:54,  5.42s/it]
 74%|███████▍  | 4519/6100 [6:49:39<2:22:45,  5.42s/it]
 74%|███████▍  | 4520/6100 [6:49:44<2:22:36,  5.42s/it]
                                                       
{'loss': 0.0704, 'learning_rate': 0.0001, 'epoch': 7.41}

 74%|███████▍  | 4520/6100 [6:49:45<2:22:36,  5.42s/it]
 74%|███████▍  | 4521/6100 [6:49:50<2:22:29,  5.41s/it]
 74%|███████▍  | 4522/6100 [6:49:55<2:22:27,  5.42s/it]
 74%|███████▍  | 4523/6100 [6:50:01<2:22:23,  5.42s/it]
 74%|███████▍  | 4524/6100 [6:50:06<2:22:17,  5.42s/it]
 74%|███████▍  | 4525/6100 [6:50:12<2:22:10,  5.42s/it]
                                                       
{'loss': 0.0715, 'learning_rate': 0.0001, 'epoch': 7.42}

 74%|███████▍  | 4525/6100 [6:50:12<2:22:10,  5.42s/it]
 74%|███████▍  | 4526/6100 [6:50:17<2:22:03,  5.42s/it]
 74%|███████▍  | 4527/6100 [6:50:22<2:21:53,  5.41s/it]
 74%|███████▍  | 4528/6100 [6:50:28<2:22:28,  5.44s/it]
 74%|███████▍  | 4529/6100 [6:50:33<2:22:17,  5.43s/it]
 74%|███████▍  | 4530/6100 [6:50:39<2:22:01,  5.43s/it]
                                                       
{'loss': 0.0705, 'learning_rate': 0.0001, 'epoch': 7.43}

 74%|███████▍  | 4530/6100 [6:50:39<2:22:01,  5.43s/it]
 74%|███████▍  | 4531/6100 [6:50:44<2:21:52,  5.43s/it]
 74%|███████▍  | 4532/6100 [6:50:50<2:21:51,  5.43s/it]
 74%|███████▍  | 4533/6100 [6:50:55<2:21:41,  5.43s/it]
 74%|███████▍  | 4534/6100 [6:51:00<2:21:29,  5.42s/it]
 74%|███████▍  | 4535/6100 [6:51:06<2:21:26,  5.42s/it]
                                                       
{'loss': 0.0731, 'learning_rate': 0.0001, 'epoch': 7.43}

 74%|███████▍  | 4535/6100 [6:51:06<2:21:26,  5.42s/it]
 74%|███████▍  | 4536/6100 [6:51:11<2:21:16,  5.42s/it]
 74%|███████▍  | 4537/6100 [6:51:17<2:21:15,  5.42s/it]
 74%|███████▍  | 4538/6100 [6:51:22<2:21:06,  5.42s/it]
 74%|███████▍  | 4539/6100 [6:51:28<2:20:59,  5.42s/it]
 74%|███████▍  | 4540/6100 [6:51:33<2:20:57,  5.42s/it]
                                                       
{'loss': 0.0667, 'learning_rate': 0.0001, 'epoch': 7.44}

 74%|███████▍  | 4540/6100 [6:51:33<2:20:57,  5.42s/it]
 74%|███████▍  | 4541/6100 [6:51:38<2:20:52,  5.42s/it]
 74%|███████▍  | 4542/6100 [6:51:44<2:20:45,  5.42s/it]
 74%|███████▍  | 4543/6100 [6:51:49<2:20:44,  5.42s/it]
 74%|███████▍  | 4544/6100 [6:51:55<2:20:42,  5.43s/it]
 75%|███████▍  | 4545/6100 [6:52:00<2:20:32,  5.42s/it]
                                                       
{'loss': 0.0702, 'learning_rate': 0.0001, 'epoch': 7.45}

 75%|███████▍  | 4545/6100 [6:52:00<2:20:32,  5.42s/it]
 75%|███████▍  | 4546/6100 [6:52:05<2:20:29,  5.42s/it]
 75%|███████▍  | 4547/6100 [6:52:11<2:20:28,  5.43s/it]
 75%|███████▍  | 4548/6100 [6:52:16<2:20:23,  5.43s/it]
 75%|███████▍  | 4549/6100 [6:52:22<2:20:18,  5.43s/it]
 75%|███████▍  | 4550/6100 [6:52:27<2:20:15,  5.43s/it]
                                                       
{'loss': 0.0706, 'learning_rate': 0.0001, 'epoch': 7.46}

 75%|███████▍  | 4550/6100 [6:52:27<2:20:15,  5.43s/it]
 75%|███████▍  | 4551/6100 [6:52:33<2:20:09,  5.43s/it]
 75%|███████▍  | 4552/6100 [6:52:38<2:20:08,  5.43s/it]
 75%|███████▍  | 4553/6100 [6:52:44<2:20:02,  5.43s/it]
 75%|███████▍  | 4554/6100 [6:52:49<2:19:49,  5.43s/it]
 75%|███████▍  | 4555/6100 [6:52:54<2:19:40,  5.42s/it]
                                                       
{'loss': 0.0675, 'learning_rate': 0.0001, 'epoch': 7.47}

 75%|███████▍  | 4555/6100 [6:52:54<2:19:40,  5.42s/it]
 75%|███████▍  | 4556/6100 [6:53:00<2:19:34,  5.42s/it]
 75%|███████▍  | 4557/6100 [6:53:05<2:19:27,  5.42s/it]
 75%|███████▍  | 4558/6100 [6:53:11<2:19:19,  5.42s/it]
 75%|███████▍  | 4559/6100 [6:53:16<2:19:13,  5.42s/it]
 75%|███████▍  | 4560/6100 [6:53:21<2:19:13,  5.42s/it]
                                                       
{'loss': 0.0672, 'learning_rate': 0.0001, 'epoch': 7.48}

 75%|███████▍  | 4560/6100 [6:53:21<2:19:13,  5.42s/it]
 75%|███████▍  | 4561/6100 [6:53:27<2:19:09,  5.42s/it]
 75%|███████▍  | 4562/6100 [6:53:32<2:19:09,  5.43s/it]
 75%|███████▍  | 4563/6100 [6:53:38<2:19:15,  5.44s/it]
 75%|███████▍  | 4564/6100 [6:53:43<2:19:07,  5.43s/it]
 75%|███████▍  | 4565/6100 [6:53:49<2:18:58,  5.43s/it]
                                                       
{'loss': 0.0678, 'learning_rate': 0.0001, 'epoch': 7.48}

 75%|███████▍  | 4565/6100 [6:53:49<2:18:58,  5.43s/it]
 75%|███████▍  | 4566/6100 [6:53:54<2:18:45,  5.43s/it]
 75%|███████▍  | 4567/6100 [6:53:59<2:18:29,  5.42s/it]
 75%|███████▍  | 4568/6100 [6:54:05<2:18:22,  5.42s/it]
 75%|███████▍  | 4569/6100 [6:54:10<2:19:09,  5.45s/it]
 75%|███████▍  | 4570/6100 [6:54:16<2:18:46,  5.44s/it]
                                                       
{'loss': 0.0691, 'learning_rate': 0.0001, 'epoch': 7.49}

 75%|███████▍  | 4570/6100 [6:54:16<2:18:46,  5.44s/it]
 75%|███████▍  | 4571/6100 [6:54:21<2:18:29,  5.43s/it]
 75%|███████▍  | 4572/6100 [6:54:27<2:18:11,  5.43s/it]
 75%|███████▍  | 4573/6100 [6:54:32<2:18:00,  5.42s/it]
 75%|███████▍  | 4574/6100 [6:54:37<2:17:59,  5.43s/it]
 75%|███████▌  | 4575/6100 [6:54:43<2:17:51,  5.42s/it]
                                                       
{'loss': 0.0687, 'learning_rate': 0.0001, 'epoch': 7.5}

 75%|███████▌  | 4575/6100 [6:54:43<2:17:51,  5.42s/it]
 75%|███████▌  | 4576/6100 [6:54:48<2:17:44,  5.42s/it]
 75%|███████▌  | 4577/6100 [6:54:54<2:17:35,  5.42s/it]
 75%|███████▌  | 4578/6100 [6:54:59<2:17:24,  5.42s/it]
 75%|███████▌  | 4579/6100 [6:55:05<2:17:18,  5.42s/it]
 75%|███████▌  | 4580/6100 [6:55:10<2:17:12,  5.42s/it]
                                                       
{'loss': 0.0669, 'learning_rate': 0.0001, 'epoch': 7.51}

 75%|███████▌  | 4580/6100 [6:55:10<2:17:12,  5.42s/it]
 75%|███████▌  | 4581/6100 [6:55:15<2:17:13,  5.42s/it]
 75%|███████▌  | 4582/6100 [6:55:21<2:17:07,  5.42s/it]
 75%|███████▌  | 4583/6100 [6:55:26<2:17:05,  5.42s/it]
 75%|███████▌  | 4584/6100 [6:55:32<2:16:57,  5.42s/it]
 75%|███████▌  | 4585/6100 [6:55:37<2:16:51,  5.42s/it]
                                                       
{'loss': 0.0707, 'learning_rate': 0.0001, 'epoch': 7.52}

 75%|███████▌  | 4585/6100 [6:55:37<2:16:51,  5.42s/it]
 75%|███████▌  | 4586/6100 [6:55:43<2:16:46,  5.42s/it]
 75%|███████▌  | 4587/6100 [6:55:48<2:16:36,  5.42s/it]
 75%|███████▌  | 4588/6100 [6:55:53<2:16:34,  5.42s/it]
 75%|███████▌  | 4589/6100 [6:55:59<2:16:34,  5.42s/it]
 75%|███████▌  | 4590/6100 [6:56:04<2:16:35,  5.43s/it]
                                                       
{'loss': 0.064, 'learning_rate': 0.0001, 'epoch': 7.52}

 75%|███████▌  | 4590/6100 [6:56:04<2:16:35,  5.43s/it]
 75%|███████▌  | 4591/6100 [6:56:10<2:16:29,  5.43s/it]
 75%|███████▌  | 4592/6100 [6:56:15<2:16:25,  5.43s/it]
 75%|███████▌  | 4593/6100 [6:56:21<2:16:15,  5.43s/it]
 75%|███████▌  | 4594/6100 [6:56:26<2:16:09,  5.42s/it]
 75%|███████▌  | 4595/6100 [6:56:31<2:16:01,  5.42s/it]
                                                       
{'loss': 0.0715, 'learning_rate': 0.0001, 'epoch': 7.53}

 75%|███████▌  | 4595/6100 [6:56:31<2:16:01,  5.42s/it]
 75%|███████▌  | 4596/6100 [6:56:37<2:16:04,  5.43s/it]
 75%|███████▌  | 4597/6100 [6:56:42<2:15:57,  5.43s/it]
 75%|███████▌  | 4598/6100 [6:56:48<2:15:53,  5.43s/it]
 75%|███████▌  | 4599/6100 [6:56:53<2:15:54,  5.43s/it]
 75%|███████▌  | 4600/6100 [6:56:58<2:15:40,  5.43s/it]
                                                       
{'loss': 0.0745, 'learning_rate': 0.0001, 'epoch': 7.54}

 75%|███████▌  | 4600/6100 [6:56:59<2:15:40,  5.43s/it]
 75%|███████▌  | 4601/6100 [6:57:04<2:15:39,  5.43s/it]
 75%|███████▌  | 4602/6100 [6:57:09<2:15:34,  5.43s/it]
 75%|███████▌  | 4603/6100 [6:57:15<2:15:21,  5.42s/it]
 75%|███████▌  | 4604/6100 [6:57:20<2:15:09,  5.42s/it]
 75%|███████▌  | 4605/6100 [6:57:26<2:15:02,  5.42s/it]
                                                       
{'loss': 0.0707, 'learning_rate': 0.0001, 'epoch': 7.55}

 75%|███████▌  | 4605/6100 [6:57:26<2:15:02,  5.42s/it]
 76%|███████▌  | 4606/6100 [6:57:31<2:14:59,  5.42s/it]
 76%|███████▌  | 4607/6100 [6:57:36<2:14:50,  5.42s/it]
 76%|███████▌  | 4608/6100 [6:57:42<2:14:46,  5.42s/it]
 76%|███████▌  | 4609/6100 [6:57:47<2:14:39,  5.42s/it]
 76%|███████▌  | 4610/6100 [6:57:53<2:14:28,  5.42s/it]
                                                       
{'loss': 0.0742, 'learning_rate': 0.0001, 'epoch': 7.56}

 76%|███████▌  | 4610/6100 [6:57:53<2:14:28,  5.42s/it]
 76%|███████▌  | 4611/6100 [6:57:58<2:14:21,  5.41s/it]
 76%|███████▌  | 4612/6100 [6:58:04<2:14:20,  5.42s/it]
 76%|███████▌  | 4613/6100 [6:58:09<2:14:18,  5.42s/it]
 76%|███████▌  | 4614/6100 [6:58:14<2:14:11,  5.42s/it]
 76%|███████▌  | 4615/6100 [6:58:20<2:14:04,  5.42s/it]
                                                       
{'loss': 0.0681, 'learning_rate': 0.0001, 'epoch': 7.57}

 76%|███████▌  | 4615/6100 [6:58:20<2:14:04,  5.42s/it]
 76%|███████▌  | 4616/6100 [6:58:25<2:14:02,  5.42s/it]
 76%|███████▌  | 4617/6100 [6:58:31<2:13:54,  5.42s/it]
 76%|███████▌  | 4618/6100 [6:58:36<2:13:49,  5.42s/it]
 76%|███████▌  | 4619/6100 [6:58:41<2:13:53,  5.42s/it]
 76%|███████▌  | 4620/6100 [6:58:47<2:13:55,  5.43s/it]
                                                       
{'loss': 0.0704, 'learning_rate': 0.0001, 'epoch': 7.57}

 76%|███████▌  | 4620/6100 [6:58:47<2:13:55,  5.43s/it]
 76%|███████▌  | 4621/6100 [6:58:52<2:13:46,  5.43s/it]
 76%|███████▌  | 4622/6100 [6:58:58<2:13:36,  5.42s/it]
 76%|███████▌  | 4623/6100 [6:59:03<2:13:24,  5.42s/it]
 76%|███████▌  | 4624/6100 [6:59:09<2:13:21,  5.42s/it]
 76%|███████▌  | 4625/6100 [6:59:14<2:13:19,  5.42s/it]
                                                       
{'loss': 0.0698, 'learning_rate': 0.0001, 'epoch': 7.58}

 76%|███████▌  | 4625/6100 [6:59:14<2:13:19,  5.42s/it]
 76%|███████▌  | 4626/6100 [6:59:19<2:13:11,  5.42s/it]
 76%|███████▌  | 4627/6100 [6:59:25<2:13:17,  5.43s/it]
 76%|███████▌  | 4628/6100 [6:59:30<2:13:11,  5.43s/it]
 76%|███████▌  | 4629/6100 [6:59:36<2:12:55,  5.42s/it]
 76%|███████▌  | 4630/6100 [6:59:41<2:12:58,  5.43s/it]
                                                       
{'loss': 0.0714, 'learning_rate': 0.0001, 'epoch': 7.59}

 76%|███████▌  | 4630/6100 [6:59:41<2:12:58,  5.43s/it]
 76%|███████▌  | 4631/6100 [6:59:47<2:12:47,  5.42s/it]
 76%|███████▌  | 4632/6100 [6:59:52<2:12:43,  5.42s/it]
 76%|███████▌  | 4633/6100 [6:59:57<2:12:27,  5.42s/it]
 76%|███████▌  | 4634/6100 [7:00:03<2:12:24,  5.42s/it]
 76%|███████▌  | 4635/6100 [7:00:08<2:12:24,  5.42s/it]
                                                       
{'loss': 0.0686, 'learning_rate': 0.0001, 'epoch': 7.6}

 76%|███████▌  | 4635/6100 [7:00:08<2:12:24,  5.42s/it]
 76%|███████▌  | 4636/6100 [7:00:14<2:12:16,  5.42s/it]
 76%|███████▌  | 4637/6100 [7:00:19<2:12:21,  5.43s/it]
 76%|███████▌  | 4638/6100 [7:00:25<2:12:11,  5.42s/it]
 76%|███████▌  | 4639/6100 [7:00:30<2:12:05,  5.42s/it]
 76%|███████▌  | 4640/6100 [7:00:35<2:11:59,  5.42s/it]
                                                       
{'loss': 0.0728, 'learning_rate': 0.0001, 'epoch': 7.61}

 76%|███████▌  | 4640/6100 [7:00:35<2:11:59,  5.42s/it]
 76%|███████▌  | 4641/6100 [7:00:41<2:11:49,  5.42s/it]
 76%|███████▌  | 4642/6100 [7:00:46<2:11:42,  5.42s/it]
 76%|███████▌  | 4643/6100 [7:00:52<2:11:35,  5.42s/it]
 76%|███████▌  | 4644/6100 [7:00:57<2:11:24,  5.41s/it]
 76%|███████▌  | 4645/6100 [7:01:02<2:11:22,  5.42s/it]
                                                       
{'loss': 0.0689, 'learning_rate': 0.0001, 'epoch': 7.61}

 76%|███████▌  | 4645/6100 [7:01:02<2:11:22,  5.42s/it]
 76%|███████▌  | 4646/6100 [7:01:08<2:11:12,  5.41s/it]
 76%|███████▌  | 4647/6100 [7:01:13<2:11:11,  5.42s/it]
 76%|███████▌  | 4648/6100 [7:01:19<2:11:02,  5.42s/it]
 76%|███████▌  | 4649/6100 [7:01:24<2:10:54,  5.41s/it]
 76%|███████▌  | 4650/6100 [7:01:30<2:10:47,  5.41s/it]
                                                       
{'loss': 0.0667, 'learning_rate': 0.0001, 'epoch': 7.62}

 76%|███████▌  | 4650/6100 [7:01:30<2:10:47,  5.41s/it]Saving model checkpoint to ./results/checkpoint-4650
Configuration saved in ./results/checkpoint-4650/config.json
Model weights saved in ./results/checkpoint-4650/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-4650/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-4650/special_tokens_map.json
[2023-02-22 02:53:45,687] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step4653 is begin to save!
[2023-02-22 02:53:45,691] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-4650/global_step4653/mp_rank_00_model_states.pt
[2023-02-22 02:53:45,691] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-4650/global_step4653/mp_rank_00_model_states.pt...
[2023-02-22 02:53:46,507] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-4650/global_step4653/mp_rank_00_model_states.pt.
[2023-02-22 02:53:46,509] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-4650/global_step4653/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 02:53:46,682] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-4650/global_step4653/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 02:53:46,682] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-4650/global_step4653/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 02:53:46,682] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4653 is ready now!
Deleting older checkpoint [results/checkpoint-4500] due to args.save_total_limit

 76%|███████▌  | 4651/6100 [7:01:37<2:26:13,  6.05s/it]
 76%|███████▋  | 4652/6100 [7:01:42<2:21:19,  5.86s/it]
 76%|███████▋  | 4653/6100 [7:01:48<2:17:55,  5.72s/it]
 76%|███████▋  | 4654/6100 [7:01:53<2:15:38,  5.63s/it]
 76%|███████▋  | 4655/6100 [7:01:59<2:13:57,  5.56s/it]
                                                       
{'loss': 0.0704, 'learning_rate': 0.0001, 'epoch': 7.63}

 76%|███████▋  | 4655/6100 [7:01:59<2:13:57,  5.56s/it]
 76%|███████▋  | 4656/6100 [7:02:04<2:12:44,  5.52s/it]
 76%|███████▋  | 4657/6100 [7:02:10<2:11:52,  5.48s/it]
 76%|███████▋  | 4658/6100 [7:02:15<2:11:15,  5.46s/it]
 76%|███████▋  | 4659/6100 [7:02:20<2:10:49,  5.45s/it]
 76%|███████▋  | 4660/6100 [7:02:26<2:10:23,  5.43s/it]
                                                       
{'loss': 0.0666, 'learning_rate': 0.0001, 'epoch': 7.64}

 76%|███████▋  | 4660/6100 [7:02:26<2:10:23,  5.43s/it]
 76%|███████▋  | 4661/6100 [7:02:31<2:10:11,  5.43s/it]
 76%|███████▋  | 4662/6100 [7:02:37<2:09:53,  5.42s/it]
 76%|███████▋  | 4663/6100 [7:02:42<2:09:39,  5.41s/it]
 76%|███████▋  | 4664/6100 [7:02:47<2:09:31,  5.41s/it]
 76%|███████▋  | 4665/6100 [7:02:53<2:09:25,  5.41s/it]
                                                       
{'loss': 0.0751, 'learning_rate': 0.0001, 'epoch': 7.65}

 76%|███████▋  | 4665/6100 [7:02:53<2:09:25,  5.41s/it]
 76%|███████▋  | 4666/6100 [7:02:58<2:09:17,  5.41s/it]
 77%|███████▋  | 4667/6100 [7:03:04<2:09:10,  5.41s/it]
 77%|███████▋  | 4668/6100 [7:03:09<2:09:07,  5.41s/it]
 77%|███████▋  | 4669/6100 [7:03:14<2:09:06,  5.41s/it]
 77%|███████▋  | 4670/6100 [7:03:20<2:09:01,  5.41s/it]
                                                       
{'loss': 0.0683, 'learning_rate': 0.0001, 'epoch': 7.66}

 77%|███████▋  | 4670/6100 [7:03:20<2:09:01,  5.41s/it]
 77%|███████▋  | 4671/6100 [7:03:25<2:08:57,  5.41s/it]
 77%|███████▋  | 4672/6100 [7:03:31<2:08:47,  5.41s/it]
 77%|███████▋  | 4673/6100 [7:03:36<2:08:48,  5.42s/it]
 77%|███████▋  | 4674/6100 [7:03:41<2:08:47,  5.42s/it]
 77%|███████▋  | 4675/6100 [7:03:47<2:08:37,  5.42s/it]
                                                       
{'loss': 0.064, 'learning_rate': 0.0001, 'epoch': 7.66}

 77%|███████▋  | 4675/6100 [7:03:47<2:08:37,  5.42s/it]
 77%|███████▋  | 4676/6100 [7:03:52<2:08:29,  5.41s/it]
 77%|███████▋  | 4677/6100 [7:03:58<2:08:22,  5.41s/it]
 77%|███████▋  | 4678/6100 [7:04:03<2:08:17,  5.41s/it]
 77%|███████▋  | 4679/6100 [7:04:09<2:08:11,  5.41s/it]
 77%|███████▋  | 4680/6100 [7:04:14<2:08:07,  5.41s/it]
                                                       
{'loss': 0.0675, 'learning_rate': 0.0001, 'epoch': 7.67}

 77%|███████▋  | 4680/6100 [7:04:14<2:08:07,  5.41s/it]
 77%|███████▋  | 4681/6100 [7:04:19<2:08:18,  5.43s/it]
 77%|███████▋  | 4682/6100 [7:04:25<2:08:03,  5.42s/it]
 77%|███████▋  | 4683/6100 [7:04:30<2:07:51,  5.41s/it]
 77%|███████▋  | 4684/6100 [7:04:36<2:07:45,  5.41s/it]
 77%|███████▋  | 4685/6100 [7:04:41<2:07:41,  5.41s/it]
                                                       
{'loss': 0.0717, 'learning_rate': 0.0001, 'epoch': 7.68}

 77%|███████▋  | 4685/6100 [7:04:41<2:07:41,  5.41s/it]
 77%|███████▋  | 4686/6100 [7:04:46<2:07:36,  5.41s/it]
 77%|███████▋  | 4687/6100 [7:04:52<2:07:29,  5.41s/it]
 77%|███████▋  | 4688/6100 [7:04:57<2:07:31,  5.42s/it]
 77%|███████▋  | 4689/6100 [7:05:03<2:08:17,  5.46s/it]
 77%|███████▋  | 4690/6100 [7:05:08<2:08:01,  5.45s/it]
                                                       
{'loss': 0.0725, 'learning_rate': 0.0001, 'epoch': 7.69}

 77%|███████▋  | 4690/6100 [7:05:08<2:08:01,  5.45s/it]
 77%|███████▋  | 4691/6100 [7:05:14<2:07:35,  5.43s/it]
 77%|███████▋  | 4692/6100 [7:05:19<2:07:14,  5.42s/it]
 77%|███████▋  | 4693/6100 [7:05:24<2:06:58,  5.42s/it]
 77%|███████▋  | 4694/6100 [7:05:30<2:06:58,  5.42s/it]
 77%|███████▋  | 4695/6100 [7:05:35<2:06:46,  5.41s/it]
                                                       
{'loss': 0.0666, 'learning_rate': 0.0001, 'epoch': 7.7}

 77%|███████▋  | 4695/6100 [7:05:35<2:06:46,  5.41s/it]
 77%|███████▋  | 4696/6100 [7:05:41<2:06:40,  5.41s/it]
 77%|███████▋  | 4697/6100 [7:05:46<2:06:32,  5.41s/it]
 77%|███████▋  | 4698/6100 [7:05:52<2:06:30,  5.41s/it]
 77%|███████▋  | 4699/6100 [7:05:57<2:06:27,  5.42s/it]
 77%|███████▋  | 4700/6100 [7:06:02<2:06:22,  5.42s/it]
                                                       
{'loss': 0.0678, 'learning_rate': 0.0001, 'epoch': 7.7}

 77%|███████▋  | 4700/6100 [7:06:02<2:06:22,  5.42s/it]
 77%|███████▋  | 4701/6100 [7:06:08<2:06:18,  5.42s/it]
 77%|███████▋  | 4702/6100 [7:06:13<2:08:10,  5.50s/it]
 77%|███████▋  | 4703/6100 [7:06:19<2:07:27,  5.47s/it]
 77%|███████▋  | 4704/6100 [7:06:24<2:06:54,  5.45s/it]
 77%|███████▋  | 4705/6100 [7:06:30<2:06:28,  5.44s/it]
                                                       
{'loss': 0.067, 'learning_rate': 0.0001, 'epoch': 7.71}

 77%|███████▋  | 4705/6100 [7:06:30<2:06:28,  5.44s/it]
 77%|███████▋  | 4706/6100 [7:06:35<2:06:11,  5.43s/it]
 77%|███████▋  | 4707/6100 [7:06:41<2:05:54,  5.42s/it]
 77%|███████▋  | 4708/6100 [7:06:46<2:05:51,  5.42s/it]
 77%|███████▋  | 4709/6100 [7:06:51<2:05:41,  5.42s/it]
 77%|███████▋  | 4710/6100 [7:06:57<2:05:31,  5.42s/it]
                                                       
{'loss': 0.0692, 'learning_rate': 0.0001, 'epoch': 7.72}

 77%|███████▋  | 4710/6100 [7:06:57<2:05:31,  5.42s/it]
 77%|███████▋  | 4711/6100 [7:07:02<2:05:27,  5.42s/it]
 77%|███████▋  | 4712/6100 [7:07:08<2:08:44,  5.56s/it]
 77%|███████▋  | 4713/6100 [7:07:14<2:07:33,  5.52s/it]
 77%|███████▋  | 4714/6100 [7:07:19<2:06:41,  5.48s/it]
 77%|███████▋  | 4715/6100 [7:07:24<2:06:11,  5.47s/it]
                                                       
{'loss': 0.0717, 'learning_rate': 0.0001, 'epoch': 7.73}

 77%|███████▋  | 4715/6100 [7:07:24<2:06:11,  5.47s/it]
 77%|███████▋  | 4716/6100 [7:07:30<2:05:42,  5.45s/it]
 77%|███████▋  | 4717/6100 [7:07:35<2:05:23,  5.44s/it]
 77%|███████▋  | 4718/6100 [7:07:41<2:05:03,  5.43s/it]
 77%|███████▋  | 4719/6100 [7:07:46<2:04:50,  5.42s/it]
 77%|███████▋  | 4720/6100 [7:07:51<2:04:38,  5.42s/it]
                                                       
{'loss': 0.069, 'learning_rate': 0.0001, 'epoch': 7.74}

 77%|███████▋  | 4720/6100 [7:07:51<2:04:38,  5.42s/it]
 77%|███████▋  | 4721/6100 [7:07:57<2:04:41,  5.43s/it]
 77%|███████▋  | 4722/6100 [7:08:02<2:04:25,  5.42s/it]
 77%|███████▋  | 4723/6100 [7:08:08<2:04:14,  5.41s/it]
 77%|███████▋  | 4724/6100 [7:08:13<2:04:07,  5.41s/it]
 77%|███████▋  | 4725/6100 [7:08:18<2:04:01,  5.41s/it]
                                                       
{'loss': 0.0651, 'learning_rate': 0.0001, 'epoch': 7.75}

 77%|███████▋  | 4725/6100 [7:08:18<2:04:01,  5.41s/it]
 77%|███████▋  | 4726/6100 [7:08:24<2:03:55,  5.41s/it]
 77%|███████▋  | 4727/6100 [7:08:29<2:03:48,  5.41s/it]
 78%|███████▊  | 4728/6100 [7:08:35<2:03:39,  5.41s/it]
 78%|███████▊  | 4729/6100 [7:08:40<2:03:33,  5.41s/it]
 78%|███████▊  | 4730/6100 [7:08:46<2:03:39,  5.42s/it]
                                                       
{'loss': 0.072, 'learning_rate': 0.0001, 'epoch': 7.75}

 78%|███████▊  | 4730/6100 [7:08:46<2:03:39,  5.42s/it]
 78%|███████▊  | 4731/6100 [7:08:51<2:03:31,  5.41s/it]
 78%|███████▊  | 4732/6100 [7:08:56<2:03:21,  5.41s/it]
 78%|███████▊  | 4733/6100 [7:09:02<2:03:16,  5.41s/it]
 78%|███████▊  | 4734/6100 [7:09:07<2:03:15,  5.41s/it]
 78%|███████▊  | 4735/6100 [7:09:13<2:03:14,  5.42s/it]
                                                       
{'loss': 0.0722, 'learning_rate': 0.0001, 'epoch': 7.76}

 78%|███████▊  | 4735/6100 [7:09:13<2:03:14,  5.42s/it]
 78%|███████▊  | 4736/6100 [7:09:18<2:03:09,  5.42s/it]
 78%|███████▊  | 4737/6100 [7:09:23<2:03:26,  5.43s/it]
 78%|███████▊  | 4738/6100 [7:09:29<2:03:09,  5.43s/it]
 78%|███████▊  | 4739/6100 [7:09:34<2:02:59,  5.42s/it]
 78%|███████▊  | 4740/6100 [7:09:40<2:02:46,  5.42s/it]
                                                       
{'loss': 0.0681, 'learning_rate': 0.0001, 'epoch': 7.77}

 78%|███████▊  | 4740/6100 [7:09:40<2:02:46,  5.42s/it]
 78%|███████▊  | 4741/6100 [7:09:45<2:02:40,  5.42s/it]
 78%|███████▊  | 4742/6100 [7:09:51<2:02:35,  5.42s/it]
 78%|███████▊  | 4743/6100 [7:09:56<2:02:25,  5.41s/it]
 78%|███████▊  | 4744/6100 [7:10:01<2:02:11,  5.41s/it]
 78%|███████▊  | 4745/6100 [7:10:07<2:02:07,  5.41s/it]
                                                       
{'loss': 0.0702, 'learning_rate': 0.0001, 'epoch': 7.78}

 78%|███████▊  | 4745/6100 [7:10:07<2:02:07,  5.41s/it]
 78%|███████▊  | 4746/6100 [7:10:12<2:02:08,  5.41s/it]
 78%|███████▊  | 4747/6100 [7:10:18<2:02:04,  5.41s/it]
 78%|███████▊  | 4748/6100 [7:10:23<2:02:01,  5.41s/it]
 78%|███████▊  | 4749/6100 [7:10:28<2:01:55,  5.42s/it]
 78%|███████▊  | 4750/6100 [7:10:34<2:01:48,  5.41s/it]
                                                       
{'loss': 0.0695, 'learning_rate': 0.0001, 'epoch': 7.79}

 78%|███████▊  | 4750/6100 [7:10:34<2:01:48,  5.41s/it]
 78%|███████▊  | 4751/6100 [7:10:39<2:01:43,  5.41s/it]
 78%|███████▊  | 4752/6100 [7:10:45<2:01:39,  5.42s/it]
 78%|███████▊  | 4753/6100 [7:10:51<2:05:32,  5.59s/it]
 78%|███████▊  | 4754/6100 [7:10:56<2:04:12,  5.54s/it]
 78%|███████▊  | 4755/6100 [7:11:02<2:03:21,  5.50s/it]
                                                       
{'loss': 0.0651, 'learning_rate': 0.0001, 'epoch': 7.79}

 78%|███████▊  | 4755/6100 [7:11:02<2:03:21,  5.50s/it]
 78%|███████▊  | 4756/6100 [7:11:07<2:02:42,  5.48s/it]
 78%|███████▊  | 4757/6100 [7:11:12<2:02:22,  5.47s/it]
 78%|███████▊  | 4758/6100 [7:11:18<2:01:50,  5.45s/it]
 78%|███████▊  | 4759/6100 [7:11:23<2:01:40,  5.44s/it]
 78%|███████▊  | 4760/6100 [7:11:29<2:01:26,  5.44s/it]
                                                       
{'loss': 0.07, 'learning_rate': 0.0001, 'epoch': 7.8}

 78%|███████▊  | 4760/6100 [7:11:29<2:01:26,  5.44s/it]
 78%|███████▊  | 4761/6100 [7:11:34<2:01:04,  5.43s/it]
 78%|███████▊  | 4762/6100 [7:11:39<2:00:54,  5.42s/it]
 78%|███████▊  | 4763/6100 [7:11:45<2:00:43,  5.42s/it]
 78%|███████▊  | 4764/6100 [7:11:50<2:00:31,  5.41s/it]
 78%|███████▊  | 4765/6100 [7:11:56<2:00:19,  5.41s/it]
                                                       
{'loss': 0.0741, 'learning_rate': 0.0001, 'epoch': 7.81}

 78%|███████▊  | 4765/6100 [7:11:56<2:00:19,  5.41s/it]
 78%|███████▊  | 4766/6100 [7:12:01<2:00:18,  5.41s/it]
 78%|███████▊  | 4767/6100 [7:12:06<2:00:20,  5.42s/it]
 78%|███████▊  | 4768/6100 [7:12:12<2:01:44,  5.48s/it]
 78%|███████▊  | 4769/6100 [7:12:18<2:01:07,  5.46s/it]
 78%|███████▊  | 4770/6100 [7:12:23<2:00:40,  5.44s/it]
                                                       
{'loss': 0.072, 'learning_rate': 0.0001, 'epoch': 7.82}

 78%|███████▊  | 4770/6100 [7:12:23<2:00:40,  5.44s/it]
 78%|███████▊  | 4771/6100 [7:12:28<2:00:16,  5.43s/it]
 78%|███████▊  | 4772/6100 [7:12:34<2:00:04,  5.43s/it]
 78%|███████▊  | 4773/6100 [7:12:39<1:59:52,  5.42s/it]
 78%|███████▊  | 4774/6100 [7:12:45<1:59:46,  5.42s/it]
 78%|███████▊  | 4775/6100 [7:12:50<1:59:42,  5.42s/it]
                                                       
{'loss': 0.0671, 'learning_rate': 0.0001, 'epoch': 7.83}

 78%|███████▊  | 4775/6100 [7:12:50<1:59:42,  5.42s/it]
 78%|███████▊  | 4776/6100 [7:12:55<1:59:40,  5.42s/it]
 78%|███████▊  | 4777/6100 [7:13:01<1:59:33,  5.42s/it]
 78%|███████▊  | 4778/6100 [7:13:06<1:59:28,  5.42s/it]
 78%|███████▊  | 4779/6100 [7:13:12<1:59:18,  5.42s/it]
 78%|███████▊  | 4780/6100 [7:13:17<1:59:14,  5.42s/it]
                                                       
{'loss': 0.0732, 'learning_rate': 0.0001, 'epoch': 7.84}

 78%|███████▊  | 4780/6100 [7:13:17<1:59:14,  5.42s/it]
 78%|███████▊  | 4781/6100 [7:13:23<1:59:07,  5.42s/it]
 78%|███████▊  | 4782/6100 [7:13:28<1:58:58,  5.42s/it]
 78%|███████▊  | 4783/6100 [7:13:33<1:58:51,  5.41s/it]
 78%|███████▊  | 4784/6100 [7:13:39<1:58:48,  5.42s/it]
 78%|███████▊  | 4785/6100 [7:13:44<1:58:42,  5.42s/it]
                                                       
{'loss': 0.074, 'learning_rate': 0.0001, 'epoch': 7.84}

 78%|███████▊  | 4785/6100 [7:13:44<1:58:42,  5.42s/it]
 78%|███████▊  | 4786/6100 [7:13:50<1:58:38,  5.42s/it]
 78%|███████▊  | 4787/6100 [7:13:55<1:58:29,  5.41s/it]
 78%|███████▊  | 4788/6100 [7:14:00<1:58:30,  5.42s/it]
 79%|███████▊  | 4789/6100 [7:14:06<1:58:19,  5.42s/it]
 79%|███████▊  | 4790/6100 [7:14:11<1:58:17,  5.42s/it]
                                                       
{'loss': 0.0657, 'learning_rate': 0.0001, 'epoch': 7.85}

 79%|███████▊  | 4790/6100 [7:14:11<1:58:17,  5.42s/it]
 79%|███████▊  | 4791/6100 [7:14:17<1:58:12,  5.42s/it]
 79%|███████▊  | 4792/6100 [7:14:22<1:58:06,  5.42s/it]
 79%|███████▊  | 4793/6100 [7:14:28<1:57:59,  5.42s/it]
 79%|███████▊  | 4794/6100 [7:14:33<1:57:55,  5.42s/it]
 79%|███████▊  | 4795/6100 [7:14:38<1:57:49,  5.42s/it]
                                                       
{'loss': 0.0715, 'learning_rate': 0.0001, 'epoch': 7.86}

 79%|███████▊  | 4795/6100 [7:14:38<1:57:49,  5.42s/it]
 79%|███████▊  | 4796/6100 [7:14:44<1:57:50,  5.42s/it]
 79%|███████▊  | 4797/6100 [7:14:49<1:57:38,  5.42s/it]
 79%|███████▊  | 4798/6100 [7:14:55<1:57:36,  5.42s/it]
 79%|███████▊  | 4799/6100 [7:15:00<1:57:34,  5.42s/it]
 79%|███████▊  | 4800/6100 [7:15:05<1:57:27,  5.42s/it]
                                                       
{'loss': 0.0703, 'learning_rate': 0.0001, 'epoch': 7.87}

 79%|███████▊  | 4800/6100 [7:15:05<1:57:27,  5.42s/it]Saving model checkpoint to ./results/checkpoint-4800
Configuration saved in ./results/checkpoint-4800/config.json
Model weights saved in ./results/checkpoint-4800/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-4800/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-4800/special_tokens_map.json
[2023-02-22 03:07:21,662] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step4803 is begin to save!
[2023-02-22 03:07:21,666] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-4800/global_step4803/mp_rank_00_model_states.pt
[2023-02-22 03:07:21,666] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-4800/global_step4803/mp_rank_00_model_states.pt...
[2023-02-22 03:07:22,495] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-4800/global_step4803/mp_rank_00_model_states.pt.
[2023-02-22 03:07:22,497] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-4800/global_step4803/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 03:07:22,672] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-4800/global_step4803/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 03:07:22,672] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-4800/global_step4803/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 03:07:22,672] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4803 is ready now!
Deleting older checkpoint [results/checkpoint-4650] due to args.save_total_limit

 79%|███████▊  | 4801/6100 [7:15:13<2:11:21,  6.07s/it]
 79%|███████▊  | 4802/6100 [7:15:18<2:06:55,  5.87s/it]
 79%|███████▊  | 4803/6100 [7:15:24<2:03:56,  5.73s/it]
 79%|███████▉  | 4804/6100 [7:15:29<2:01:43,  5.64s/it]
 79%|███████▉  | 4805/6100 [7:15:35<2:00:13,  5.57s/it]
                                                       
{'loss': 0.0644, 'learning_rate': 0.0001, 'epoch': 7.88}

 79%|███████▉  | 4805/6100 [7:15:35<2:00:13,  5.57s/it]
 79%|███████▉  | 4806/6100 [7:15:40<1:59:06,  5.52s/it]
 79%|███████▉  | 4807/6100 [7:15:46<1:58:17,  5.49s/it]
 79%|███████▉  | 4808/6100 [7:15:51<1:57:40,  5.46s/it]
 79%|███████▉  | 4809/6100 [7:15:56<1:57:17,  5.45s/it]
 79%|███████▉  | 4810/6100 [7:16:02<1:56:54,  5.44s/it]
                                                       
{'loss': 0.0733, 'learning_rate': 0.0001, 'epoch': 7.88}

 79%|███████▉  | 4810/6100 [7:16:02<1:56:54,  5.44s/it]
 79%|███████▉  | 4811/6100 [7:16:07<1:56:34,  5.43s/it]
 79%|███████▉  | 4812/6100 [7:16:13<1:56:21,  5.42s/it]
 79%|███████▉  | 4813/6100 [7:16:18<1:56:07,  5.41s/it]
 79%|███████▉  | 4814/6100 [7:16:23<1:55:57,  5.41s/it]
 79%|███████▉  | 4815/6100 [7:16:29<1:55:51,  5.41s/it]
                                                       
{'loss': 0.0721, 'learning_rate': 0.0001, 'epoch': 7.89}

 79%|███████▉  | 4815/6100 [7:16:29<1:55:51,  5.41s/it]
 79%|███████▉  | 4816/6100 [7:16:34<1:55:46,  5.41s/it]
 79%|███████▉  | 4817/6100 [7:16:40<1:55:43,  5.41s/it]
 79%|███████▉  | 4818/6100 [7:16:45<1:55:35,  5.41s/it]
 79%|███████▉  | 4819/6100 [7:16:50<1:55:28,  5.41s/it]
 79%|███████▉  | 4820/6100 [7:16:56<1:55:29,  5.41s/it]
                                                       
{'loss': 0.0662, 'learning_rate': 0.0001, 'epoch': 7.9}

 79%|███████▉  | 4820/6100 [7:16:56<1:55:29,  5.41s/it]
 79%|███████▉  | 4821/6100 [7:17:01<1:55:24,  5.41s/it]
 79%|███████▉  | 4822/6100 [7:17:07<1:55:15,  5.41s/it]
 79%|███████▉  | 4823/6100 [7:17:12<1:55:08,  5.41s/it]
 79%|███████▉  | 4824/6100 [7:17:17<1:55:07,  5.41s/it]
 79%|███████▉  | 4825/6100 [7:17:23<1:55:06,  5.42s/it]
                                                       
{'loss': 0.0685, 'learning_rate': 0.0001, 'epoch': 7.91}

 79%|███████▉  | 4825/6100 [7:17:23<1:55:06,  5.42s/it]
 79%|███████▉  | 4826/6100 [7:17:28<1:54:57,  5.41s/it]
 79%|███████▉  | 4827/6100 [7:17:34<1:54:50,  5.41s/it]
 79%|███████▉  | 4828/6100 [7:17:39<1:54:50,  5.42s/it]
 79%|███████▉  | 4829/6100 [7:17:45<1:54:48,  5.42s/it]
 79%|███████▉  | 4830/6100 [7:17:50<1:54:41,  5.42s/it]
                                                       
{'loss': 0.0671, 'learning_rate': 0.0001, 'epoch': 7.92}

 79%|███████▉  | 4830/6100 [7:17:50<1:54:41,  5.42s/it]
 79%|███████▉  | 4831/6100 [7:17:55<1:54:44,  5.43s/it]
 79%|███████▉  | 4832/6100 [7:18:01<1:54:29,  5.42s/it]
 79%|███████▉  | 4833/6100 [7:18:06<1:54:30,  5.42s/it]
 79%|███████▉  | 4834/6100 [7:18:12<1:54:24,  5.42s/it]
 79%|███████▉  | 4835/6100 [7:18:17<1:54:16,  5.42s/it]
                                                       
{'loss': 0.068, 'learning_rate': 0.0001, 'epoch': 7.93}

 79%|███████▉  | 4835/6100 [7:18:17<1:54:16,  5.42s/it]
 79%|███████▉  | 4836/6100 [7:18:23<1:54:09,  5.42s/it]
 79%|███████▉  | 4837/6100 [7:18:28<1:54:04,  5.42s/it]
 79%|███████▉  | 4838/6100 [7:18:33<1:53:56,  5.42s/it]
 79%|███████▉  | 4839/6100 [7:18:39<1:53:48,  5.42s/it]
 79%|███████▉  | 4840/6100 [7:18:44<1:53:39,  5.41s/it]
                                                       
{'loss': 0.0724, 'learning_rate': 0.0001, 'epoch': 7.93}

 79%|███████▉  | 4840/6100 [7:18:44<1:53:39,  5.41s/it]
 79%|███████▉  | 4841/6100 [7:18:50<1:53:36,  5.41s/it]
 79%|███████▉  | 4842/6100 [7:18:55<1:53:29,  5.41s/it]
 79%|███████▉  | 4843/6100 [7:19:00<1:53:27,  5.42s/it]
 79%|███████▉  | 4844/6100 [7:19:06<1:53:17,  5.41s/it]
 79%|███████▉  | 4845/6100 [7:19:11<1:53:11,  5.41s/it]
                                                       
{'loss': 0.0642, 'learning_rate': 0.0001, 'epoch': 7.94}

 79%|███████▉  | 4845/6100 [7:19:11<1:53:11,  5.41s/it]
 79%|███████▉  | 4846/6100 [7:19:17<1:53:06,  5.41s/it]
 79%|███████▉  | 4847/6100 [7:19:22<1:53:04,  5.41s/it]
 79%|███████▉  | 4848/6100 [7:19:27<1:52:58,  5.41s/it]
 79%|███████▉  | 4849/6100 [7:19:33<1:52:53,  5.41s/it]
 80%|███████▉  | 4850/6100 [7:19:38<1:52:48,  5.41s/it]
                                                       
{'loss': 0.0666, 'learning_rate': 0.0001, 'epoch': 7.95}

 80%|███████▉  | 4850/6100 [7:19:38<1:52:48,  5.41s/it]
 80%|███████▉  | 4851/6100 [7:19:44<1:52:38,  5.41s/it]
 80%|███████▉  | 4852/6100 [7:19:49<1:52:33,  5.41s/it]
 80%|███████▉  | 4853/6100 [7:19:55<1:52:33,  5.42s/it]
 80%|███████▉  | 4854/6100 [7:20:00<1:52:31,  5.42s/it]
 80%|███████▉  | 4855/6100 [7:20:05<1:52:22,  5.42s/it]
                                                       
{'loss': 0.0715, 'learning_rate': 0.0001, 'epoch': 7.96}

 80%|███████▉  | 4855/6100 [7:20:05<1:52:22,  5.42s/it]
 80%|███████▉  | 4856/6100 [7:20:11<1:52:16,  5.42s/it]
 80%|███████▉  | 4857/6100 [7:20:16<1:52:21,  5.42s/it]
 80%|███████▉  | 4858/6100 [7:20:22<1:52:23,  5.43s/it]
 80%|███████▉  | 4859/6100 [7:20:27<1:52:12,  5.43s/it]
 80%|███████▉  | 4860/6100 [7:20:33<1:52:02,  5.42s/it]
                                                       
{'loss': 0.0641, 'learning_rate': 0.0001, 'epoch': 7.97}

 80%|███████▉  | 4860/6100 [7:20:33<1:52:02,  5.42s/it]
 80%|███████▉  | 4861/6100 [7:20:38<1:51:52,  5.42s/it]
 80%|███████▉  | 4862/6100 [7:20:43<1:51:44,  5.42s/it]
 80%|███████▉  | 4863/6100 [7:20:49<1:51:39,  5.42s/it]
 80%|███████▉  | 4864/6100 [7:20:54<1:51:27,  5.41s/it]
 80%|███████▉  | 4865/6100 [7:21:00<1:51:25,  5.41s/it]
                                                       
{'loss': 0.0639, 'learning_rate': 0.0001, 'epoch': 7.97}

 80%|███████▉  | 4865/6100 [7:21:00<1:51:25,  5.41s/it]
 80%|███████▉  | 4866/6100 [7:21:05<1:51:23,  5.42s/it]
 80%|███████▉  | 4867/6100 [7:21:10<1:51:15,  5.41s/it]
 80%|███████▉  | 4868/6100 [7:21:16<1:51:06,  5.41s/it]
 80%|███████▉  | 4869/6100 [7:21:21<1:50:58,  5.41s/it]
 80%|███████▉  | 4870/6100 [7:21:27<1:50:55,  5.41s/it]
                                                       
{'loss': 0.0654, 'learning_rate': 0.0001, 'epoch': 7.98}

 80%|███████▉  | 4870/6100 [7:21:27<1:50:55,  5.41s/it]
 80%|███████▉  | 4871/6100 [7:21:32<1:51:51,  5.46s/it]
 80%|███████▉  | 4872/6100 [7:21:38<1:51:23,  5.44s/it]
 80%|███████▉  | 4873/6100 [7:21:43<1:51:10,  5.44s/it]
 80%|███████▉  | 4874/6100 [7:21:48<1:51:00,  5.43s/it]
 80%|███████▉  | 4875/6100 [7:21:54<1:50:46,  5.43s/it]
                                                       
{'loss': 0.0726, 'learning_rate': 0.0001, 'epoch': 7.99}

 80%|███████▉  | 4875/6100 [7:21:54<1:50:46,  5.43s/it]
 80%|███████▉  | 4876/6100 [7:21:59<1:50:38,  5.42s/it]
 80%|███████▉  | 4877/6100 [7:22:05<1:50:25,  5.42s/it]
 80%|███████▉  | 4878/6100 [7:22:10<1:50:19,  5.42s/it]
 80%|███████▉  | 4879/6100 [7:22:16<1:50:14,  5.42s/it]
 80%|████████  | 4880/6100 [7:22:21<1:50:03,  5.41s/it]
                                                       
{'loss': 0.0671, 'learning_rate': 0.0001, 'epoch': 8.0}

 80%|████████  | 4880/6100 [7:22:21<1:50:03,  5.41s/it]
 80%|████████  | 4881/6100 [7:22:29<2:07:43,  6.29s/it]
 80%|████████  | 4882/6100 [7:22:35<2:02:19,  6.03s/it]
 80%|████████  | 4883/6100 [7:22:40<1:58:34,  5.85s/it]
 80%|████████  | 4884/6100 [7:22:45<1:55:50,  5.72s/it]
 80%|████████  | 4885/6100 [7:22:51<1:53:51,  5.62s/it]
                                                       
{'loss': 0.0712, 'learning_rate': 0.0001, 'epoch': 8.01}

 80%|████████  | 4885/6100 [7:22:51<1:53:51,  5.62s/it]
 80%|████████  | 4886/6100 [7:22:56<1:52:26,  5.56s/it]
 80%|████████  | 4887/6100 [7:23:02<1:51:25,  5.51s/it]
 80%|████████  | 4888/6100 [7:23:07<1:50:45,  5.48s/it]
 80%|████████  | 4889/6100 [7:23:13<1:50:16,  5.46s/it]
 80%|████████  | 4890/6100 [7:23:18<1:50:42,  5.49s/it]
                                                       
{'loss': 0.0672, 'learning_rate': 0.0001, 'epoch': 8.02}

 80%|████████  | 4890/6100 [7:23:18<1:50:42,  5.49s/it]
 80%|████████  | 4891/6100 [7:23:24<1:50:07,  5.47s/it]
 80%|████████  | 4892/6100 [7:23:29<1:49:41,  5.45s/it]
 80%|████████  | 4893/6100 [7:23:34<1:49:24,  5.44s/it]
 80%|████████  | 4894/6100 [7:23:40<1:49:14,  5.43s/it]
 80%|████████  | 4895/6100 [7:23:45<1:48:58,  5.43s/it]
                                                       
{'loss': 0.0685, 'learning_rate': 0.0001, 'epoch': 8.02}

 80%|████████  | 4895/6100 [7:23:45<1:48:58,  5.43s/it]
 80%|████████  | 4896/6100 [7:23:51<1:48:48,  5.42s/it]
 80%|████████  | 4897/6100 [7:23:56<1:48:41,  5.42s/it]
 80%|████████  | 4898/6100 [7:24:01<1:48:38,  5.42s/it]
 80%|████████  | 4899/6100 [7:24:07<1:48:30,  5.42s/it]
 80%|████████  | 4900/6100 [7:24:12<1:48:26,  5.42s/it]
                                                       
{'loss': 0.0686, 'learning_rate': 0.0001, 'epoch': 8.03}

 80%|████████  | 4900/6100 [7:24:12<1:48:26,  5.42s/it]
 80%|████████  | 4901/6100 [7:24:18<1:48:17,  5.42s/it]
 80%|████████  | 4902/6100 [7:24:23<1:48:13,  5.42s/it]
 80%|████████  | 4903/6100 [7:24:29<1:48:05,  5.42s/it]
 80%|████████  | 4904/6100 [7:24:34<1:48:00,  5.42s/it]
 80%|████████  | 4905/6100 [7:24:39<1:47:55,  5.42s/it]
                                                       
{'loss': 0.0626, 'learning_rate': 0.0001, 'epoch': 8.04}

 80%|████████  | 4905/6100 [7:24:39<1:47:55,  5.42s/it]
 80%|████████  | 4906/6100 [7:24:45<1:47:46,  5.42s/it]
 80%|████████  | 4907/6100 [7:24:50<1:47:36,  5.41s/it]
 80%|████████  | 4908/6100 [7:24:56<1:47:30,  5.41s/it]
 80%|████████  | 4909/6100 [7:25:01<1:47:28,  5.41s/it]
 80%|████████  | 4910/6100 [7:25:06<1:47:24,  5.42s/it]
                                                       
{'loss': 0.071, 'learning_rate': 0.0001, 'epoch': 8.05}

 80%|████████  | 4910/6100 [7:25:06<1:47:24,  5.42s/it]
 81%|████████  | 4911/6100 [7:25:12<1:47:22,  5.42s/it]
 81%|████████  | 4912/6100 [7:25:17<1:47:16,  5.42s/it]
 81%|████████  | 4913/6100 [7:25:23<1:47:08,  5.42s/it]
 81%|████████  | 4914/6100 [7:25:28<1:47:04,  5.42s/it]
 81%|████████  | 4915/6100 [7:25:34<1:47:03,  5.42s/it]
                                                       
{'loss': 0.0696, 'learning_rate': 0.0001, 'epoch': 8.06}

 81%|████████  | 4915/6100 [7:25:34<1:47:03,  5.42s/it]
 81%|████████  | 4916/6100 [7:25:39<1:46:57,  5.42s/it]
 81%|████████  | 4917/6100 [7:25:44<1:46:47,  5.42s/it]
 81%|████████  | 4918/6100 [7:25:50<1:46:41,  5.42s/it]
 81%|████████  | 4919/6100 [7:25:55<1:46:36,  5.42s/it]
 81%|████████  | 4920/6100 [7:26:01<1:46:36,  5.42s/it]
                                                       
{'loss': 0.0689, 'learning_rate': 0.0001, 'epoch': 8.07}

 81%|████████  | 4920/6100 [7:26:01<1:46:36,  5.42s/it]
 81%|████████  | 4921/6100 [7:26:06<1:46:27,  5.42s/it]
 81%|████████  | 4922/6100 [7:26:11<1:46:15,  5.41s/it]
 81%|████████  | 4923/6100 [7:26:17<1:46:09,  5.41s/it]
 81%|████████  | 4924/6100 [7:26:22<1:46:05,  5.41s/it]
 81%|████████  | 4925/6100 [7:26:28<1:45:56,  5.41s/it]
                                                       
{'loss': 0.0667, 'learning_rate': 0.0001, 'epoch': 8.07}

 81%|████████  | 4925/6100 [7:26:28<1:45:56,  5.41s/it]
 81%|████████  | 4926/6100 [7:26:33<1:45:52,  5.41s/it]
 81%|████████  | 4927/6100 [7:26:38<1:45:51,  5.41s/it]
 81%|████████  | 4928/6100 [7:26:44<1:45:43,  5.41s/it]
 81%|████████  | 4929/6100 [7:26:49<1:45:39,  5.41s/it]
 81%|████████  | 4930/6100 [7:26:55<1:45:36,  5.42s/it]
                                                       
{'loss': 0.0621, 'learning_rate': 0.0001, 'epoch': 8.08}

 81%|████████  | 4930/6100 [7:26:55<1:45:36,  5.42s/it]
 81%|████████  | 4931/6100 [7:27:00<1:45:39,  5.42s/it]
 81%|████████  | 4932/6100 [7:27:06<1:45:31,  5.42s/it]
 81%|████████  | 4933/6100 [7:27:11<1:45:22,  5.42s/it]
 81%|████████  | 4934/6100 [7:27:16<1:45:12,  5.41s/it]
 81%|████████  | 4935/6100 [7:27:22<1:45:08,  5.41s/it]
                                                       
{'loss': 0.0648, 'learning_rate': 0.0001, 'epoch': 8.09}

 81%|████████  | 4935/6100 [7:27:22<1:45:08,  5.41s/it]
 81%|████████  | 4936/6100 [7:27:27<1:45:03,  5.42s/it]
 81%|████████  | 4937/6100 [7:27:33<1:44:59,  5.42s/it]
 81%|████████  | 4938/6100 [7:27:38<1:44:56,  5.42s/it]
 81%|████████  | 4939/6100 [7:27:43<1:44:50,  5.42s/it]
 81%|████████  | 4940/6100 [7:27:49<1:44:48,  5.42s/it]
                                                       
{'loss': 0.0694, 'learning_rate': 0.0001, 'epoch': 8.1}

 81%|████████  | 4940/6100 [7:27:49<1:44:48,  5.42s/it]
 81%|████████  | 4941/6100 [7:27:54<1:44:41,  5.42s/it]
 81%|████████  | 4942/6100 [7:28:00<1:44:33,  5.42s/it]
 81%|████████  | 4943/6100 [7:28:05<1:44:29,  5.42s/it]
 81%|████████  | 4944/6100 [7:28:11<1:44:21,  5.42s/it]
 81%|████████  | 4945/6100 [7:28:16<1:44:15,  5.42s/it]
                                                       
{'loss': 0.062, 'learning_rate': 0.0001, 'epoch': 8.11}

 81%|████████  | 4945/6100 [7:28:16<1:44:15,  5.42s/it]
 81%|████████  | 4946/6100 [7:28:21<1:44:07,  5.41s/it]
 81%|████████  | 4947/6100 [7:28:27<1:44:02,  5.41s/it]
 81%|████████  | 4948/6100 [7:28:32<1:44:04,  5.42s/it]
 81%|████████  | 4949/6100 [7:28:38<1:43:59,  5.42s/it]
 81%|████████  | 4950/6100 [7:28:43<1:43:50,  5.42s/it]
                                                       
{'loss': 0.0652, 'learning_rate': 0.0001, 'epoch': 8.11}

 81%|████████  | 4950/6100 [7:28:43<1:43:50,  5.42s/it]Saving model checkpoint to ./results/checkpoint-4950
Configuration saved in ./results/checkpoint-4950/config.json
Model weights saved in ./results/checkpoint-4950/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-4950/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-4950/special_tokens_map.json
[2023-02-22 03:20:59,263] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step4954 is begin to save!
[2023-02-22 03:20:59,266] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-4950/global_step4954/mp_rank_00_model_states.pt
[2023-02-22 03:20:59,266] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-4950/global_step4954/mp_rank_00_model_states.pt...
[2023-02-22 03:21:00,092] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-4950/global_step4954/mp_rank_00_model_states.pt.
[2023-02-22 03:21:00,095] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-4950/global_step4954/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 03:21:00,270] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-4950/global_step4954/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 03:21:00,271] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-4950/global_step4954/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 03:21:00,271] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4954 is ready now!
Deleting older checkpoint [results/checkpoint-4800] due to args.save_total_limit

 81%|████████  | 4951/6100 [7:28:51<1:56:08,  6.06s/it]
 81%|████████  | 4952/6100 [7:28:56<1:52:15,  5.87s/it]
 81%|████████  | 4953/6100 [7:29:01<1:49:34,  5.73s/it]
 81%|████████  | 4954/6100 [7:29:07<1:47:35,  5.63s/it]
 81%|████████  | 4955/6100 [7:29:12<1:46:09,  5.56s/it]
                                                       
{'loss': 0.0605, 'learning_rate': 0.0001, 'epoch': 8.12}

 81%|████████  | 4955/6100 [7:29:12<1:46:09,  5.56s/it]
 81%|████████  | 4956/6100 [7:29:18<1:45:11,  5.52s/it]
 81%|████████▏ | 4957/6100 [7:29:23<1:44:24,  5.48s/it]
 81%|████████▏ | 4958/6100 [7:29:29<1:43:56,  5.46s/it]
 81%|████████▏ | 4959/6100 [7:29:34<1:43:29,  5.44s/it]
 81%|████████▏ | 4960/6100 [7:29:39<1:43:19,  5.44s/it]
                                                       
{'loss': 0.066, 'learning_rate': 0.0001, 'epoch': 8.13}

 81%|████████▏ | 4960/6100 [7:29:39<1:43:19,  5.44s/it]
 81%|████████▏ | 4961/6100 [7:29:45<1:43:03,  5.43s/it]
 81%|████████▏ | 4962/6100 [7:29:50<1:42:49,  5.42s/it]
 81%|████████▏ | 4963/6100 [7:29:56<1:42:40,  5.42s/it]
 81%|████████▏ | 4964/6100 [7:30:01<1:43:31,  5.47s/it]
 81%|████████▏ | 4965/6100 [7:30:07<1:43:08,  5.45s/it]
                                                       
{'loss': 0.0642, 'learning_rate': 0.0001, 'epoch': 8.14}

 81%|████████▏ | 4965/6100 [7:30:07<1:43:08,  5.45s/it]
 81%|████████▏ | 4966/6100 [7:30:12<1:42:48,  5.44s/it]
 81%|████████▏ | 4967/6100 [7:30:17<1:42:40,  5.44s/it]
 81%|████████▏ | 4968/6100 [7:30:23<1:42:25,  5.43s/it]
 81%|████████▏ | 4969/6100 [7:30:28<1:42:19,  5.43s/it]
 81%|████████▏ | 4970/6100 [7:30:34<1:42:08,  5.42s/it]
                                                       
{'loss': 0.0663, 'learning_rate': 0.0001, 'epoch': 8.15}

 81%|████████▏ | 4970/6100 [7:30:34<1:42:08,  5.42s/it]
 81%|████████▏ | 4971/6100 [7:30:39<1:41:57,  5.42s/it]
 82%|████████▏ | 4972/6100 [7:30:44<1:41:51,  5.42s/it]
 82%|████████▏ | 4973/6100 [7:30:50<1:41:41,  5.41s/it]
 82%|████████▏ | 4974/6100 [7:30:55<1:41:37,  5.41s/it]
 82%|████████▏ | 4975/6100 [7:31:01<1:41:31,  5.41s/it]
                                                       
{'loss': 0.0644, 'learning_rate': 0.0001, 'epoch': 8.16}

 82%|████████▏ | 4975/6100 [7:31:01<1:41:31,  5.41s/it]
 82%|████████▏ | 4976/6100 [7:31:06<1:41:24,  5.41s/it]
 82%|████████▏ | 4977/6100 [7:31:12<1:41:19,  5.41s/it]
 82%|████████▏ | 4978/6100 [7:31:17<1:41:27,  5.43s/it]
 82%|████████▏ | 4979/6100 [7:31:22<1:41:17,  5.42s/it]
 82%|████████▏ | 4980/6100 [7:31:28<1:41:08,  5.42s/it]
                                                       
{'loss': 0.0645, 'learning_rate': 0.0001, 'epoch': 8.16}

 82%|████████▏ | 4980/6100 [7:31:28<1:41:08,  5.42s/it]
 82%|████████▏ | 4981/6100 [7:31:33<1:41:08,  5.42s/it]
 82%|████████▏ | 4982/6100 [7:31:39<1:40:57,  5.42s/it]
 82%|████████▏ | 4983/6100 [7:31:44<1:40:51,  5.42s/it]
 82%|████████▏ | 4984/6100 [7:31:49<1:40:46,  5.42s/it]
 82%|████████▏ | 4985/6100 [7:31:55<1:40:37,  5.41s/it]
                                                       
{'loss': 0.0701, 'learning_rate': 0.0001, 'epoch': 8.17}

 82%|████████▏ | 4985/6100 [7:31:55<1:40:37,  5.41s/it]
 82%|████████▏ | 4986/6100 [7:32:00<1:40:27,  5.41s/it]
 82%|████████▏ | 4987/6100 [7:32:06<1:40:23,  5.41s/it]
 82%|████████▏ | 4988/6100 [7:32:11<1:40:26,  5.42s/it]
 82%|████████▏ | 4989/6100 [7:32:17<1:40:25,  5.42s/it]
 82%|████████▏ | 4990/6100 [7:32:22<1:40:22,  5.43s/it]
                                                       
{'loss': 0.0691, 'learning_rate': 0.0001, 'epoch': 8.18}

 82%|████████▏ | 4990/6100 [7:32:22<1:40:22,  5.43s/it]
 82%|████████▏ | 4991/6100 [7:32:27<1:40:10,  5.42s/it]
 82%|████████▏ | 4992/6100 [7:32:33<1:40:02,  5.42s/it]
 82%|████████▏ | 4993/6100 [7:32:38<1:39:58,  5.42s/it]
 82%|████████▏ | 4994/6100 [7:32:44<1:39:54,  5.42s/it]
 82%|████████▏ | 4995/6100 [7:32:49<1:39:46,  5.42s/it]
                                                       
{'loss': 0.0667, 'learning_rate': 0.0001, 'epoch': 8.19}

 82%|████████▏ | 4995/6100 [7:32:49<1:39:46,  5.42s/it]
 82%|████████▏ | 4996/6100 [7:32:55<1:39:44,  5.42s/it]
 82%|████████▏ | 4997/6100 [7:33:00<1:39:39,  5.42s/it]
 82%|████████▏ | 4998/6100 [7:33:05<1:39:31,  5.42s/it]
 82%|████████▏ | 4999/6100 [7:33:11<1:39:17,  5.41s/it]
 82%|████████▏ | 5000/6100 [7:33:16<1:39:16,  5.41s/it]
                                                       
{'loss': 0.0673, 'learning_rate': 0.0001, 'epoch': 8.2}

 82%|████████▏ | 5000/6100 [7:33:16<1:39:16,  5.41s/it]
 82%|████████▏ | 5001/6100 [7:33:22<1:39:10,  5.41s/it]
 82%|████████▏ | 5002/6100 [7:33:27<1:39:03,  5.41s/it]
 82%|████████▏ | 5003/6100 [7:33:33<1:42:55,  5.63s/it]
 82%|████████▏ | 5004/6100 [7:33:39<1:41:41,  5.57s/it]
 82%|████████▏ | 5005/6100 [7:33:44<1:40:44,  5.52s/it]
                                                       
{'loss': 0.0707, 'learning_rate': 0.0001, 'epoch': 8.2}

 82%|████████▏ | 5005/6100 [7:33:44<1:40:44,  5.52s/it]
 82%|████████▏ | 5006/6100 [7:33:49<1:40:01,  5.49s/it]
 82%|████████▏ | 5007/6100 [7:33:55<1:39:28,  5.46s/it]
 82%|████████▏ | 5008/6100 [7:34:00<1:39:07,  5.45s/it]
 82%|████████▏ | 5009/6100 [7:34:06<1:38:51,  5.44s/it]
 82%|████████▏ | 5010/6100 [7:34:11<1:38:33,  5.43s/it]
                                                       
{'loss': 0.0676, 'learning_rate': 0.0001, 'epoch': 8.21}

 82%|████████▏ | 5010/6100 [7:34:11<1:38:33,  5.43s/it]
 82%|████████▏ | 5011/6100 [7:34:16<1:38:26,  5.42s/it]
 82%|████████▏ | 5012/6100 [7:34:22<1:38:16,  5.42s/it]
 82%|████████▏ | 5013/6100 [7:34:27<1:38:05,  5.41s/it]
 82%|████████▏ | 5014/6100 [7:34:33<1:37:56,  5.41s/it]
 82%|████████▏ | 5015/6100 [7:34:38<1:37:52,  5.41s/it]
                                                       
{'loss': 0.0686, 'learning_rate': 0.0001, 'epoch': 8.22}

 82%|████████▏ | 5015/6100 [7:34:38<1:37:52,  5.41s/it]
 82%|████████▏ | 5016/6100 [7:34:43<1:37:49,  5.41s/it]
 82%|████████▏ | 5017/6100 [7:34:49<1:37:43,  5.41s/it]
 82%|████████▏ | 5018/6100 [7:34:54<1:37:36,  5.41s/it]
 82%|████████▏ | 5019/6100 [7:35:00<1:37:30,  5.41s/it]
 82%|████████▏ | 5020/6100 [7:35:05<1:37:24,  5.41s/it]
                                                       
{'loss': 0.0695, 'learning_rate': 0.0001, 'epoch': 8.23}

 82%|████████▏ | 5020/6100 [7:35:05<1:37:24,  5.41s/it]
 82%|████████▏ | 5021/6100 [7:35:11<1:37:20,  5.41s/it]
 82%|████████▏ | 5022/6100 [7:35:16<1:37:19,  5.42s/it]
 82%|████████▏ | 5023/6100 [7:35:21<1:37:16,  5.42s/it]
 82%|████████▏ | 5024/6100 [7:35:27<1:37:03,  5.41s/it]
 82%|████████▏ | 5025/6100 [7:35:32<1:37:00,  5.41s/it]
                                                       
{'loss': 0.0717, 'learning_rate': 0.0001, 'epoch': 8.24}

 82%|████████▏ | 5025/6100 [7:35:32<1:37:00,  5.41s/it]
 82%|████████▏ | 5026/6100 [7:35:38<1:36:52,  5.41s/it]
 82%|████████▏ | 5027/6100 [7:35:43<1:36:47,  5.41s/it]
 82%|████████▏ | 5028/6100 [7:35:48<1:36:49,  5.42s/it]
 82%|████████▏ | 5029/6100 [7:35:54<1:36:38,  5.41s/it]
 82%|████████▏ | 5030/6100 [7:35:59<1:36:50,  5.43s/it]
                                                       
{'loss': 0.0666, 'learning_rate': 0.0001, 'epoch': 8.25}

 82%|████████▏ | 5030/6100 [7:35:59<1:36:50,  5.43s/it]
 82%|████████▏ | 5031/6100 [7:36:05<1:36:41,  5.43s/it]
 82%|████████▏ | 5032/6100 [7:36:10<1:36:38,  5.43s/it]
 83%|████████▎ | 5033/6100 [7:36:16<1:36:34,  5.43s/it]
 83%|████████▎ | 5034/6100 [7:36:21<1:36:24,  5.43s/it]
 83%|████████▎ | 5035/6100 [7:36:26<1:36:16,  5.42s/it]
                                                       
{'loss': 0.0657, 'learning_rate': 0.0001, 'epoch': 8.25}

 83%|████████▎ | 5035/6100 [7:36:26<1:36:16,  5.42s/it]
 83%|████████▎ | 5036/6100 [7:36:32<1:36:16,  5.43s/it]
 83%|████████▎ | 5037/6100 [7:36:37<1:36:09,  5.43s/it]
 83%|████████▎ | 5038/6100 [7:36:43<1:36:00,  5.42s/it]
 83%|████████▎ | 5039/6100 [7:36:48<1:35:49,  5.42s/it]
 83%|████████▎ | 5040/6100 [7:36:54<1:35:42,  5.42s/it]
                                                       
{'loss': 0.062, 'learning_rate': 0.0001, 'epoch': 8.26}

 83%|████████▎ | 5040/6100 [7:36:54<1:35:42,  5.42s/it]
 83%|████████▎ | 5041/6100 [7:36:59<1:35:33,  5.41s/it]
 83%|████████▎ | 5042/6100 [7:37:05<1:38:58,  5.61s/it]
 83%|████████▎ | 5043/6100 [7:37:10<1:37:47,  5.55s/it]
 83%|████████▎ | 5044/6100 [7:37:16<1:38:22,  5.59s/it]
 83%|████████▎ | 5045/6100 [7:37:22<1:37:23,  5.54s/it]
                                                       
{'loss': 0.0669, 'learning_rate': 0.0001, 'epoch': 8.27}

 83%|████████▎ | 5045/6100 [7:37:22<1:37:23,  5.54s/it]
 83%|████████▎ | 5046/6100 [7:37:27<1:36:38,  5.50s/it]
 83%|████████▎ | 5047/6100 [7:37:32<1:36:09,  5.48s/it]
 83%|████████▎ | 5048/6100 [7:37:38<1:35:42,  5.46s/it]
 83%|████████▎ | 5049/6100 [7:37:43<1:35:22,  5.45s/it]
 83%|████████▎ | 5050/6100 [7:37:49<1:35:04,  5.43s/it]
                                                       
{'loss': 0.0671, 'learning_rate': 0.0001, 'epoch': 8.28}

 83%|████████▎ | 5050/6100 [7:37:49<1:35:04,  5.43s/it]
 83%|████████▎ | 5051/6100 [7:37:54<1:34:57,  5.43s/it]
 83%|████████▎ | 5052/6100 [7:37:59<1:34:44,  5.42s/it]
 83%|████████▎ | 5053/6100 [7:38:05<1:34:39,  5.42s/it]
 83%|████████▎ | 5054/6100 [7:38:10<1:34:27,  5.42s/it]
 83%|████████▎ | 5055/6100 [7:38:16<1:34:18,  5.41s/it]
                                                       
{'loss': 0.0662, 'learning_rate': 0.0001, 'epoch': 8.29}

 83%|████████▎ | 5055/6100 [7:38:16<1:34:18,  5.41s/it]
 83%|████████▎ | 5056/6100 [7:38:21<1:34:10,  5.41s/it]
 83%|████████▎ | 5057/6100 [7:38:26<1:34:02,  5.41s/it]
 83%|████████▎ | 5058/6100 [7:38:32<1:33:59,  5.41s/it]
 83%|████████▎ | 5059/6100 [7:38:37<1:33:52,  5.41s/it]
 83%|████████▎ | 5060/6100 [7:38:43<1:33:46,  5.41s/it]
                                                       
{'loss': 0.0656, 'learning_rate': 0.0001, 'epoch': 8.29}

 83%|████████▎ | 5060/6100 [7:38:43<1:33:46,  5.41s/it]
 83%|████████▎ | 5061/6100 [7:38:48<1:34:24,  5.45s/it]
 83%|████████▎ | 5062/6100 [7:38:54<1:34:04,  5.44s/it]
 83%|████████▎ | 5063/6100 [7:38:59<1:33:46,  5.43s/it]
 83%|████████▎ | 5064/6100 [7:39:04<1:33:35,  5.42s/it]
 83%|████████▎ | 5065/6100 [7:39:10<1:33:27,  5.42s/it]
                                                       
{'loss': 0.0635, 'learning_rate': 0.0001, 'epoch': 8.3}

 83%|████████▎ | 5065/6100 [7:39:10<1:33:27,  5.42s/it]
 83%|████████▎ | 5066/6100 [7:39:15<1:33:24,  5.42s/it]
 83%|████████▎ | 5067/6100 [7:39:21<1:33:09,  5.41s/it]
 83%|████████▎ | 5068/6100 [7:39:26<1:33:05,  5.41s/it]
 83%|████████▎ | 5069/6100 [7:39:32<1:33:02,  5.41s/it]
 83%|████████▎ | 5070/6100 [7:39:37<1:32:56,  5.41s/it]
                                                       
{'loss': 0.0659, 'learning_rate': 0.0001, 'epoch': 8.31}

 83%|████████▎ | 5070/6100 [7:39:37<1:32:56,  5.41s/it]
 83%|████████▎ | 5071/6100 [7:39:42<1:32:51,  5.41s/it]
 83%|████████▎ | 5072/6100 [7:39:48<1:32:45,  5.41s/it]
 83%|████████▎ | 5073/6100 [7:39:53<1:32:39,  5.41s/it]
 83%|████████▎ | 5074/6100 [7:39:59<1:32:33,  5.41s/it]
 83%|████████▎ | 5075/6100 [7:40:04<1:32:31,  5.42s/it]
                                                       
{'loss': 0.0657, 'learning_rate': 0.0001, 'epoch': 8.32}

 83%|████████▎ | 5075/6100 [7:40:04<1:32:31,  5.42s/it]
 83%|████████▎ | 5076/6100 [7:40:09<1:32:29,  5.42s/it]
 83%|████████▎ | 5077/6100 [7:40:15<1:32:22,  5.42s/it]
 83%|████████▎ | 5078/6100 [7:40:20<1:32:15,  5.42s/it]
 83%|████████▎ | 5079/6100 [7:40:26<1:32:10,  5.42s/it]
 83%|████████▎ | 5080/6100 [7:40:31<1:32:03,  5.42s/it]
                                                       
{'loss': 0.0685, 'learning_rate': 0.0001, 'epoch': 8.33}

 83%|████████▎ | 5080/6100 [7:40:31<1:32:03,  5.42s/it]
 83%|████████▎ | 5081/6100 [7:40:37<1:31:59,  5.42s/it]
 83%|████████▎ | 5082/6100 [7:40:42<1:31:56,  5.42s/it]
 83%|████████▎ | 5083/6100 [7:40:47<1:31:47,  5.42s/it]
 83%|████████▎ | 5084/6100 [7:40:53<1:31:42,  5.42s/it]
 83%|████████▎ | 5085/6100 [7:40:58<1:31:34,  5.41s/it]
                                                       
{'loss': 0.071, 'learning_rate': 0.0001, 'epoch': 8.34}

 83%|████████▎ | 5085/6100 [7:40:58<1:31:34,  5.41s/it]
 83%|████████▎ | 5086/6100 [7:41:04<1:31:30,  5.42s/it]
 83%|████████▎ | 5087/6100 [7:41:09<1:31:29,  5.42s/it]
 83%|████████▎ | 5088/6100 [7:41:14<1:31:19,  5.41s/it]
 83%|████████▎ | 5089/6100 [7:41:20<1:31:21,  5.42s/it]
 83%|████████▎ | 5090/6100 [7:41:25<1:31:12,  5.42s/it]
                                                       
{'loss': 0.0689, 'learning_rate': 0.0001, 'epoch': 8.34}

 83%|████████▎ | 5090/6100 [7:41:25<1:31:12,  5.42s/it]
 83%|████████▎ | 5091/6100 [7:41:31<1:31:05,  5.42s/it]
 83%|████████▎ | 5092/6100 [7:41:36<1:30:58,  5.42s/it]
 83%|████████▎ | 5093/6100 [7:41:42<1:30:55,  5.42s/it]
 84%|████████▎ | 5094/6100 [7:41:47<1:30:45,  5.41s/it]
 84%|████████▎ | 5095/6100 [7:41:52<1:30:39,  5.41s/it]
                                                       
{'loss': 0.0671, 'learning_rate': 0.0001, 'epoch': 8.35}

 84%|████████▎ | 5095/6100 [7:41:52<1:30:39,  5.41s/it]
 84%|████████▎ | 5096/6100 [7:41:58<1:30:31,  5.41s/it]
 84%|████████▎ | 5097/6100 [7:42:03<1:30:27,  5.41s/it]
 84%|████████▎ | 5098/6100 [7:42:09<1:30:23,  5.41s/it]
 84%|████████▎ | 5099/6100 [7:42:14<1:30:16,  5.41s/it]
 84%|████████▎ | 5100/6100 [7:42:19<1:30:16,  5.42s/it]
                                                       
{'loss': 0.0688, 'learning_rate': 0.0001, 'epoch': 8.36}

 84%|████████▎ | 5100/6100 [7:42:19<1:30:16,  5.42s/it]Saving model checkpoint to ./results/checkpoint-5100
Configuration saved in ./results/checkpoint-5100/config.json
Model weights saved in ./results/checkpoint-5100/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-5100/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-5100/special_tokens_map.json
[2023-02-22 03:34:35,581] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step5104 is begin to save!
[2023-02-22 03:34:35,584] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-5100/global_step5104/mp_rank_00_model_states.pt
[2023-02-22 03:34:35,585] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-5100/global_step5104/mp_rank_00_model_states.pt...
[2023-02-22 03:34:36,403] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-5100/global_step5104/mp_rank_00_model_states.pt.
[2023-02-22 03:34:36,406] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-5100/global_step5104/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 03:34:36,578] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-5100/global_step5104/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 03:34:36,578] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-5100/global_step5104/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 03:34:36,578] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5104 is ready now!
Deleting older checkpoint [results/checkpoint-4950] due to args.save_total_limit

 84%|████████▎ | 5101/6100 [7:42:27<1:40:56,  6.06s/it]
 84%|████████▎ | 5102/6100 [7:42:32<1:37:36,  5.87s/it]
 84%|████████▎ | 5103/6100 [7:42:38<1:35:10,  5.73s/it]
 84%|████████▎ | 5104/6100 [7:42:43<1:33:26,  5.63s/it]
 84%|████████▎ | 5105/6100 [7:42:49<1:32:19,  5.57s/it]
                                                       
{'loss': 0.0652, 'learning_rate': 0.0001, 'epoch': 8.37}

 84%|████████▎ | 5105/6100 [7:42:49<1:32:19,  5.57s/it]
 84%|████████▎ | 5106/6100 [7:42:54<1:31:25,  5.52s/it]
 84%|████████▎ | 5107/6100 [7:42:59<1:30:49,  5.49s/it]
 84%|████████▎ | 5108/6100 [7:43:05<1:30:19,  5.46s/it]
 84%|████████▍ | 5109/6100 [7:43:10<1:29:55,  5.44s/it]
 84%|████████▍ | 5110/6100 [7:43:16<1:29:41,  5.44s/it]
                                                       
{'loss': 0.0686, 'learning_rate': 0.0001, 'epoch': 8.38}

 84%|████████▍ | 5110/6100 [7:43:16<1:29:41,  5.44s/it]
 84%|████████▍ | 5111/6100 [7:43:21<1:29:32,  5.43s/it]
 84%|████████▍ | 5112/6100 [7:43:26<1:29:20,  5.43s/it]
 84%|████████▍ | 5113/6100 [7:43:32<1:29:09,  5.42s/it]
 84%|████████▍ | 5114/6100 [7:43:37<1:29:00,  5.42s/it]
 84%|████████▍ | 5115/6100 [7:43:43<1:28:54,  5.42s/it]
                                                       
{'loss': 0.07, 'learning_rate': 0.0001, 'epoch': 8.38}

 84%|████████▍ | 5115/6100 [7:43:43<1:28:54,  5.42s/it]
 84%|████████▍ | 5116/6100 [7:43:48<1:28:43,  5.41s/it]
 84%|████████▍ | 5117/6100 [7:43:54<1:28:41,  5.41s/it]
 84%|████████▍ | 5118/6100 [7:43:59<1:28:36,  5.41s/it]
 84%|████████▍ | 5119/6100 [7:44:04<1:28:31,  5.41s/it]
 84%|████████▍ | 5120/6100 [7:44:10<1:28:24,  5.41s/it]
                                                       
{'loss': 0.0626, 'learning_rate': 0.0001, 'epoch': 8.39}

 84%|████████▍ | 5120/6100 [7:44:10<1:28:24,  5.41s/it]
 84%|████████▍ | 5121/6100 [7:44:15<1:28:17,  5.41s/it]
 84%|████████▍ | 5122/6100 [7:44:21<1:28:09,  5.41s/it]
 84%|████████▍ | 5123/6100 [7:44:26<1:28:06,  5.41s/it]
 84%|████████▍ | 5124/6100 [7:44:31<1:27:58,  5.41s/it]
 84%|████████▍ | 5125/6100 [7:44:37<1:27:53,  5.41s/it]
                                                       
{'loss': 0.0684, 'learning_rate': 0.0001, 'epoch': 8.4}

 84%|████████▍ | 5125/6100 [7:44:37<1:27:53,  5.41s/it]
 84%|████████▍ | 5126/6100 [7:44:42<1:27:49,  5.41s/it]
 84%|████████▍ | 5127/6100 [7:44:48<1:27:49,  5.42s/it]
 84%|████████▍ | 5128/6100 [7:44:53<1:27:45,  5.42s/it]
 84%|████████▍ | 5129/6100 [7:44:59<1:27:42,  5.42s/it]
 84%|████████▍ | 5130/6100 [7:45:04<1:27:39,  5.42s/it]
                                                       
{'loss': 0.0671, 'learning_rate': 0.0001, 'epoch': 8.41}

 84%|████████▍ | 5130/6100 [7:45:04<1:27:39,  5.42s/it]
 84%|████████▍ | 5131/6100 [7:45:09<1:27:29,  5.42s/it]
 84%|████████▍ | 5132/6100 [7:45:15<1:27:20,  5.41s/it]
 84%|████████▍ | 5133/6100 [7:45:20<1:27:14,  5.41s/it]
 84%|████████▍ | 5134/6100 [7:45:26<1:27:06,  5.41s/it]
 84%|████████▍ | 5135/6100 [7:45:31<1:27:02,  5.41s/it]
                                                       
{'loss': 0.062, 'learning_rate': 0.0001, 'epoch': 8.42}

 84%|████████▍ | 5135/6100 [7:45:31<1:27:02,  5.41s/it]
 84%|████████▍ | 5136/6100 [7:45:36<1:26:53,  5.41s/it]
 84%|████████▍ | 5137/6100 [7:45:42<1:26:45,  5.41s/it]
 84%|████████▍ | 5138/6100 [7:45:47<1:26:44,  5.41s/it]
 84%|████████▍ | 5139/6100 [7:45:53<1:26:43,  5.42s/it]
 84%|████████▍ | 5140/6100 [7:45:58<1:26:38,  5.41s/it]
                                                       
{'loss': 0.0676, 'learning_rate': 0.0001, 'epoch': 8.43}

 84%|████████▍ | 5140/6100 [7:45:58<1:26:38,  5.41s/it]
 84%|████████▍ | 5141/6100 [7:46:03<1:26:30,  5.41s/it]
 84%|████████▍ | 5142/6100 [7:46:09<1:26:29,  5.42s/it]
 84%|████████▍ | 5143/6100 [7:46:14<1:26:20,  5.41s/it]
 84%|████████▍ | 5144/6100 [7:46:20<1:26:12,  5.41s/it]
 84%|████████▍ | 5145/6100 [7:46:25<1:26:08,  5.41s/it]
                                                       
{'loss': 0.068, 'learning_rate': 0.0001, 'epoch': 8.43}

 84%|████████▍ | 5145/6100 [7:46:25<1:26:08,  5.41s/it]
 84%|████████▍ | 5146/6100 [7:46:31<1:26:05,  5.41s/it]
 84%|████████▍ | 5147/6100 [7:46:36<1:25:57,  5.41s/it]
 84%|████████▍ | 5148/6100 [7:46:41<1:25:52,  5.41s/it]
 84%|████████▍ | 5149/6100 [7:46:47<1:25:49,  5.42s/it]
 84%|████████▍ | 5150/6100 [7:46:52<1:25:43,  5.41s/it]
                                                       
{'loss': 0.0665, 'learning_rate': 0.0001, 'epoch': 8.44}

 84%|████████▍ | 5150/6100 [7:46:52<1:25:43,  5.41s/it]
 84%|████████▍ | 5151/6100 [7:46:58<1:25:41,  5.42s/it]
 84%|████████▍ | 5152/6100 [7:47:03<1:25:31,  5.41s/it]
 84%|████████▍ | 5153/6100 [7:47:08<1:25:22,  5.41s/it]
 84%|████████▍ | 5154/6100 [7:47:14<1:25:17,  5.41s/it]
 85%|████████▍ | 5155/6100 [7:47:19<1:25:16,  5.41s/it]
                                                       
{'loss': 0.0685, 'learning_rate': 0.0001, 'epoch': 8.45}

 85%|████████▍ | 5155/6100 [7:47:19<1:25:16,  5.41s/it]
 85%|████████▍ | 5156/6100 [7:47:25<1:25:11,  5.41s/it]
 85%|████████▍ | 5157/6100 [7:47:30<1:25:04,  5.41s/it]
 85%|████████▍ | 5158/6100 [7:47:35<1:24:55,  5.41s/it]
 85%|████████▍ | 5159/6100 [7:47:41<1:24:50,  5.41s/it]
 85%|████████▍ | 5160/6100 [7:47:46<1:24:45,  5.41s/it]
                                                       
{'loss': 0.0647, 'learning_rate': 0.0001, 'epoch': 8.46}

 85%|████████▍ | 5160/6100 [7:47:46<1:24:45,  5.41s/it]
 85%|████████▍ | 5161/6100 [7:47:52<1:24:45,  5.42s/it]
 85%|████████▍ | 5162/6100 [7:47:57<1:24:43,  5.42s/it]
 85%|████████▍ | 5163/6100 [7:48:03<1:24:37,  5.42s/it]
 85%|████████▍ | 5164/6100 [7:48:08<1:24:33,  5.42s/it]
 85%|████████▍ | 5165/6100 [7:48:13<1:24:28,  5.42s/it]
                                                       
{'loss': 0.0669, 'learning_rate': 0.0001, 'epoch': 8.47}

 85%|████████▍ | 5165/6100 [7:48:13<1:24:28,  5.42s/it]
 85%|████████▍ | 5166/6100 [7:48:19<1:24:20,  5.42s/it]
 85%|████████▍ | 5167/6100 [7:48:24<1:24:13,  5.42s/it]
 85%|████████▍ | 5168/6100 [7:48:30<1:24:08,  5.42s/it]
 85%|████████▍ | 5169/6100 [7:48:35<1:24:04,  5.42s/it]
 85%|████████▍ | 5170/6100 [7:48:40<1:24:01,  5.42s/it]
                                                       
{'loss': 0.0633, 'learning_rate': 0.0001, 'epoch': 8.48}

 85%|████████▍ | 5170/6100 [7:48:40<1:24:01,  5.42s/it]
 85%|████████▍ | 5171/6100 [7:48:46<1:23:53,  5.42s/it]
 85%|████████▍ | 5172/6100 [7:48:51<1:23:41,  5.41s/it]
 85%|████████▍ | 5173/6100 [7:48:57<1:23:37,  5.41s/it]
 85%|████████▍ | 5174/6100 [7:49:02<1:23:38,  5.42s/it]
 85%|████████▍ | 5175/6100 [7:49:08<1:23:26,  5.41s/it]
                                                       
{'loss': 0.0699, 'learning_rate': 0.0001, 'epoch': 8.48}

 85%|████████▍ | 5175/6100 [7:49:08<1:23:26,  5.41s/it]
 85%|████████▍ | 5176/6100 [7:49:13<1:23:24,  5.42s/it]
 85%|████████▍ | 5177/6100 [7:49:18<1:23:14,  5.41s/it]
 85%|████████▍ | 5178/6100 [7:49:24<1:23:09,  5.41s/it]
 85%|████████▍ | 5179/6100 [7:49:29<1:23:05,  5.41s/it]
 85%|████████▍ | 5180/6100 [7:49:35<1:22:59,  5.41s/it]
                                                       
{'loss': 0.0654, 'learning_rate': 0.0001, 'epoch': 8.49}

 85%|████████▍ | 5180/6100 [7:49:35<1:22:59,  5.41s/it]
 85%|████████▍ | 5181/6100 [7:49:40<1:22:59,  5.42s/it]
 85%|████████▍ | 5182/6100 [7:49:45<1:22:52,  5.42s/it]
 85%|████████▍ | 5183/6100 [7:49:51<1:22:48,  5.42s/it]
 85%|████████▍ | 5184/6100 [7:49:56<1:22:45,  5.42s/it]
 85%|████████▌ | 5185/6100 [7:50:02<1:22:33,  5.41s/it]
                                                       
{'loss': 0.0655, 'learning_rate': 0.0001, 'epoch': 8.5}

 85%|████████▌ | 5185/6100 [7:50:02<1:22:33,  5.41s/it]
 85%|████████▌ | 5186/6100 [7:50:07<1:22:23,  5.41s/it]
 85%|████████▌ | 5187/6100 [7:50:13<1:22:20,  5.41s/it]
 85%|████████▌ | 5188/6100 [7:50:18<1:22:12,  5.41s/it]
 85%|████████▌ | 5189/6100 [7:50:23<1:22:08,  5.41s/it]
 85%|████████▌ | 5190/6100 [7:50:29<1:22:03,  5.41s/it]
                                                       
{'loss': 0.0676, 'learning_rate': 0.0001, 'epoch': 8.51}

 85%|████████▌ | 5190/6100 [7:50:29<1:22:03,  5.41s/it]
 85%|████████▌ | 5191/6100 [7:50:34<1:22:01,  5.41s/it]
 85%|████████▌ | 5192/6100 [7:50:40<1:21:56,  5.41s/it]
 85%|████████▌ | 5193/6100 [7:50:45<1:21:48,  5.41s/it]
 85%|████████▌ | 5194/6100 [7:50:50<1:21:41,  5.41s/it]
 85%|████████▌ | 5195/6100 [7:50:56<1:21:37,  5.41s/it]
                                                       
{'loss': 0.0645, 'learning_rate': 0.0001, 'epoch': 8.52}

 85%|████████▌ | 5195/6100 [7:50:56<1:21:37,  5.41s/it]
 85%|████████▌ | 5196/6100 [7:51:01<1:21:36,  5.42s/it]
 85%|████████▌ | 5197/6100 [7:51:07<1:21:31,  5.42s/it]
 85%|████████▌ | 5198/6100 [7:51:12<1:21:22,  5.41s/it]
 85%|████████▌ | 5199/6100 [7:51:17<1:21:12,  5.41s/it]
 85%|████████▌ | 5200/6100 [7:51:23<1:21:07,  5.41s/it]
                                                       
{'loss': 0.0617, 'learning_rate': 0.0001, 'epoch': 8.52}

 85%|████████▌ | 5200/6100 [7:51:23<1:21:07,  5.41s/it]
 85%|████████▌ | 5201/6100 [7:51:28<1:21:10,  5.42s/it]
 85%|████████▌ | 5202/6100 [7:51:34<1:21:02,  5.41s/it]
 85%|████████▌ | 5203/6100 [7:51:39<1:20:54,  5.41s/it]
 85%|████████▌ | 5204/6100 [7:51:45<1:20:48,  5.41s/it]
 85%|████████▌ | 5205/6100 [7:51:50<1:20:42,  5.41s/it]
                                                       
{'loss': 0.0625, 'learning_rate': 0.0001, 'epoch': 8.53}

 85%|████████▌ | 5205/6100 [7:51:50<1:20:42,  5.41s/it]
 85%|████████▌ | 5206/6100 [7:51:55<1:20:55,  5.43s/it]
 85%|████████▌ | 5207/6100 [7:52:01<1:20:48,  5.43s/it]
 85%|████████▌ | 5208/6100 [7:52:06<1:20:36,  5.42s/it]
 85%|████████▌ | 5209/6100 [7:52:12<1:20:29,  5.42s/it]
 85%|████████▌ | 5210/6100 [7:52:17<1:20:23,  5.42s/it]
                                                       
{'loss': 0.068, 'learning_rate': 0.0001, 'epoch': 8.54}

 85%|████████▌ | 5210/6100 [7:52:17<1:20:23,  5.42s/it]
 85%|████████▌ | 5211/6100 [7:52:23<1:20:36,  5.44s/it]
 85%|████████▌ | 5212/6100 [7:52:28<1:20:24,  5.43s/it]
 85%|████████▌ | 5213/6100 [7:52:33<1:20:10,  5.42s/it]
 85%|████████▌ | 5214/6100 [7:52:39<1:20:03,  5.42s/it]
 85%|████████▌ | 5215/6100 [7:52:44<1:19:56,  5.42s/it]
                                                       
{'loss': 0.0675, 'learning_rate': 0.0001, 'epoch': 8.55}

 85%|████████▌ | 5215/6100 [7:52:44<1:19:56,  5.42s/it]
 86%|████████▌ | 5216/6100 [7:52:50<1:19:50,  5.42s/it]
 86%|████████▌ | 5217/6100 [7:52:55<1:19:40,  5.41s/it]
 86%|████████▌ | 5218/6100 [7:53:00<1:19:36,  5.42s/it]
 86%|████████▌ | 5219/6100 [7:53:06<1:19:29,  5.41s/it]
 86%|████████▌ | 5220/6100 [7:53:11<1:19:33,  5.42s/it]
                                                       
{'loss': 0.065, 'learning_rate': 0.0001, 'epoch': 8.56}

 86%|████████▌ | 5220/6100 [7:53:11<1:19:33,  5.42s/it]
 86%|████████▌ | 5221/6100 [7:53:17<1:19:29,  5.43s/it]
 86%|████████▌ | 5222/6100 [7:53:22<1:19:21,  5.42s/it]
 86%|████████▌ | 5223/6100 [7:53:28<1:19:15,  5.42s/it]
 86%|████████▌ | 5224/6100 [7:53:33<1:19:06,  5.42s/it]
 86%|████████▌ | 5225/6100 [7:53:38<1:19:00,  5.42s/it]
                                                       
{'loss': 0.0663, 'learning_rate': 0.0001, 'epoch': 8.57}

 86%|████████▌ | 5225/6100 [7:53:38<1:19:00,  5.42s/it]
 86%|████████▌ | 5226/6100 [7:53:44<1:18:56,  5.42s/it]
 86%|████████▌ | 5227/6100 [7:53:49<1:18:50,  5.42s/it]
 86%|████████▌ | 5228/6100 [7:53:55<1:18:41,  5.41s/it]
 86%|████████▌ | 5229/6100 [7:54:00<1:18:41,  5.42s/it]
 86%|████████▌ | 5230/6100 [7:54:05<1:18:32,  5.42s/it]
                                                       
{'loss': 0.0693, 'learning_rate': 0.0001, 'epoch': 8.57}

 86%|████████▌ | 5230/6100 [7:54:05<1:18:32,  5.42s/it]
 86%|████████▌ | 5231/6100 [7:54:11<1:18:26,  5.42s/it]
 86%|████████▌ | 5232/6100 [7:54:16<1:18:20,  5.42s/it]
 86%|████████▌ | 5233/6100 [7:54:22<1:18:21,  5.42s/it]
 86%|████████▌ | 5234/6100 [7:54:27<1:18:13,  5.42s/it]
 86%|████████▌ | 5235/6100 [7:54:33<1:18:08,  5.42s/it]
                                                       
{'loss': 0.0631, 'learning_rate': 0.0001, 'epoch': 8.58}

 86%|████████▌ | 5235/6100 [7:54:33<1:18:08,  5.42s/it]
 86%|████████▌ | 5236/6100 [7:54:38<1:17:58,  5.41s/it]
 86%|████████▌ | 5237/6100 [7:54:43<1:17:49,  5.41s/it]
 86%|████████▌ | 5238/6100 [7:54:49<1:17:47,  5.42s/it]
 86%|████████▌ | 5239/6100 [7:54:54<1:17:41,  5.41s/it]
 86%|████████▌ | 5240/6100 [7:55:00<1:17:34,  5.41s/it]
                                                       
{'loss': 0.0682, 'learning_rate': 0.0001, 'epoch': 8.59}

 86%|████████▌ | 5240/6100 [7:55:00<1:17:34,  5.41s/it]
 86%|████████▌ | 5241/6100 [7:55:05<1:17:30,  5.41s/it]
 86%|████████▌ | 5242/6100 [7:55:10<1:17:21,  5.41s/it]
 86%|████████▌ | 5243/6100 [7:55:16<1:17:17,  5.41s/it]
 86%|████████▌ | 5244/6100 [7:55:21<1:17:15,  5.41s/it]
 86%|████████▌ | 5245/6100 [7:55:27<1:17:06,  5.41s/it]
                                                       
{'loss': 0.0705, 'learning_rate': 0.0001, 'epoch': 8.6}

 86%|████████▌ | 5245/6100 [7:55:27<1:17:06,  5.41s/it]
 86%|████████▌ | 5246/6100 [7:55:32<1:17:01,  5.41s/it]
 86%|████████▌ | 5247/6100 [7:55:38<1:16:54,  5.41s/it]
 86%|████████▌ | 5248/6100 [7:55:43<1:16:46,  5.41s/it]
 86%|████████▌ | 5249/6100 [7:55:48<1:16:47,  5.41s/it]
 86%|████████▌ | 5250/6100 [7:55:54<1:16:40,  5.41s/it]
                                                       
{'loss': 0.0673, 'learning_rate': 0.0001, 'epoch': 8.61}

 86%|████████▌ | 5250/6100 [7:55:54<1:16:40,  5.41s/it]Saving model checkpoint to ./results/checkpoint-5250
Configuration saved in ./results/checkpoint-5250/config.json
Model weights saved in ./results/checkpoint-5250/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-5250/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-5250/special_tokens_map.json
[2023-02-22 03:48:09,921] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step5254 is begin to save!
[2023-02-22 03:48:09,924] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-5250/global_step5254/mp_rank_00_model_states.pt
[2023-02-22 03:48:09,924] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-5250/global_step5254/mp_rank_00_model_states.pt...
[2023-02-22 03:48:10,736] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-5250/global_step5254/mp_rank_00_model_states.pt.
[2023-02-22 03:48:10,738] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-5250/global_step5254/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 03:48:10,911] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-5250/global_step5254/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 03:48:10,911] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-5250/global_step5254/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 03:48:10,911] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5254 is ready now!
Deleting older checkpoint [results/checkpoint-5100] due to args.save_total_limit

 86%|████████▌ | 5251/6100 [7:56:01<1:25:39,  6.05s/it]
 86%|████████▌ | 5252/6100 [7:56:07<1:22:56,  5.87s/it]
 86%|████████▌ | 5253/6100 [7:56:12<1:20:54,  5.73s/it]
 86%|████████▌ | 5254/6100 [7:56:18<1:19:29,  5.64s/it]
 86%|████████▌ | 5255/6100 [7:56:23<1:18:25,  5.57s/it]
                                                       
{'loss': 0.0683, 'learning_rate': 0.0001, 'epoch': 8.61}

 86%|████████▌ | 5255/6100 [7:56:23<1:18:25,  5.57s/it]
 86%|████████▌ | 5256/6100 [7:56:28<1:17:39,  5.52s/it]
 86%|████████▌ | 5257/6100 [7:56:34<1:16:59,  5.48s/it]
 86%|████████▌ | 5258/6100 [7:56:39<1:16:32,  5.45s/it]
 86%|████████▌ | 5259/6100 [7:56:45<1:17:01,  5.49s/it]
 86%|████████▌ | 5260/6100 [7:56:50<1:16:33,  5.47s/it]
                                                       
{'loss': 0.07, 'learning_rate': 0.0001, 'epoch': 8.62}

 86%|████████▌ | 5260/6100 [7:56:50<1:16:33,  5.47s/it]
 86%|████████▌ | 5261/6100 [7:56:56<1:16:14,  5.45s/it]
 86%|████████▋ | 5262/6100 [7:57:01<1:15:56,  5.44s/it]
 86%|████████▋ | 5263/6100 [7:57:06<1:15:40,  5.42s/it]
 86%|████████▋ | 5264/6100 [7:57:12<1:15:39,  5.43s/it]
 86%|████████▋ | 5265/6100 [7:57:17<1:15:29,  5.42s/it]
                                                       
{'loss': 0.0673, 'learning_rate': 0.0001, 'epoch': 8.63}

 86%|████████▋ | 5265/6100 [7:57:17<1:15:29,  5.42s/it]
 86%|████████▋ | 5266/6100 [7:57:23<1:15:19,  5.42s/it]
 86%|████████▋ | 5267/6100 [7:57:28<1:16:14,  5.49s/it]
 86%|████████▋ | 5268/6100 [7:57:34<1:15:54,  5.47s/it]
 86%|████████▋ | 5269/6100 [7:57:39<1:15:31,  5.45s/it]
 86%|████████▋ | 5270/6100 [7:57:45<1:15:16,  5.44s/it]
                                                       
{'loss': 0.0675, 'learning_rate': 0.0001, 'epoch': 8.64}

 86%|████████▋ | 5270/6100 [7:57:45<1:15:16,  5.44s/it]
 86%|████████▋ | 5271/6100 [7:57:50<1:15:06,  5.44s/it]
 86%|████████▋ | 5272/6100 [7:57:55<1:15:01,  5.44s/it]
 86%|████████▋ | 5273/6100 [7:58:01<1:14:46,  5.43s/it]
 86%|████████▋ | 5274/6100 [7:58:06<1:14:34,  5.42s/it]
 86%|████████▋ | 5275/6100 [7:58:12<1:14:30,  5.42s/it]
                                                       
{'loss': 0.0675, 'learning_rate': 0.0001, 'epoch': 8.65}

 86%|████████▋ | 5275/6100 [7:58:12<1:14:30,  5.42s/it]
 86%|████████▋ | 5276/6100 [7:58:17<1:14:24,  5.42s/it]
 87%|████████▋ | 5277/6100 [7:58:22<1:14:14,  5.41s/it]
 87%|████████▋ | 5278/6100 [7:58:28<1:14:06,  5.41s/it]
 87%|████████▋ | 5279/6100 [7:58:33<1:14:00,  5.41s/it]
 87%|████████▋ | 5280/6100 [7:58:39<1:13:57,  5.41s/it]
                                                       
{'loss': 0.0654, 'learning_rate': 0.0001, 'epoch': 8.66}

 87%|████████▋ | 5280/6100 [7:58:39<1:13:57,  5.41s/it]
 87%|████████▋ | 5281/6100 [7:58:44<1:13:51,  5.41s/it]
 87%|████████▋ | 5282/6100 [7:58:50<1:13:47,  5.41s/it]
 87%|████████▋ | 5283/6100 [7:58:55<1:13:41,  5.41s/it]
 87%|████████▋ | 5284/6100 [7:59:00<1:13:36,  5.41s/it]
 87%|████████▋ | 5285/6100 [7:59:06<1:13:35,  5.42s/it]
                                                       
{'loss': 0.0648, 'learning_rate': 0.0001, 'epoch': 8.66}

 87%|████████▋ | 5285/6100 [7:59:06<1:13:35,  5.42s/it]
 87%|████████▋ | 5286/6100 [7:59:11<1:13:29,  5.42s/it]
 87%|████████▋ | 5287/6100 [7:59:17<1:13:21,  5.41s/it]
 87%|████████▋ | 5288/6100 [7:59:22<1:13:17,  5.42s/it]
 87%|████████▋ | 5289/6100 [7:59:27<1:13:15,  5.42s/it]
 87%|████████▋ | 5290/6100 [7:59:33<1:13:09,  5.42s/it]
                                                       
{'loss': 0.0678, 'learning_rate': 0.0001, 'epoch': 8.67}

 87%|████████▋ | 5290/6100 [7:59:33<1:13:09,  5.42s/it]
 87%|████████▋ | 5291/6100 [7:59:38<1:13:05,  5.42s/it]
 87%|████████▋ | 5292/6100 [7:59:44<1:12:59,  5.42s/it]
 87%|████████▋ | 5293/6100 [7:59:49<1:12:55,  5.42s/it]
 87%|████████▋ | 5294/6100 [7:59:55<1:12:45,  5.42s/it]
 87%|████████▋ | 5295/6100 [8:00:00<1:12:38,  5.41s/it]
                                                       
{'loss': 0.0646, 'learning_rate': 0.0001, 'epoch': 8.68}

 87%|████████▋ | 5295/6100 [8:00:00<1:12:38,  5.41s/it]
 87%|████████▋ | 5296/6100 [8:00:05<1:12:31,  5.41s/it]
 87%|████████▋ | 5297/6100 [8:00:11<1:12:23,  5.41s/it]
 87%|████████▋ | 5298/6100 [8:00:16<1:12:20,  5.41s/it]
 87%|████████▋ | 5299/6100 [8:00:22<1:12:14,  5.41s/it]
 87%|████████▋ | 5300/6100 [8:00:27<1:12:07,  5.41s/it]
                                                       
{'loss': 0.0662, 'learning_rate': 0.0001, 'epoch': 8.69}

 87%|████████▋ | 5300/6100 [8:00:27<1:12:07,  5.41s/it]
 87%|████████▋ | 5301/6100 [8:00:32<1:12:04,  5.41s/it]
 87%|████████▋ | 5302/6100 [8:00:38<1:12:00,  5.41s/it]
 87%|████████▋ | 5303/6100 [8:00:43<1:11:53,  5.41s/it]
 87%|████████▋ | 5304/6100 [8:00:49<1:11:50,  5.42s/it]
 87%|████████▋ | 5305/6100 [8:00:54<1:11:50,  5.42s/it]
                                                       
{'loss': 0.0686, 'learning_rate': 0.0001, 'epoch': 8.7}

 87%|████████▋ | 5305/6100 [8:00:54<1:11:50,  5.42s/it]
 87%|████████▋ | 5306/6100 [8:00:59<1:11:45,  5.42s/it]
 87%|████████▋ | 5307/6100 [8:01:05<1:11:48,  5.43s/it]
 87%|████████▋ | 5308/6100 [8:01:10<1:11:37,  5.43s/it]
 87%|████████▋ | 5309/6100 [8:01:16<1:11:31,  5.43s/it]
 87%|████████▋ | 5310/6100 [8:01:21<1:11:25,  5.42s/it]
                                                       
{'loss': 0.071, 'learning_rate': 0.0001, 'epoch': 8.7}

 87%|████████▋ | 5310/6100 [8:01:21<1:11:25,  5.42s/it]
 87%|████████▋ | 5311/6100 [8:01:27<1:11:21,  5.43s/it]
 87%|████████▋ | 5312/6100 [8:01:32<1:11:11,  5.42s/it]
 87%|████████▋ | 5313/6100 [8:01:37<1:11:07,  5.42s/it]
 87%|████████▋ | 5314/6100 [8:01:43<1:10:57,  5.42s/it]
 87%|████████▋ | 5315/6100 [8:01:48<1:10:55,  5.42s/it]
                                                       
{'loss': 0.0678, 'learning_rate': 0.0001, 'epoch': 8.71}

 87%|████████▋ | 5315/6100 [8:01:48<1:10:55,  5.42s/it]
 87%|████████▋ | 5316/6100 [8:01:54<1:10:48,  5.42s/it]
 87%|████████▋ | 5317/6100 [8:01:59<1:10:39,  5.41s/it]
 87%|████████▋ | 5318/6100 [8:02:05<1:10:48,  5.43s/it]
 87%|████████▋ | 5319/6100 [8:02:10<1:10:38,  5.43s/it]
 87%|████████▋ | 5320/6100 [8:02:15<1:10:34,  5.43s/it]
                                                       
{'loss': 0.0662, 'learning_rate': 0.0001, 'epoch': 8.72}

 87%|████████▋ | 5320/6100 [8:02:15<1:10:34,  5.43s/it]
 87%|████████▋ | 5321/6100 [8:02:21<1:10:26,  5.42s/it]
 87%|████████▋ | 5322/6100 [8:02:26<1:10:17,  5.42s/it]
 87%|████████▋ | 5323/6100 [8:02:32<1:10:10,  5.42s/it]
 87%|████████▋ | 5324/6100 [8:02:37<1:10:05,  5.42s/it]
 87%|████████▋ | 5325/6100 [8:02:43<1:10:03,  5.42s/it]
                                                       
{'loss': 0.0677, 'learning_rate': 0.0001, 'epoch': 8.73}

 87%|████████▋ | 5325/6100 [8:02:43<1:10:03,  5.42s/it]
 87%|████████▋ | 5326/6100 [8:02:48<1:09:55,  5.42s/it]
 87%|████████▋ | 5327/6100 [8:02:53<1:09:50,  5.42s/it]
 87%|████████▋ | 5328/6100 [8:02:59<1:09:45,  5.42s/it]
 87%|████████▋ | 5329/6100 [8:03:04<1:09:36,  5.42s/it]
 87%|████████▋ | 5330/6100 [8:03:10<1:09:29,  5.41s/it]
                                                       
{'loss': 0.0646, 'learning_rate': 0.0001, 'epoch': 8.74}

 87%|████████▋ | 5330/6100 [8:03:10<1:09:29,  5.41s/it]
 87%|████████▋ | 5331/6100 [8:03:15<1:09:25,  5.42s/it]
 87%|████████▋ | 5332/6100 [8:03:20<1:09:20,  5.42s/it]
 87%|████████▋ | 5333/6100 [8:03:26<1:09:12,  5.41s/it]
 87%|████████▋ | 5334/6100 [8:03:31<1:09:07,  5.41s/it]
 87%|████████▋ | 5335/6100 [8:03:37<1:09:02,  5.41s/it]
                                                       
{'loss': 0.0724, 'learning_rate': 0.0001, 'epoch': 8.75}

 87%|████████▋ | 5335/6100 [8:03:37<1:09:02,  5.41s/it]
 87%|████████▋ | 5336/6100 [8:03:42<1:09:00,  5.42s/it]
 87%|████████▋ | 5337/6100 [8:03:48<1:08:56,  5.42s/it]
 88%|████████▊ | 5338/6100 [8:03:53<1:08:46,  5.42s/it]
 88%|████████▊ | 5339/6100 [8:03:58<1:08:42,  5.42s/it]
 88%|████████▊ | 5340/6100 [8:04:04<1:08:36,  5.42s/it]
                                                       
{'loss': 0.0667, 'learning_rate': 0.0001, 'epoch': 8.75}

 88%|████████▊ | 5340/6100 [8:04:04<1:08:36,  5.42s/it]
 88%|████████▊ | 5341/6100 [8:04:09<1:08:35,  5.42s/it]
 88%|████████▊ | 5342/6100 [8:04:15<1:08:33,  5.43s/it]
 88%|████████▊ | 5343/6100 [8:04:20<1:08:25,  5.42s/it]
 88%|████████▊ | 5344/6100 [8:04:26<1:08:18,  5.42s/it]
 88%|████████▊ | 5345/6100 [8:04:31<1:08:10,  5.42s/it]
                                                       
{'loss': 0.0723, 'learning_rate': 0.0001, 'epoch': 8.76}

 88%|████████▊ | 5345/6100 [8:04:31<1:08:10,  5.42s/it]
 88%|████████▊ | 5346/6100 [8:04:36<1:08:01,  5.41s/it]
 88%|████████▊ | 5347/6100 [8:04:42<1:07:57,  5.42s/it]
 88%|████████▊ | 5348/6100 [8:04:47<1:07:53,  5.42s/it]
 88%|████████▊ | 5349/6100 [8:04:53<1:07:48,  5.42s/it]
 88%|████████▊ | 5350/6100 [8:04:58<1:07:40,  5.41s/it]
                                                       
{'loss': 0.0647, 'learning_rate': 0.0001, 'epoch': 8.77}

 88%|████████▊ | 5350/6100 [8:04:58<1:07:40,  5.41s/it]
 88%|████████▊ | 5351/6100 [8:05:03<1:07:35,  5.41s/it]
 88%|████████▊ | 5352/6100 [8:05:09<1:07:31,  5.42s/it]
 88%|████████▊ | 5353/6100 [8:05:14<1:07:24,  5.41s/it]
 88%|████████▊ | 5354/6100 [8:05:20<1:07:23,  5.42s/it]
 88%|████████▊ | 5355/6100 [8:05:25<1:07:15,  5.42s/it]
                                                       
{'loss': 0.0661, 'learning_rate': 0.0001, 'epoch': 8.78}

 88%|████████▊ | 5355/6100 [8:05:25<1:07:15,  5.42s/it]
 88%|████████▊ | 5356/6100 [8:05:30<1:07:07,  5.41s/it]
 88%|████████▊ | 5357/6100 [8:05:36<1:07:03,  5.41s/it]
 88%|████████▊ | 5358/6100 [8:05:41<1:07:00,  5.42s/it]
 88%|████████▊ | 5359/6100 [8:05:47<1:06:52,  5.42s/it]
 88%|████████▊ | 5360/6100 [8:05:52<1:06:48,  5.42s/it]
                                                       
{'loss': 0.0665, 'learning_rate': 0.0001, 'epoch': 8.79}

 88%|████████▊ | 5360/6100 [8:05:52<1:06:48,  5.42s/it]
 88%|████████▊ | 5361/6100 [8:05:58<1:06:41,  5.41s/it]
 88%|████████▊ | 5362/6100 [8:06:03<1:06:34,  5.41s/it]
 88%|████████▊ | 5363/6100 [8:06:08<1:06:30,  5.41s/it]
 88%|████████▊ | 5364/6100 [8:06:14<1:06:24,  5.41s/it]
 88%|████████▊ | 5365/6100 [8:06:19<1:06:18,  5.41s/it]
                                                       
{'loss': 0.0687, 'learning_rate': 0.0001, 'epoch': 8.79}

 88%|████████▊ | 5365/6100 [8:06:19<1:06:18,  5.41s/it]
 88%|████████▊ | 5366/6100 [8:06:25<1:06:17,  5.42s/it]
 88%|████████▊ | 5367/6100 [8:06:30<1:06:09,  5.42s/it]
 88%|████████▊ | 5368/6100 [8:06:35<1:06:00,  5.41s/it]
 88%|████████▊ | 5369/6100 [8:06:41<1:05:54,  5.41s/it]
 88%|████████▊ | 5370/6100 [8:06:46<1:05:47,  5.41s/it]
                                                       
{'loss': 0.0627, 'learning_rate': 0.0001, 'epoch': 8.8}

 88%|████████▊ | 5370/6100 [8:06:46<1:05:47,  5.41s/it]
 88%|████████▊ | 5371/6100 [8:06:52<1:05:43,  5.41s/it]
 88%|████████▊ | 5372/6100 [8:06:57<1:05:38,  5.41s/it]
 88%|████████▊ | 5373/6100 [8:07:03<1:05:53,  5.44s/it]
 88%|████████▊ | 5374/6100 [8:07:08<1:05:55,  5.45s/it]
 88%|████████▊ | 5375/6100 [8:07:13<1:05:42,  5.44s/it]
                                                       
{'loss': 0.0697, 'learning_rate': 0.0001, 'epoch': 8.81}

 88%|████████▊ | 5375/6100 [8:07:13<1:05:42,  5.44s/it]
 88%|████████▊ | 5376/6100 [8:07:19<1:05:32,  5.43s/it]
 88%|████████▊ | 5377/6100 [8:07:24<1:05:25,  5.43s/it]
 88%|████████▊ | 5378/6100 [8:07:30<1:05:16,  5.42s/it]
 88%|████████▊ | 5379/6100 [8:07:35<1:05:10,  5.42s/it]
 88%|████████▊ | 5380/6100 [8:07:41<1:05:04,  5.42s/it]
                                                       
{'loss': 0.0681, 'learning_rate': 0.0001, 'epoch': 8.82}

 88%|████████▊ | 5380/6100 [8:07:41<1:05:04,  5.42s/it]
 88%|████████▊ | 5381/6100 [8:07:46<1:04:59,  5.42s/it]
 88%|████████▊ | 5382/6100 [8:07:51<1:04:50,  5.42s/it]
 88%|████████▊ | 5383/6100 [8:07:57<1:04:44,  5.42s/it]
 88%|████████▊ | 5384/6100 [8:08:02<1:04:42,  5.42s/it]
 88%|████████▊ | 5385/6100 [8:08:08<1:04:35,  5.42s/it]
                                                       
{'loss': 0.0701, 'learning_rate': 0.0001, 'epoch': 8.83}

 88%|████████▊ | 5385/6100 [8:08:08<1:04:35,  5.42s/it]
 88%|████████▊ | 5386/6100 [8:08:13<1:04:27,  5.42s/it]
 88%|████████▊ | 5387/6100 [8:08:19<1:04:26,  5.42s/it]
 88%|████████▊ | 5388/6100 [8:08:24<1:04:21,  5.42s/it]
 88%|████████▊ | 5389/6100 [8:08:29<1:04:12,  5.42s/it]
 88%|████████▊ | 5390/6100 [8:08:35<1:04:06,  5.42s/it]
                                                       
{'loss': 0.0703, 'learning_rate': 0.0001, 'epoch': 8.84}

 88%|████████▊ | 5390/6100 [8:08:35<1:04:06,  5.42s/it]
 88%|████████▊ | 5391/6100 [8:08:40<1:04:09,  5.43s/it]
 88%|████████▊ | 5392/6100 [8:08:46<1:03:58,  5.42s/it]
 88%|████████▊ | 5393/6100 [8:08:51<1:03:53,  5.42s/it]
 88%|████████▊ | 5394/6100 [8:08:56<1:03:45,  5.42s/it]
 88%|████████▊ | 5395/6100 [8:09:02<1:03:37,  5.42s/it]
                                                       
{'loss': 0.0687, 'learning_rate': 0.0001, 'epoch': 8.84}

 88%|████████▊ | 5395/6100 [8:09:02<1:03:37,  5.42s/it]
 88%|████████▊ | 5396/6100 [8:09:07<1:03:34,  5.42s/it]
 88%|████████▊ | 5397/6100 [8:09:13<1:03:27,  5.42s/it]
 88%|████████▊ | 5398/6100 [8:09:18<1:03:22,  5.42s/it]
 89%|████████▊ | 5399/6100 [8:09:24<1:03:21,  5.42s/it]
 89%|████████▊ | 5400/6100 [8:09:29<1:03:16,  5.42s/it]
                                                       
{'loss': 0.0699, 'learning_rate': 0.0001, 'epoch': 8.85}

 89%|████████▊ | 5400/6100 [8:09:29<1:03:16,  5.42s/it]Saving model checkpoint to ./results/checkpoint-5400
Configuration saved in ./results/checkpoint-5400/config.json
Model weights saved in ./results/checkpoint-5400/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-5400/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-5400/special_tokens_map.json
[2023-02-22 04:01:45,142] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step5404 is begin to save!
[2023-02-22 04:01:45,146] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-5400/global_step5404/mp_rank_00_model_states.pt
[2023-02-22 04:01:45,146] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-5400/global_step5404/mp_rank_00_model_states.pt...
[2023-02-22 04:01:45,955] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-5400/global_step5404/mp_rank_00_model_states.pt.
[2023-02-22 04:01:45,957] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-5400/global_step5404/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 04:01:46,129] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-5400/global_step5404/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 04:01:46,130] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-5400/global_step5404/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 04:01:46,130] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5404 is ready now!
Deleting older checkpoint [results/checkpoint-5250] due to args.save_total_limit

 89%|████████▊ | 5401/6100 [8:09:36<1:10:31,  6.05s/it]
 89%|████████▊ | 5402/6100 [8:09:42<1:08:10,  5.86s/it]
 89%|████████▊ | 5403/6100 [8:09:47<1:06:31,  5.73s/it]
 89%|████████▊ | 5404/6100 [8:09:53<1:05:16,  5.63s/it]
 89%|████████▊ | 5405/6100 [8:09:58<1:04:22,  5.56s/it]
                                                       
{'loss': 0.0677, 'learning_rate': 0.0001, 'epoch': 8.86}

 89%|████████▊ | 5405/6100 [8:09:58<1:04:22,  5.56s/it]
 89%|████████▊ | 5406/6100 [8:10:04<1:03:48,  5.52s/it]
 89%|████████▊ | 5407/6100 [8:10:09<1:03:18,  5.48s/it]
 89%|████████▊ | 5408/6100 [8:10:14<1:02:56,  5.46s/it]
 89%|████████▊ | 5409/6100 [8:10:20<1:02:41,  5.44s/it]
 89%|████████▊ | 5410/6100 [8:10:25<1:02:31,  5.44s/it]
                                                       
{'loss': 0.0684, 'learning_rate': 0.0001, 'epoch': 8.87}

 89%|████████▊ | 5410/6100 [8:10:25<1:02:31,  5.44s/it]
 89%|████████▊ | 5411/6100 [8:10:31<1:02:22,  5.43s/it]
 89%|████████▊ | 5412/6100 [8:10:36<1:02:12,  5.43s/it]
 89%|████████▊ | 5413/6100 [8:10:41<1:02:05,  5.42s/it]
 89%|████████▉ | 5414/6100 [8:10:47<1:01:57,  5.42s/it]
 89%|████████▉ | 5415/6100 [8:10:52<1:01:55,  5.42s/it]
                                                       
{'loss': 0.0712, 'learning_rate': 0.0001, 'epoch': 8.88}

 89%|████████▉ | 5415/6100 [8:10:52<1:01:55,  5.42s/it]
 89%|████████▉ | 5416/6100 [8:10:58<1:01:47,  5.42s/it]
 89%|████████▉ | 5417/6100 [8:11:03<1:01:41,  5.42s/it]
 89%|████████▉ | 5418/6100 [8:11:08<1:01:34,  5.42s/it]
 89%|████████▉ | 5419/6100 [8:11:14<1:01:25,  5.41s/it]
 89%|████████▉ | 5420/6100 [8:11:19<1:01:34,  5.43s/it]
                                                       
{'loss': 0.0592, 'learning_rate': 0.0001, 'epoch': 8.88}

 89%|████████▉ | 5420/6100 [8:11:19<1:01:34,  5.43s/it]
 89%|████████▉ | 5421/6100 [8:11:25<1:01:25,  5.43s/it]
 89%|████████▉ | 5422/6100 [8:11:30<1:01:16,  5.42s/it]
 89%|████████▉ | 5423/6100 [8:11:36<1:01:11,  5.42s/it]
 89%|████████▉ | 5424/6100 [8:11:41<1:01:06,  5.42s/it]
 89%|████████▉ | 5425/6100 [8:11:47<1:01:13,  5.44s/it]
                                                       
{'loss': 0.0679, 'learning_rate': 0.0001, 'epoch': 8.89}

 89%|████████▉ | 5425/6100 [8:11:47<1:01:13,  5.44s/it]
 89%|████████▉ | 5426/6100 [8:11:52<1:01:01,  5.43s/it]
 89%|████████▉ | 5427/6100 [8:11:57<1:00:53,  5.43s/it]
 89%|████████▉ | 5428/6100 [8:12:03<1:00:43,  5.42s/it]
 89%|████████▉ | 5429/6100 [8:12:08<1:00:36,  5.42s/it]
 89%|████████▉ | 5430/6100 [8:12:14<1:00:27,  5.41s/it]
                                                       
{'loss': 0.0691, 'learning_rate': 0.0001, 'epoch': 8.9}

 89%|████████▉ | 5430/6100 [8:12:14<1:00:27,  5.41s/it]
 89%|████████▉ | 5431/6100 [8:12:19<1:00:22,  5.41s/it]
 89%|████████▉ | 5432/6100 [8:12:24<1:00:14,  5.41s/it]
 89%|████████▉ | 5433/6100 [8:12:30<1:00:09,  5.41s/it]
 89%|████████▉ | 5434/6100 [8:12:35<1:00:04,  5.41s/it]
 89%|████████▉ | 5435/6100 [8:12:41<1:00:00,  5.41s/it]
                                                       
{'loss': 0.0598, 'learning_rate': 0.0001, 'epoch': 8.91}

 89%|████████▉ | 5435/6100 [8:12:41<1:00:00,  5.41s/it]
 89%|████████▉ | 5436/6100 [8:12:46<59:57,  5.42s/it]  
 89%|████████▉ | 5437/6100 [8:12:52<59:54,  5.42s/it]
 89%|████████▉ | 5438/6100 [8:12:57<59:45,  5.42s/it]
 89%|████████▉ | 5439/6100 [8:13:02<59:35,  5.41s/it]
 89%|████████▉ | 5440/6100 [8:13:08<59:30,  5.41s/it]
                                                     
{'loss': 0.0611, 'learning_rate': 0.0001, 'epoch': 8.92}

 89%|████████▉ | 5440/6100 [8:13:08<59:30,  5.41s/it]
 89%|████████▉ | 5441/6100 [8:13:13<59:39,  5.43s/it]
 89%|████████▉ | 5442/6100 [8:13:19<59:33,  5.43s/it]
 89%|████████▉ | 5443/6100 [8:13:24<59:23,  5.42s/it]
 89%|████████▉ | 5444/6100 [8:13:29<59:15,  5.42s/it]
 89%|████████▉ | 5445/6100 [8:13:35<59:10,  5.42s/it]
                                                     
{'loss': 0.0609, 'learning_rate': 0.0001, 'epoch': 8.93}

 89%|████████▉ | 5445/6100 [8:13:35<59:10,  5.42s/it]
 89%|████████▉ | 5446/6100 [8:13:40<59:04,  5.42s/it]
 89%|████████▉ | 5447/6100 [8:13:46<59:00,  5.42s/it]
 89%|████████▉ | 5448/6100 [8:13:51<58:54,  5.42s/it]
 89%|████████▉ | 5449/6100 [8:13:57<58:45,  5.42s/it]
 89%|████████▉ | 5450/6100 [8:14:02<58:38,  5.41s/it]
                                                     
{'loss': 0.0656, 'learning_rate': 0.0001, 'epoch': 8.93}

 89%|████████▉ | 5450/6100 [8:14:02<58:38,  5.41s/it]
 89%|████████▉ | 5451/6100 [8:14:07<58:31,  5.41s/it]
 89%|████████▉ | 5452/6100 [8:14:13<58:27,  5.41s/it]
 89%|████████▉ | 5453/6100 [8:14:18<58:21,  5.41s/it]
 89%|████████▉ | 5454/6100 [8:14:24<58:20,  5.42s/it]
 89%|████████▉ | 5455/6100 [8:14:29<58:16,  5.42s/it]
                                                     
{'loss': 0.065, 'learning_rate': 0.0001, 'epoch': 8.94}

 89%|████████▉ | 5455/6100 [8:14:29<58:16,  5.42s/it]
 89%|████████▉ | 5456/6100 [8:14:34<58:10,  5.42s/it]
 89%|████████▉ | 5457/6100 [8:14:40<58:04,  5.42s/it]
 89%|████████▉ | 5458/6100 [8:14:45<58:02,  5.42s/it]
 89%|████████▉ | 5459/6100 [8:14:51<57:55,  5.42s/it]
 90%|████████▉ | 5460/6100 [8:14:56<57:46,  5.42s/it]
                                                     
{'loss': 0.0681, 'learning_rate': 0.0001, 'epoch': 8.95}

 90%|████████▉ | 5460/6100 [8:14:56<57:46,  5.42s/it]
 90%|████████▉ | 5461/6100 [8:15:02<57:44,  5.42s/it]
 90%|████████▉ | 5462/6100 [8:15:07<57:37,  5.42s/it]
 90%|████████▉ | 5463/6100 [8:15:12<57:30,  5.42s/it]
 90%|████████▉ | 5464/6100 [8:15:18<57:25,  5.42s/it]
 90%|████████▉ | 5465/6100 [8:15:23<57:17,  5.41s/it]
                                                     
{'loss': 0.0672, 'learning_rate': 0.0001, 'epoch': 8.96}

 90%|████████▉ | 5465/6100 [8:15:23<57:17,  5.41s/it]
 90%|████████▉ | 5466/6100 [8:15:29<57:15,  5.42s/it]
 90%|████████▉ | 5467/6100 [8:15:35<58:42,  5.56s/it]
 90%|████████▉ | 5468/6100 [8:15:40<58:09,  5.52s/it]
 90%|████████▉ | 5469/6100 [8:15:45<57:41,  5.49s/it]
 90%|████████▉ | 5470/6100 [8:15:51<57:19,  5.46s/it]
                                                     
{'loss': 0.0653, 'learning_rate': 0.0001, 'epoch': 8.97}

 90%|████████▉ | 5470/6100 [8:15:51<57:19,  5.46s/it]
 90%|████████▉ | 5471/6100 [8:15:56<57:05,  5.45s/it]
 90%|████████▉ | 5472/6100 [8:16:02<56:52,  5.43s/it]
 90%|████████▉ | 5473/6100 [8:16:07<56:42,  5.43s/it]
 90%|████████▉ | 5474/6100 [8:16:12<56:35,  5.42s/it]
 90%|████████▉ | 5475/6100 [8:16:18<56:28,  5.42s/it]
                                                     
{'loss': 0.0672, 'learning_rate': 0.0001, 'epoch': 8.97}

 90%|████████▉ | 5475/6100 [8:16:18<56:28,  5.42s/it]
 90%|████████▉ | 5476/6100 [8:16:23<56:24,  5.42s/it]
 90%|████████▉ | 5477/6100 [8:16:29<56:16,  5.42s/it]
 90%|████████▉ | 5478/6100 [8:16:34<56:09,  5.42s/it]
 90%|████████▉ | 5479/6100 [8:16:39<56:03,  5.42s/it]
 90%|████████▉ | 5480/6100 [8:16:45<55:55,  5.41s/it]
                                                     
{'loss': 0.0622, 'learning_rate': 0.0001, 'epoch': 8.98}

 90%|████████▉ | 5480/6100 [8:16:45<55:55,  5.41s/it]
 90%|████████▉ | 5481/6100 [8:16:50<55:52,  5.42s/it]
 90%|████████▉ | 5482/6100 [8:16:56<55:46,  5.42s/it]
 90%|████████▉ | 5483/6100 [8:17:01<55:41,  5.42s/it]
 90%|████████▉ | 5484/6100 [8:17:07<55:32,  5.41s/it]
 90%|████████▉ | 5485/6100 [8:17:12<55:27,  5.41s/it]
                                                     
{'loss': 0.0649, 'learning_rate': 0.0001, 'epoch': 8.99}

 90%|████████▉ | 5485/6100 [8:17:12<55:27,  5.41s/it]
 90%|████████▉ | 5486/6100 [8:17:17<55:21,  5.41s/it]
 90%|████████▉ | 5487/6100 [8:17:23<55:18,  5.41s/it]
 90%|████████▉ | 5488/6100 [8:17:28<55:15,  5.42s/it]
 90%|████████▉ | 5489/6100 [8:17:34<55:08,  5.42s/it]
 90%|█████████ | 5490/6100 [8:17:39<55:07,  5.42s/it]
                                                     
{'loss': 0.0682, 'learning_rate': 0.0001, 'epoch': 9.0}

 90%|█████████ | 5490/6100 [8:17:39<55:07,  5.42s/it]
 90%|█████████ | 5491/6100 [8:17:47<1:02:54,  6.20s/it]
 90%|█████████ | 5492/6100 [8:17:53<1:00:30,  5.97s/it]
 90%|█████████ | 5493/6100 [8:17:58<58:42,  5.80s/it]  
 90%|█████████ | 5494/6100 [8:18:03<57:32,  5.70s/it]
 90%|█████████ | 5495/6100 [8:18:09<56:40,  5.62s/it]
                                                     
{'loss': 0.0722, 'learning_rate': 0.0001, 'epoch': 9.01}

 90%|█████████ | 5495/6100 [8:18:09<56:40,  5.62s/it]
 90%|█████████ | 5496/6100 [8:18:14<55:57,  5.56s/it]
 90%|█████████ | 5497/6100 [8:18:20<55:26,  5.52s/it]
 90%|█████████ | 5498/6100 [8:18:25<54:59,  5.48s/it]
 90%|█████████ | 5499/6100 [8:18:30<54:41,  5.46s/it]
 90%|█████████ | 5500/6100 [8:18:36<54:26,  5.44s/it]
                                                     
{'loss': 0.0611, 'learning_rate': 0.0001, 'epoch': 9.02}

 90%|█████████ | 5500/6100 [8:18:36<54:26,  5.44s/it]
 90%|█████████ | 5501/6100 [8:18:41<54:16,  5.44s/it]
 90%|█████████ | 5502/6100 [8:18:47<54:06,  5.43s/it]
 90%|█████████ | 5503/6100 [8:18:52<53:56,  5.42s/it]
 90%|█████████ | 5504/6100 [8:18:58<53:48,  5.42s/it]
 90%|█████████ | 5505/6100 [8:19:03<53:41,  5.41s/it]
                                                     
{'loss': 0.0669, 'learning_rate': 0.0001, 'epoch': 9.02}

 90%|█████████ | 5505/6100 [8:19:03<53:41,  5.41s/it]
 90%|█████████ | 5506/6100 [8:19:08<53:37,  5.42s/it]
 90%|█████████ | 5507/6100 [8:19:14<53:32,  5.42s/it]
 90%|█████████ | 5508/6100 [8:19:19<53:26,  5.42s/it]
 90%|█████████ | 5509/6100 [8:19:25<53:21,  5.42s/it]
 90%|█████████ | 5510/6100 [8:19:30<53:14,  5.42s/it]
                                                     
{'loss': 0.0653, 'learning_rate': 0.0001, 'epoch': 9.03}

 90%|█████████ | 5510/6100 [8:19:30<53:14,  5.42s/it]
 90%|█████████ | 5511/6100 [8:19:35<53:10,  5.42s/it]
 90%|█████████ | 5512/6100 [8:19:41<53:04,  5.42s/it]
 90%|█████████ | 5513/6100 [8:19:46<52:59,  5.42s/it]
 90%|█████████ | 5514/6100 [8:19:52<52:53,  5.42s/it]
 90%|█████████ | 5515/6100 [8:19:57<52:50,  5.42s/it]
                                                     
{'loss': 0.0618, 'learning_rate': 0.0001, 'epoch': 9.04}

 90%|█████████ | 5515/6100 [8:19:57<52:50,  5.42s/it]
 90%|█████████ | 5516/6100 [8:20:03<52:46,  5.42s/it]
 90%|█████████ | 5517/6100 [8:20:08<52:41,  5.42s/it]
 90%|█████████ | 5518/6100 [8:20:13<52:38,  5.43s/it]
 90%|█████████ | 5519/6100 [8:20:19<52:28,  5.42s/it]
 90%|█████████ | 5520/6100 [8:20:24<52:22,  5.42s/it]
                                                     
{'loss': 0.0646, 'learning_rate': 0.0001, 'epoch': 9.05}

 90%|█████████ | 5520/6100 [8:20:24<52:22,  5.42s/it]
 91%|█████████ | 5521/6100 [8:20:30<52:16,  5.42s/it]
 91%|█████████ | 5522/6100 [8:20:35<52:11,  5.42s/it]
 91%|█████████ | 5523/6100 [8:20:40<52:05,  5.42s/it]
 91%|█████████ | 5524/6100 [8:20:46<51:57,  5.41s/it]
 91%|█████████ | 5525/6100 [8:20:51<51:49,  5.41s/it]
                                                     
{'loss': 0.067, 'learning_rate': 0.0001, 'epoch': 9.06}

 91%|█████████ | 5525/6100 [8:20:51<51:49,  5.41s/it]
 91%|█████████ | 5526/6100 [8:20:57<51:45,  5.41s/it]
 91%|█████████ | 5527/6100 [8:21:02<51:39,  5.41s/it]
 91%|█████████ | 5528/6100 [8:21:07<51:34,  5.41s/it]
 91%|█████████ | 5529/6100 [8:21:13<51:29,  5.41s/it]
 91%|█████████ | 5530/6100 [8:21:18<51:26,  5.41s/it]
                                                     
{'loss': 0.0654, 'learning_rate': 0.0001, 'epoch': 9.07}

 91%|█████████ | 5530/6100 [8:21:18<51:26,  5.41s/it]
 91%|█████████ | 5531/6100 [8:21:24<51:22,  5.42s/it]
 91%|█████████ | 5532/6100 [8:21:29<51:15,  5.41s/it]
 91%|█████████ | 5533/6100 [8:21:35<51:11,  5.42s/it]
 91%|█████████ | 5534/6100 [8:21:40<51:04,  5.41s/it]
 91%|█████████ | 5535/6100 [8:21:45<50:58,  5.41s/it]
                                                     
{'loss': 0.0637, 'learning_rate': 0.0001, 'epoch': 9.07}

 91%|█████████ | 5535/6100 [8:21:45<50:58,  5.41s/it]
 91%|█████████ | 5536/6100 [8:21:51<50:53,  5.41s/it]
 91%|█████████ | 5537/6100 [8:21:56<50:47,  5.41s/it]
 91%|█████████ | 5538/6100 [8:22:02<50:42,  5.41s/it]
 91%|█████████ | 5539/6100 [8:22:07<50:36,  5.41s/it]
 91%|█████████ | 5540/6100 [8:22:12<50:31,  5.41s/it]
                                                     
{'loss': 0.0662, 'learning_rate': 0.0001, 'epoch': 9.08}

 91%|█████████ | 5540/6100 [8:22:12<50:31,  5.41s/it]
 91%|█████████ | 5541/6100 [8:22:18<50:29,  5.42s/it]
 91%|█████████ | 5542/6100 [8:22:23<50:24,  5.42s/it]
 91%|█████████ | 5543/6100 [8:22:29<50:32,  5.44s/it]
 91%|█████████ | 5544/6100 [8:22:34<50:19,  5.43s/it]
 91%|█████████ | 5545/6100 [8:22:40<50:09,  5.42s/it]
                                                     
{'loss': 0.0635, 'learning_rate': 0.0001, 'epoch': 9.09}

 91%|█████████ | 5545/6100 [8:22:40<50:09,  5.42s/it]
 91%|█████████ | 5546/6100 [8:22:45<50:02,  5.42s/it]
 91%|█████████ | 5547/6100 [8:22:50<49:55,  5.42s/it]
 91%|█████████ | 5548/6100 [8:22:56<49:53,  5.42s/it]
 91%|█████████ | 5549/6100 [8:23:01<49:49,  5.43s/it]
 91%|█████████ | 5550/6100 [8:23:07<49:42,  5.42s/it]
                                                     
{'loss': 0.0662, 'learning_rate': 0.0001, 'epoch': 9.1}

 91%|█████████ | 5550/6100 [8:23:07<49:42,  5.42s/it]Saving model checkpoint to ./results/checkpoint-5550
Configuration saved in ./results/checkpoint-5550/config.json
Model weights saved in ./results/checkpoint-5550/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-5550/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-5550/special_tokens_map.json
[2023-02-22 04:15:22,904] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step5554 is begin to save!
[2023-02-22 04:15:22,907] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-5550/global_step5554/mp_rank_00_model_states.pt
[2023-02-22 04:15:22,907] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-5550/global_step5554/mp_rank_00_model_states.pt...
[2023-02-22 04:15:23,723] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-5550/global_step5554/mp_rank_00_model_states.pt.
[2023-02-22 04:15:23,725] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-5550/global_step5554/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 04:15:23,897] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-5550/global_step5554/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 04:15:23,898] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-5550/global_step5554/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 04:15:23,898] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5554 is ready now!
Deleting older checkpoint [results/checkpoint-5400] due to args.save_total_limit

 91%|█████████ | 5551/6100 [8:23:14<55:27,  6.06s/it]
 91%|█████████ | 5552/6100 [8:23:20<53:36,  5.87s/it]
 91%|█████████ | 5553/6100 [8:23:25<52:14,  5.73s/it]
 91%|█████████ | 5554/6100 [8:23:31<51:29,  5.66s/it]
 91%|█████████ | 5555/6100 [8:23:36<50:45,  5.59s/it]
                                                     
{'loss': 0.0641, 'learning_rate': 0.0001, 'epoch': 9.11}

 91%|█████████ | 5555/6100 [8:23:36<50:45,  5.59s/it]
 91%|█████████ | 5556/6100 [8:23:41<50:11,  5.54s/it]
 91%|█████████ | 5557/6100 [8:23:47<49:44,  5.50s/it]
 91%|█████████ | 5558/6100 [8:23:52<49:27,  5.47s/it]
 91%|█████████ | 5559/6100 [8:23:58<49:11,  5.46s/it]
 91%|█████████ | 5560/6100 [8:24:03<49:00,  5.45s/it]
                                                     
{'loss': 0.0602, 'learning_rate': 0.0001, 'epoch': 9.11}

 91%|█████████ | 5560/6100 [8:24:03<49:00,  5.45s/it]
 91%|█████████ | 5561/6100 [8:24:09<48:51,  5.44s/it]
 91%|█████████ | 5562/6100 [8:24:14<48:41,  5.43s/it]
 91%|█████████ | 5563/6100 [8:24:19<48:33,  5.43s/it]
 91%|█████████ | 5564/6100 [8:24:25<48:24,  5.42s/it]
 91%|█████████ | 5565/6100 [8:24:30<48:15,  5.41s/it]
                                                     
{'loss': 0.065, 'learning_rate': 0.0001, 'epoch': 9.12}

 91%|█████████ | 5565/6100 [8:24:30<48:15,  5.41s/it]
 91%|█████████ | 5566/6100 [8:24:36<48:09,  5.41s/it]
 91%|█████████▏| 5567/6100 [8:24:41<48:04,  5.41s/it]
 91%|█████████▏| 5568/6100 [8:24:46<47:59,  5.41s/it]
 91%|█████████▏| 5569/6100 [8:24:52<47:55,  5.41s/it]
 91%|█████████▏| 5570/6100 [8:24:57<47:50,  5.42s/it]
                                                     
{'loss': 0.0628, 'learning_rate': 0.0001, 'epoch': 9.13}

 91%|█████████▏| 5570/6100 [8:24:57<47:50,  5.42s/it]
 91%|█████████▏| 5571/6100 [8:25:03<47:47,  5.42s/it]
 91%|█████████▏| 5572/6100 [8:25:08<47:40,  5.42s/it]
 91%|█████████▏| 5573/6100 [8:25:13<47:33,  5.41s/it]
 91%|█████████▏| 5574/6100 [8:25:19<47:29,  5.42s/it]
 91%|█████████▏| 5575/6100 [8:25:24<47:24,  5.42s/it]
                                                     
{'loss': 0.0655, 'learning_rate': 0.0001, 'epoch': 9.14}

 91%|█████████▏| 5575/6100 [8:25:24<47:24,  5.42s/it]
 91%|█████████▏| 5576/6100 [8:25:30<47:20,  5.42s/it]
 91%|█████████▏| 5577/6100 [8:25:35<47:15,  5.42s/it]
 91%|█████████▏| 5578/6100 [8:25:41<47:09,  5.42s/it]
 91%|█████████▏| 5579/6100 [8:25:46<47:01,  5.42s/it]
 91%|█████████▏| 5580/6100 [8:25:51<46:56,  5.42s/it]
                                                     
{'loss': 0.0618, 'learning_rate': 0.0001, 'epoch': 9.15}

 91%|█████████▏| 5580/6100 [8:25:51<46:56,  5.42s/it]
 91%|█████████▏| 5581/6100 [8:25:57<46:50,  5.41s/it]
 92%|█████████▏| 5582/6100 [8:26:02<46:47,  5.42s/it]
 92%|█████████▏| 5583/6100 [8:26:08<46:41,  5.42s/it]
 92%|█████████▏| 5584/6100 [8:26:13<46:34,  5.42s/it]
 92%|█████████▏| 5585/6100 [8:26:18<46:29,  5.42s/it]
                                                     
{'loss': 0.0623, 'learning_rate': 0.0001, 'epoch': 9.16}

 92%|█████████▏| 5585/6100 [8:26:19<46:29,  5.42s/it]
 92%|█████████▏| 5586/6100 [8:26:24<46:23,  5.42s/it]
 92%|█████████▏| 5587/6100 [8:26:30<47:52,  5.60s/it]
 92%|█████████▏| 5588/6100 [8:26:35<47:19,  5.55s/it]
 92%|█████████▏| 5589/6100 [8:26:41<46:55,  5.51s/it]
 92%|█████████▏| 5590/6100 [8:26:46<46:38,  5.49s/it]
                                                     
{'loss': 0.0665, 'learning_rate': 0.0001, 'epoch': 9.16}

 92%|█████████▏| 5590/6100 [8:26:46<46:38,  5.49s/it]
 92%|█████████▏| 5591/6100 [8:26:52<46:25,  5.47s/it]
 92%|█████████▏| 5592/6100 [8:26:57<46:11,  5.45s/it]
 92%|█████████▏| 5593/6100 [8:27:03<46:01,  5.45s/it]
 92%|█████████▏| 5594/6100 [8:27:08<45:52,  5.44s/it]
 92%|█████████▏| 5595/6100 [8:27:13<45:43,  5.43s/it]
                                                     
{'loss': 0.0635, 'learning_rate': 0.0001, 'epoch': 9.17}

 92%|█████████▏| 5595/6100 [8:27:13<45:43,  5.43s/it]
 92%|█████████▏| 5596/6100 [8:27:19<45:36,  5.43s/it]
 92%|█████████▏| 5597/6100 [8:27:24<45:29,  5.43s/it]
 92%|█████████▏| 5598/6100 [8:27:30<45:20,  5.42s/it]
 92%|█████████▏| 5599/6100 [8:27:35<45:17,  5.42s/it]
 92%|█████████▏| 5600/6100 [8:27:40<45:14,  5.43s/it]
                                                     
{'loss': 0.0675, 'learning_rate': 0.0001, 'epoch': 9.18}

 92%|█████████▏| 5600/6100 [8:27:40<45:14,  5.43s/it]
 92%|█████████▏| 5601/6100 [8:27:46<45:05,  5.42s/it]
 92%|█████████▏| 5602/6100 [8:27:51<44:58,  5.42s/it]
 92%|█████████▏| 5603/6100 [8:27:57<44:54,  5.42s/it]
 92%|█████████▏| 5604/6100 [8:28:02<44:45,  5.41s/it]
 92%|█████████▏| 5605/6100 [8:28:08<44:40,  5.41s/it]
                                                     
{'loss': 0.0607, 'learning_rate': 0.0001, 'epoch': 9.19}

 92%|█████████▏| 5605/6100 [8:28:08<44:40,  5.41s/it]
 92%|█████████▏| 5606/6100 [8:28:13<44:34,  5.41s/it]
 92%|█████████▏| 5607/6100 [8:28:18<44:45,  5.45s/it]
 92%|█████████▏| 5608/6100 [8:28:24<44:34,  5.44s/it]
 92%|█████████▏| 5609/6100 [8:28:29<44:24,  5.43s/it]
 92%|█████████▏| 5610/6100 [8:28:35<44:17,  5.42s/it]
                                                     
{'loss': 0.0616, 'learning_rate': 0.0001, 'epoch': 9.2}

 92%|█████████▏| 5610/6100 [8:28:35<44:17,  5.42s/it]
 92%|█████████▏| 5611/6100 [8:28:40<44:11,  5.42s/it]
 92%|█████████▏| 5612/6100 [8:28:46<44:04,  5.42s/it]
 92%|█████████▏| 5613/6100 [8:28:51<43:59,  5.42s/it]
 92%|█████████▏| 5614/6100 [8:28:56<43:53,  5.42s/it]
 92%|█████████▏| 5615/6100 [8:29:02<43:47,  5.42s/it]
                                                     
{'loss': 0.0617, 'learning_rate': 0.0001, 'epoch': 9.2}

 92%|█████████▏| 5615/6100 [8:29:02<43:47,  5.42s/it]
 92%|█████████▏| 5616/6100 [8:29:07<43:41,  5.42s/it]
 92%|█████████▏| 5617/6100 [8:29:13<43:34,  5.41s/it]
 92%|█████████▏| 5618/6100 [8:29:18<43:30,  5.41s/it]
 92%|█████████▏| 5619/6100 [8:29:23<43:26,  5.42s/it]
 92%|█████████▏| 5620/6100 [8:29:29<43:18,  5.41s/it]
                                                     
{'loss': 0.0631, 'learning_rate': 0.0001, 'epoch': 9.21}

 92%|█████████▏| 5620/6100 [8:29:29<43:18,  5.41s/it]
 92%|█████████▏| 5621/6100 [8:29:34<43:15,  5.42s/it]
 92%|█████████▏| 5622/6100 [8:29:40<43:09,  5.42s/it]
 92%|█████████▏| 5623/6100 [8:29:45<43:03,  5.42s/it]
 92%|█████████▏| 5624/6100 [8:29:51<42:59,  5.42s/it]
 92%|█████████▏| 5625/6100 [8:29:56<42:55,  5.42s/it]
                                                     
{'loss': 0.0649, 'learning_rate': 0.0001, 'epoch': 9.22}

 92%|█████████▏| 5625/6100 [8:29:56<42:55,  5.42s/it]
 92%|█████████▏| 5626/6100 [8:30:01<42:49,  5.42s/it]
 92%|█████████▏| 5627/6100 [8:30:07<42:43,  5.42s/it]
 92%|█████████▏| 5628/6100 [8:30:12<42:35,  5.41s/it]
 92%|█████████▏| 5629/6100 [8:30:18<42:29,  5.41s/it]
 92%|█████████▏| 5630/6100 [8:30:23<42:23,  5.41s/it]
                                                     
{'loss': 0.0636, 'learning_rate': 0.0001, 'epoch': 9.23}

 92%|█████████▏| 5630/6100 [8:30:23<42:23,  5.41s/it]
 92%|█████████▏| 5631/6100 [8:30:28<42:19,  5.42s/it]
 92%|█████████▏| 5632/6100 [8:30:34<42:13,  5.41s/it]
 92%|█████████▏| 5633/6100 [8:30:39<42:06,  5.41s/it]
 92%|█████████▏| 5634/6100 [8:30:45<42:02,  5.41s/it]
 92%|█████████▏| 5635/6100 [8:30:50<41:56,  5.41s/it]
                                                     
{'loss': 0.0608, 'learning_rate': 0.0001, 'epoch': 9.24}

 92%|█████████▏| 5635/6100 [8:30:50<41:56,  5.41s/it]
 92%|█████████▏| 5636/6100 [8:30:55<41:53,  5.42s/it]
 92%|█████████▏| 5637/6100 [8:31:01<41:47,  5.42s/it]
 92%|█████████▏| 5638/6100 [8:31:06<41:42,  5.42s/it]
 92%|█████████▏| 5639/6100 [8:31:12<41:37,  5.42s/it]
 92%|█████████▏| 5640/6100 [8:31:17<41:31,  5.42s/it]
                                                     
{'loss': 0.0585, 'learning_rate': 0.0001, 'epoch': 9.25}

 92%|█████████▏| 5640/6100 [8:31:17<41:31,  5.42s/it]
 92%|█████████▏| 5641/6100 [8:31:23<41:25,  5.42s/it]
 92%|█████████▏| 5642/6100 [8:31:28<41:17,  5.41s/it]
 93%|█████████▎| 5643/6100 [8:31:33<41:13,  5.41s/it]
 93%|█████████▎| 5644/6100 [8:31:39<41:09,  5.42s/it]
 93%|█████████▎| 5645/6100 [8:31:44<41:06,  5.42s/it]
                                                     
{'loss': 0.0669, 'learning_rate': 0.0001, 'epoch': 9.25}

 93%|█████████▎| 5645/6100 [8:31:44<41:06,  5.42s/it]
 93%|█████████▎| 5646/6100 [8:31:50<41:01,  5.42s/it]
 93%|█████████▎| 5647/6100 [8:31:55<40:57,  5.42s/it]
 93%|█████████▎| 5648/6100 [8:32:01<40:49,  5.42s/it]
 93%|█████████▎| 5649/6100 [8:32:06<40:43,  5.42s/it]
 93%|█████████▎| 5650/6100 [8:32:11<40:39,  5.42s/it]
                                                     
{'loss': 0.0615, 'learning_rate': 0.0001, 'epoch': 9.26}

 93%|█████████▎| 5650/6100 [8:32:11<40:39,  5.42s/it]
 93%|█████████▎| 5651/6100 [8:32:17<40:31,  5.42s/it]
 93%|█████████▎| 5652/6100 [8:32:22<40:30,  5.43s/it]
 93%|█████████▎| 5653/6100 [8:32:28<40:24,  5.42s/it]
 93%|█████████▎| 5654/6100 [8:32:33<40:15,  5.42s/it]
 93%|█████████▎| 5655/6100 [8:32:38<40:09,  5.42s/it]
                                                     
{'loss': 0.0613, 'learning_rate': 0.0001, 'epoch': 9.27}

 93%|█████████▎| 5655/6100 [8:32:38<40:09,  5.42s/it]
 93%|█████████▎| 5656/6100 [8:32:44<40:03,  5.41s/it]
 93%|█████████▎| 5657/6100 [8:32:49<39:59,  5.42s/it]
 93%|█████████▎| 5658/6100 [8:32:55<39:53,  5.41s/it]
 93%|█████████▎| 5659/6100 [8:33:00<39:49,  5.42s/it]
 93%|█████████▎| 5660/6100 [8:33:06<39:44,  5.42s/it]
                                                     
{'loss': 0.0658, 'learning_rate': 0.0001, 'epoch': 9.28}

 93%|█████████▎| 5660/6100 [8:33:06<39:44,  5.42s/it]
 93%|█████████▎| 5661/6100 [8:33:11<39:40,  5.42s/it]
 93%|█████████▎| 5662/6100 [8:33:16<39:33,  5.42s/it]
 93%|█████████▎| 5663/6100 [8:33:22<39:30,  5.42s/it]
 93%|█████████▎| 5664/6100 [8:33:27<39:25,  5.42s/it]
 93%|█████████▎| 5665/6100 [8:33:33<39:18,  5.42s/it]
                                                     
{'loss': 0.0645, 'learning_rate': 0.0001, 'epoch': 9.29}

 93%|█████████▎| 5665/6100 [8:33:33<39:18,  5.42s/it]
 93%|█████████▎| 5666/6100 [8:33:38<39:12,  5.42s/it]
 93%|█████████▎| 5667/6100 [8:33:43<39:05,  5.42s/it]
 93%|█████████▎| 5668/6100 [8:33:49<38:58,  5.41s/it]
 93%|█████████▎| 5669/6100 [8:33:54<38:55,  5.42s/it]
 93%|█████████▎| 5670/6100 [8:34:00<38:49,  5.42s/it]
                                                     
{'loss': 0.0689, 'learning_rate': 0.0001, 'epoch': 9.29}

 93%|█████████▎| 5670/6100 [8:34:00<38:49,  5.42s/it]
 93%|█████████▎| 5671/6100 [8:34:05<38:45,  5.42s/it]
 93%|█████████▎| 5672/6100 [8:34:11<38:38,  5.42s/it]
 93%|█████████▎| 5673/6100 [8:34:16<38:32,  5.42s/it]
 93%|█████████▎| 5674/6100 [8:34:21<38:28,  5.42s/it]
 93%|█████████▎| 5675/6100 [8:34:27<38:21,  5.42s/it]
                                                     
{'loss': 0.0661, 'learning_rate': 0.0001, 'epoch': 9.3}

 93%|█████████▎| 5675/6100 [8:34:27<38:21,  5.42s/it]
 93%|█████████▎| 5676/6100 [8:34:32<38:17,  5.42s/it]
 93%|█████████▎| 5677/6100 [8:34:38<38:11,  5.42s/it]
 93%|█████████▎| 5678/6100 [8:34:43<38:07,  5.42s/it]
 93%|█████████▎| 5679/6100 [8:34:48<38:01,  5.42s/it]
 93%|█████████▎| 5680/6100 [8:34:54<37:55,  5.42s/it]
                                                     
{'loss': 0.0639, 'learning_rate': 0.0001, 'epoch': 9.31}

 93%|█████████▎| 5680/6100 [8:34:54<37:55,  5.42s/it]
 93%|█████████▎| 5681/6100 [8:34:59<37:49,  5.42s/it]
 93%|█████████▎| 5682/6100 [8:35:05<37:44,  5.42s/it]
 93%|█████████▎| 5683/6100 [8:35:10<37:38,  5.42s/it]
 93%|█████████▎| 5684/6100 [8:35:16<37:34,  5.42s/it]
 93%|█████████▎| 5685/6100 [8:35:21<37:27,  5.42s/it]
                                                     
{'loss': 0.0667, 'learning_rate': 0.0001, 'epoch': 9.32}

 93%|█████████▎| 5685/6100 [8:35:21<37:27,  5.42s/it]
 93%|█████████▎| 5686/6100 [8:35:26<37:22,  5.42s/it]
 93%|█████████▎| 5687/6100 [8:35:32<37:17,  5.42s/it]
 93%|█████████▎| 5688/6100 [8:35:37<37:12,  5.42s/it]
 93%|█████████▎| 5689/6100 [8:35:43<37:08,  5.42s/it]
 93%|█████████▎| 5690/6100 [8:35:48<36:59,  5.41s/it]
                                                     
{'loss': 0.0675, 'learning_rate': 0.0001, 'epoch': 9.33}

 93%|█████████▎| 5690/6100 [8:35:48<36:59,  5.41s/it]
 93%|█████████▎| 5691/6100 [8:35:53<36:55,  5.42s/it]
 93%|█████████▎| 5692/6100 [8:35:59<36:51,  5.42s/it]
 93%|█████████▎| 5693/6100 [8:36:04<36:46,  5.42s/it]
 93%|█████████▎| 5694/6100 [8:36:10<36:39,  5.42s/it]
 93%|█████████▎| 5695/6100 [8:36:15<36:33,  5.42s/it]
                                                     
{'loss': 0.0652, 'learning_rate': 0.0001, 'epoch': 9.34}

 93%|█████████▎| 5695/6100 [8:36:15<36:33,  5.42s/it]
 93%|█████████▎| 5696/6100 [8:36:21<36:29,  5.42s/it]
 93%|█████████▎| 5697/6100 [8:36:26<36:23,  5.42s/it]
 93%|█████████▎| 5698/6100 [8:36:31<36:18,  5.42s/it]
 93%|█████████▎| 5699/6100 [8:36:37<36:26,  5.45s/it]
 93%|█████████▎| 5700/6100 [8:36:42<36:19,  5.45s/it]
                                                     
{'loss': 0.0656, 'learning_rate': 0.0001, 'epoch': 9.34}

 93%|█████████▎| 5700/6100 [8:36:42<36:19,  5.45s/it]Saving model checkpoint to ./results/checkpoint-5700
Configuration saved in ./results/checkpoint-5700/config.json
Model weights saved in ./results/checkpoint-5700/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-5700/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-5700/special_tokens_map.json
[2023-02-22 04:28:58,565] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step5704 is begin to save!
[2023-02-22 04:28:58,569] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-5700/global_step5704/mp_rank_00_model_states.pt
[2023-02-22 04:28:58,569] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-5700/global_step5704/mp_rank_00_model_states.pt...
[2023-02-22 04:28:59,389] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-5700/global_step5704/mp_rank_00_model_states.pt.
[2023-02-22 04:28:59,391] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-5700/global_step5704/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 04:28:59,563] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-5700/global_step5704/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 04:28:59,563] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-5700/global_step5704/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 04:28:59,564] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5704 is ready now!
Deleting older checkpoint [results/checkpoint-5550] due to args.save_total_limit

 93%|█████████▎| 5701/6100 [8:36:50<40:28,  6.09s/it]
 93%|█████████▎| 5702/6100 [8:36:55<39:02,  5.88s/it]
 93%|█████████▎| 5703/6100 [8:37:01<38:01,  5.75s/it]
 94%|█████████▎| 5704/6100 [8:37:06<37:17,  5.65s/it]
 94%|█████████▎| 5705/6100 [8:37:12<36:44,  5.58s/it]
                                                     
{'loss': 0.0647, 'learning_rate': 0.0001, 'epoch': 9.35}

 94%|█████████▎| 5705/6100 [8:37:12<36:44,  5.58s/it]
 94%|█████████▎| 5706/6100 [8:37:17<36:18,  5.53s/it]
 94%|█████████▎| 5707/6100 [8:37:22<36:00,  5.50s/it]
 94%|█████████▎| 5708/6100 [8:37:28<35:44,  5.47s/it]
 94%|█████████▎| 5709/6100 [8:37:33<35:33,  5.46s/it]
 94%|█████████▎| 5710/6100 [8:37:39<35:22,  5.44s/it]
                                                     
{'loss': 0.0605, 'learning_rate': 0.0001, 'epoch': 9.36}

 94%|█████████▎| 5710/6100 [8:37:39<35:22,  5.44s/it]
 94%|█████████▎| 5711/6100 [8:37:44<35:13,  5.43s/it]
 94%|█████████▎| 5712/6100 [8:37:50<35:05,  5.43s/it]
 94%|█████████▎| 5713/6100 [8:37:55<34:57,  5.42s/it]
 94%|█████████▎| 5714/6100 [8:38:00<34:54,  5.43s/it]
 94%|█████████▎| 5715/6100 [8:38:06<34:46,  5.42s/it]
                                                     
{'loss': 0.0646, 'learning_rate': 0.0001, 'epoch': 9.37}

 94%|█████████▎| 5715/6100 [8:38:06<34:46,  5.42s/it]
 94%|█████████▎| 5716/6100 [8:38:11<34:39,  5.42s/it]
 94%|█████████▎| 5717/6100 [8:38:17<34:36,  5.42s/it]
 94%|█████████▎| 5718/6100 [8:38:22<34:31,  5.42s/it]
 94%|█████████▍| 5719/6100 [8:38:27<34:25,  5.42s/it]
 94%|█████████▍| 5720/6100 [8:38:33<34:18,  5.42s/it]
                                                     
{'loss': 0.0652, 'learning_rate': 0.0001, 'epoch': 9.38}

 94%|█████████▍| 5720/6100 [8:38:33<34:18,  5.42s/it]
 94%|█████████▍| 5721/6100 [8:38:38<34:14,  5.42s/it]
 94%|█████████▍| 5722/6100 [8:38:44<34:11,  5.43s/it]
 94%|█████████▍| 5723/6100 [8:38:49<34:08,  5.43s/it]
 94%|█████████▍| 5724/6100 [8:38:55<34:01,  5.43s/it]
 94%|█████████▍| 5725/6100 [8:39:00<33:54,  5.43s/it]
                                                     
{'loss': 0.0616, 'learning_rate': 0.0001, 'epoch': 9.38}

 94%|█████████▍| 5725/6100 [8:39:00<33:54,  5.43s/it]
 94%|█████████▍| 5726/6100 [8:39:05<33:51,  5.43s/it]
 94%|█████████▍| 5727/6100 [8:39:11<33:43,  5.42s/it]
 94%|█████████▍| 5728/6100 [8:39:16<33:37,  5.42s/it]
 94%|█████████▍| 5729/6100 [8:39:22<33:30,  5.42s/it]
 94%|█████████▍| 5730/6100 [8:39:27<33:24,  5.42s/it]
                                                     
{'loss': 0.063, 'learning_rate': 0.0001, 'epoch': 9.39}

 94%|█████████▍| 5730/6100 [8:39:27<33:24,  5.42s/it]
 94%|█████████▍| 5731/6100 [8:39:33<33:19,  5.42s/it]
 94%|█████████▍| 5732/6100 [8:39:38<33:14,  5.42s/it]
 94%|█████████▍| 5733/6100 [8:39:43<33:09,  5.42s/it]
 94%|█████████▍| 5734/6100 [8:39:49<33:04,  5.42s/it]
 94%|█████████▍| 5735/6100 [8:39:54<32:57,  5.42s/it]
                                                     
{'loss': 0.0662, 'learning_rate': 0.0001, 'epoch': 9.4}

 94%|█████████▍| 5735/6100 [8:39:54<32:57,  5.42s/it]
 94%|█████████▍| 5736/6100 [8:40:00<32:51,  5.42s/it]
 94%|█████████▍| 5737/6100 [8:40:05<32:46,  5.42s/it]
 94%|█████████▍| 5738/6100 [8:40:10<32:40,  5.41s/it]
 94%|█████████▍| 5739/6100 [8:40:16<32:36,  5.42s/it]
 94%|█████████▍| 5740/6100 [8:40:21<32:33,  5.43s/it]
                                                     
{'loss': 0.0635, 'learning_rate': 0.0001, 'epoch': 9.41}

 94%|█████████▍| 5740/6100 [8:40:21<32:33,  5.43s/it]
 94%|█████████▍| 5741/6100 [8:40:27<32:27,  5.43s/it]
 94%|█████████▍| 5742/6100 [8:40:32<32:20,  5.42s/it]
 94%|█████████▍| 5743/6100 [8:40:38<32:42,  5.50s/it]
 94%|█████████▍| 5744/6100 [8:40:43<32:27,  5.47s/it]
 94%|█████████▍| 5745/6100 [8:40:49<32:16,  5.45s/it]
                                                     
{'loss': 0.0638, 'learning_rate': 0.0001, 'epoch': 9.42}

 94%|█████████▍| 5745/6100 [8:40:49<32:16,  5.45s/it]
 94%|█████████▍| 5746/6100 [8:40:54<32:06,  5.44s/it]
 94%|█████████▍| 5747/6100 [8:41:00<31:57,  5.43s/it]
 94%|█████████▍| 5748/6100 [8:41:05<31:49,  5.42s/it]
 94%|█████████▍| 5749/6100 [8:41:10<31:42,  5.42s/it]
 94%|█████████▍| 5750/6100 [8:41:16<31:38,  5.43s/it]
                                                     
{'loss': 0.0651, 'learning_rate': 0.0001, 'epoch': 9.43}

 94%|█████████▍| 5750/6100 [8:41:16<31:38,  5.43s/it]
 94%|█████████▍| 5751/6100 [8:41:21<31:32,  5.42s/it]
 94%|█████████▍| 5752/6100 [8:41:27<31:27,  5.42s/it]
 94%|█████████▍| 5753/6100 [8:41:32<31:21,  5.42s/it]
 94%|█████████▍| 5754/6100 [8:41:37<31:15,  5.42s/it]
 94%|█████████▍| 5755/6100 [8:41:43<31:09,  5.42s/it]
                                                     
{'loss': 0.0606, 'learning_rate': 0.0001, 'epoch': 9.43}

 94%|█████████▍| 5755/6100 [8:41:43<31:09,  5.42s/it]
 94%|█████████▍| 5756/6100 [8:41:48<31:05,  5.42s/it]
 94%|█████████▍| 5757/6100 [8:41:54<30:59,  5.42s/it]
 94%|█████████▍| 5758/6100 [8:41:59<30:53,  5.42s/it]
 94%|█████████▍| 5759/6100 [8:42:05<30:49,  5.42s/it]
 94%|█████████▍| 5760/6100 [8:42:10<30:43,  5.42s/it]
                                                     
{'loss': 0.0617, 'learning_rate': 0.0001, 'epoch': 9.44}

 94%|█████████▍| 5760/6100 [8:42:10<30:43,  5.42s/it]
 94%|█████████▍| 5761/6100 [8:42:15<30:40,  5.43s/it]
 94%|█████████▍| 5762/6100 [8:42:21<30:34,  5.43s/it]
 94%|█████████▍| 5763/6100 [8:42:26<30:28,  5.42s/it]
 94%|█████████▍| 5764/6100 [8:42:32<30:23,  5.43s/it]
 95%|█████████▍| 5765/6100 [8:42:37<30:18,  5.43s/it]
                                                     
{'loss': 0.0654, 'learning_rate': 0.0001, 'epoch': 9.45}

 95%|█████████▍| 5765/6100 [8:42:37<30:18,  5.43s/it]
 95%|█████████▍| 5766/6100 [8:42:43<30:10,  5.42s/it]
 95%|█████████▍| 5767/6100 [8:42:48<30:04,  5.42s/it]
 95%|█████████▍| 5768/6100 [8:42:53<29:57,  5.41s/it]
 95%|█████████▍| 5769/6100 [8:42:59<29:53,  5.42s/it]
 95%|█████████▍| 5770/6100 [8:43:04<29:47,  5.42s/it]
                                                     
{'loss': 0.063, 'learning_rate': 0.0001, 'epoch': 9.46}

 95%|█████████▍| 5770/6100 [8:43:04<29:47,  5.42s/it]
 95%|█████████▍| 5771/6100 [8:43:10<29:42,  5.42s/it]
 95%|█████████▍| 5772/6100 [8:43:15<29:35,  5.41s/it]
 95%|█████████▍| 5773/6100 [8:43:20<29:28,  5.41s/it]
 95%|█████████▍| 5774/6100 [8:43:26<29:24,  5.41s/it]
 95%|█████████▍| 5775/6100 [8:43:31<29:19,  5.41s/it]
                                                     
{'loss': 0.0589, 'learning_rate': 0.0001, 'epoch': 9.47}

 95%|█████████▍| 5775/6100 [8:43:31<29:19,  5.41s/it]
 95%|█████████▍| 5776/6100 [8:43:37<29:14,  5.42s/it]
 95%|█████████▍| 5777/6100 [8:43:42<29:08,  5.41s/it]
 95%|█████████▍| 5778/6100 [8:43:48<29:05,  5.42s/it]
 95%|█████████▍| 5779/6100 [8:43:53<28:59,  5.42s/it]
 95%|█████████▍| 5780/6100 [8:43:58<28:53,  5.42s/it]
                                                     
{'loss': 0.0682, 'learning_rate': 0.0001, 'epoch': 9.48}

 95%|█████████▍| 5780/6100 [8:43:58<28:53,  5.42s/it]
 95%|█████████▍| 5781/6100 [8:44:04<28:47,  5.42s/it]
 95%|█████████▍| 5782/6100 [8:44:09<28:44,  5.42s/it]
 95%|█████████▍| 5783/6100 [8:44:15<28:39,  5.42s/it]
 95%|█████████▍| 5784/6100 [8:44:20<28:34,  5.42s/it]
 95%|█████████▍| 5785/6100 [8:44:25<28:29,  5.43s/it]
                                                     
{'loss': 0.063, 'learning_rate': 0.0001, 'epoch': 9.48}

 95%|█████████▍| 5785/6100 [8:44:25<28:29,  5.43s/it]
 95%|█████████▍| 5786/6100 [8:44:31<28:27,  5.44s/it]
 95%|█████████▍| 5787/6100 [8:44:36<28:22,  5.44s/it]
 95%|█████████▍| 5788/6100 [8:44:42<28:15,  5.43s/it]
 95%|█████████▍| 5789/6100 [8:44:47<28:07,  5.43s/it]
 95%|█████████▍| 5790/6100 [8:44:53<28:01,  5.42s/it]
                                                     
{'loss': 0.0624, 'learning_rate': 0.0001, 'epoch': 9.49}

 95%|█████████▍| 5790/6100 [8:44:53<28:01,  5.42s/it]
 95%|█████████▍| 5791/6100 [8:44:58<27:55,  5.42s/it]
 95%|█████████▍| 5792/6100 [8:45:03<27:50,  5.42s/it]
 95%|█████████▍| 5793/6100 [8:45:09<27:45,  5.43s/it]
 95%|█████████▍| 5794/6100 [8:45:14<27:42,  5.43s/it]
 95%|█████████▌| 5795/6100 [8:45:20<27:37,  5.44s/it]
                                                     
{'loss': 0.06, 'learning_rate': 0.0001, 'epoch': 9.5}

 95%|█████████▌| 5795/6100 [8:45:20<27:37,  5.44s/it]
 95%|█████████▌| 5796/6100 [8:45:25<27:31,  5.43s/it]
 95%|█████████▌| 5797/6100 [8:45:31<27:24,  5.43s/it]
 95%|█████████▌| 5798/6100 [8:45:36<27:16,  5.42s/it]
 95%|█████████▌| 5799/6100 [8:45:41<27:09,  5.41s/it]
 95%|█████████▌| 5800/6100 [8:45:47<27:04,  5.42s/it]
                                                     
{'loss': 0.0612, 'learning_rate': 0.0001, 'epoch': 9.51}

 95%|█████████▌| 5800/6100 [8:45:47<27:04,  5.42s/it]
 95%|█████████▌| 5801/6100 [8:45:52<27:00,  5.42s/it]
 95%|█████████▌| 5802/6100 [8:45:58<26:55,  5.42s/it]
 95%|█████████▌| 5803/6100 [8:46:03<26:49,  5.42s/it]
 95%|█████████▌| 5804/6100 [8:46:09<26:44,  5.42s/it]
 95%|█████████▌| 5805/6100 [8:46:14<26:38,  5.42s/it]
                                                     
{'loss': 0.066, 'learning_rate': 0.0001, 'epoch': 9.52}

 95%|█████████▌| 5805/6100 [8:46:14<26:38,  5.42s/it]
 95%|█████████▌| 5806/6100 [8:46:19<26:32,  5.42s/it]
 95%|█████████▌| 5807/6100 [8:46:25<26:26,  5.41s/it]
 95%|█████████▌| 5808/6100 [8:46:30<26:21,  5.42s/it]
 95%|█████████▌| 5809/6100 [8:46:36<26:17,  5.42s/it]
 95%|█████████▌| 5810/6100 [8:46:41<26:10,  5.42s/it]
                                                     
{'loss': 0.0647, 'learning_rate': 0.0001, 'epoch': 9.52}

 95%|█████████▌| 5810/6100 [8:46:41<26:10,  5.42s/it]
 95%|█████████▌| 5811/6100 [8:46:46<26:05,  5.42s/it]
 95%|█████████▌| 5812/6100 [8:46:52<25:58,  5.41s/it]
 95%|█████████▌| 5813/6100 [8:46:57<25:54,  5.42s/it]
 95%|█████████▌| 5814/6100 [8:47:03<25:50,  5.42s/it]
 95%|█████████▌| 5815/6100 [8:47:08<25:45,  5.42s/it]
                                                     
{'loss': 0.0639, 'learning_rate': 0.0001, 'epoch': 9.53}

 95%|█████████▌| 5815/6100 [8:47:08<25:45,  5.42s/it]
 95%|█████████▌| 5816/6100 [8:47:14<25:39,  5.42s/it]
 95%|█████████▌| 5817/6100 [8:47:19<25:33,  5.42s/it]
 95%|█████████▌| 5818/6100 [8:47:24<25:27,  5.42s/it]
 95%|█████████▌| 5819/6100 [8:47:30<25:22,  5.42s/it]
 95%|█████████▌| 5820/6100 [8:47:35<25:16,  5.42s/it]
                                                     
{'loss': 0.0643, 'learning_rate': 0.0001, 'epoch': 9.54}

 95%|█████████▌| 5820/6100 [8:47:35<25:16,  5.42s/it]
 95%|█████████▌| 5821/6100 [8:47:41<25:11,  5.42s/it]
 95%|█████████▌| 5822/6100 [8:47:46<25:06,  5.42s/it]
 95%|█████████▌| 5823/6100 [8:47:52<25:02,  5.42s/it]
 95%|█████████▌| 5824/6100 [8:47:57<24:56,  5.42s/it]
 95%|█████████▌| 5825/6100 [8:48:02<24:50,  5.42s/it]
                                                     
{'loss': 0.0659, 'learning_rate': 0.0001, 'epoch': 9.55}

 95%|█████████▌| 5825/6100 [8:48:02<24:50,  5.42s/it]
 96%|█████████▌| 5826/6100 [8:48:09<25:49,  5.65s/it]
 96%|█████████▌| 5827/6100 [8:48:14<25:23,  5.58s/it]
 96%|█████████▌| 5828/6100 [8:48:19<25:03,  5.53s/it]
 96%|█████████▌| 5829/6100 [8:48:25<24:48,  5.49s/it]
 96%|█████████▌| 5830/6100 [8:48:30<24:36,  5.47s/it]
                                                     
{'loss': 0.0586, 'learning_rate': 0.0001, 'epoch': 9.56}

 96%|█████████▌| 5830/6100 [8:48:30<24:36,  5.47s/it]
 96%|█████████▌| 5831/6100 [8:48:36<24:26,  5.45s/it]
 96%|█████████▌| 5832/6100 [8:48:41<24:17,  5.44s/it]
 96%|█████████▌| 5833/6100 [8:48:46<24:10,  5.43s/it]
 96%|█████████▌| 5834/6100 [8:48:52<24:02,  5.42s/it]
 96%|█████████▌| 5835/6100 [8:48:57<23:57,  5.42s/it]
                                                     
{'loss': 0.0645, 'learning_rate': 0.0001, 'epoch': 9.57}

 96%|█████████▌| 5835/6100 [8:48:57<23:57,  5.42s/it]
 96%|█████████▌| 5836/6100 [8:49:03<23:51,  5.42s/it]
 96%|█████████▌| 5837/6100 [8:49:08<23:48,  5.43s/it]
 96%|█████████▌| 5838/6100 [8:49:14<23:42,  5.43s/it]
 96%|█████████▌| 5839/6100 [8:49:19<23:35,  5.42s/it]
 96%|█████████▌| 5840/6100 [8:49:24<23:29,  5.42s/it]
                                                     
{'loss': 0.0649, 'learning_rate': 0.0001, 'epoch': 9.57}

 96%|█████████▌| 5840/6100 [8:49:24<23:29,  5.42s/it]
 96%|█████████▌| 5841/6100 [8:49:30<23:24,  5.42s/it]
 96%|█████████▌| 5842/6100 [8:49:35<23:18,  5.42s/it]
 96%|█████████▌| 5843/6100 [8:49:41<23:13,  5.42s/it]
 96%|█████████▌| 5844/6100 [8:49:46<23:08,  5.43s/it]
 96%|█████████▌| 5845/6100 [8:49:52<23:09,  5.45s/it]
                                                     
{'loss': 0.065, 'learning_rate': 0.0001, 'epoch': 9.58}

 96%|█████████▌| 5845/6100 [8:49:52<23:09,  5.45s/it]
 96%|█████████▌| 5846/6100 [8:49:57<23:02,  5.44s/it]
 96%|█████████▌| 5847/6100 [8:50:02<22:54,  5.43s/it]
 96%|█████████▌| 5848/6100 [8:50:08<22:48,  5.43s/it]
 96%|█████████▌| 5849/6100 [8:50:13<22:41,  5.43s/it]
 96%|█████████▌| 5850/6100 [8:50:19<22:36,  5.43s/it]
                                                     
{'loss': 0.0628, 'learning_rate': 0.0001, 'epoch': 9.59}

 96%|█████████▌| 5850/6100 [8:50:19<22:36,  5.43s/it]Saving model checkpoint to ./results/checkpoint-5850
Configuration saved in ./results/checkpoint-5850/config.json
Model weights saved in ./results/checkpoint-5850/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-5850/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-5850/special_tokens_map.json
[2023-02-22 04:42:34,859] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step5854 is begin to save!
[2023-02-22 04:42:34,862] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-5850/global_step5854/mp_rank_00_model_states.pt
[2023-02-22 04:42:34,862] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-5850/global_step5854/mp_rank_00_model_states.pt...
[2023-02-22 04:42:35,675] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-5850/global_step5854/mp_rank_00_model_states.pt.
[2023-02-22 04:42:35,677] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-5850/global_step5854/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 04:42:35,851] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-5850/global_step5854/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 04:42:35,851] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-5850/global_step5854/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 04:42:35,851] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5854 is ready now!
Deleting older checkpoint [results/checkpoint-5700] due to args.save_total_limit

 96%|█████████▌| 5851/6100 [8:50:26<25:11,  6.07s/it]
 96%|█████████▌| 5852/6100 [8:50:32<24:16,  5.87s/it]
 96%|█████████▌| 5853/6100 [8:50:37<23:36,  5.73s/it]
 96%|█████████▌| 5854/6100 [8:50:42<23:05,  5.63s/it]
 96%|█████████▌| 5855/6100 [8:50:48<22:43,  5.57s/it]
                                                     
{'loss': 0.0637, 'learning_rate': 0.0001, 'epoch': 9.6}

 96%|█████████▌| 5855/6100 [8:50:48<22:43,  5.57s/it]
 96%|█████████▌| 5856/6100 [8:50:53<22:27,  5.52s/it]
 96%|█████████▌| 5857/6100 [8:50:59<22:13,  5.49s/it]
 96%|█████████▌| 5858/6100 [8:51:04<22:02,  5.46s/it]
 96%|█████████▌| 5859/6100 [8:51:10<21:52,  5.45s/it]
 96%|█████████▌| 5860/6100 [8:51:15<21:44,  5.44s/it]
                                                     
{'loss': 0.0638, 'learning_rate': 0.0001, 'epoch': 9.61}

 96%|█████████▌| 5860/6100 [8:51:15<21:44,  5.44s/it]
 96%|█████████▌| 5861/6100 [8:51:20<21:37,  5.43s/it]
 96%|█████████▌| 5862/6100 [8:51:26<21:30,  5.42s/it]
 96%|█████████▌| 5863/6100 [8:51:31<21:25,  5.42s/it]
 96%|█████████▌| 5864/6100 [8:51:37<21:19,  5.42s/it]
 96%|█████████▌| 5865/6100 [8:51:42<21:13,  5.42s/it]
                                                     
{'loss': 0.0641, 'learning_rate': 0.0001, 'epoch': 9.61}

 96%|█████████▌| 5865/6100 [8:51:42<21:13,  5.42s/it]
 96%|█████████▌| 5866/6100 [8:51:47<21:08,  5.42s/it]
 96%|█████████▌| 5867/6100 [8:51:53<21:02,  5.42s/it]
 96%|█████████▌| 5868/6100 [8:51:58<20:56,  5.42s/it]
 96%|█████████▌| 5869/6100 [8:52:04<20:50,  5.41s/it]
 96%|█████████▌| 5870/6100 [8:52:09<20:46,  5.42s/it]
                                                     
{'loss': 0.0631, 'learning_rate': 0.0001, 'epoch': 9.62}

 96%|█████████▌| 5870/6100 [8:52:09<20:46,  5.42s/it]
 96%|█████████▌| 5871/6100 [8:52:15<20:41,  5.42s/it]
 96%|█████████▋| 5872/6100 [8:52:20<20:35,  5.42s/it]
 96%|█████████▋| 5873/6100 [8:52:25<20:28,  5.41s/it]
 96%|█████████▋| 5874/6100 [8:52:31<20:23,  5.41s/it]
 96%|█████████▋| 5875/6100 [8:52:36<20:18,  5.42s/it]
                                                     
{'loss': 0.0669, 'learning_rate': 0.0001, 'epoch': 9.63}

 96%|█████████▋| 5875/6100 [8:52:36<20:18,  5.42s/it]
 96%|█████████▋| 5876/6100 [8:52:42<20:13,  5.42s/it]
 96%|█████████▋| 5877/6100 [8:52:47<20:08,  5.42s/it]
 96%|█████████▋| 5878/6100 [8:52:52<20:02,  5.42s/it]
 96%|█████████▋| 5879/6100 [8:52:58<20:17,  5.51s/it]
 96%|█████████▋| 5880/6100 [8:53:04<20:04,  5.48s/it]
                                                     
{'loss': 0.0687, 'learning_rate': 0.0001, 'epoch': 9.64}

 96%|█████████▋| 5880/6100 [8:53:04<20:04,  5.48s/it]
 96%|█████████▋| 5881/6100 [8:53:09<19:54,  5.46s/it]
 96%|█████████▋| 5882/6100 [8:53:14<19:46,  5.44s/it]
 96%|█████████▋| 5883/6100 [8:53:20<19:38,  5.43s/it]
 96%|█████████▋| 5884/6100 [8:53:25<19:32,  5.43s/it]
 96%|█████████▋| 5885/6100 [8:53:31<19:25,  5.42s/it]
                                                     
{'loss': 0.066, 'learning_rate': 0.0001, 'epoch': 9.65}

 96%|█████████▋| 5885/6100 [8:53:31<19:25,  5.42s/it]
 96%|█████████▋| 5886/6100 [8:53:36<19:18,  5.42s/it]
 97%|█████████▋| 5887/6100 [8:53:41<19:13,  5.42s/it]
 97%|█████████▋| 5888/6100 [8:53:47<19:08,  5.42s/it]
 97%|█████████▋| 5889/6100 [8:53:52<19:02,  5.42s/it]
 97%|█████████▋| 5890/6100 [8:53:58<18:57,  5.42s/it]
                                                     
{'loss': 0.062, 'learning_rate': 0.0001, 'epoch': 9.66}

 97%|█████████▋| 5890/6100 [8:53:58<18:57,  5.42s/it]
 97%|█████████▋| 5891/6100 [8:54:03<18:53,  5.42s/it]
 97%|█████████▋| 5892/6100 [8:54:09<18:47,  5.42s/it]
 97%|█████████▋| 5893/6100 [8:54:14<18:42,  5.42s/it]
 97%|█████████▋| 5894/6100 [8:54:19<18:35,  5.41s/it]
 97%|█████████▋| 5895/6100 [8:54:25<18:30,  5.42s/it]
                                                     
{'loss': 0.064, 'learning_rate': 0.0001, 'epoch': 9.66}

 97%|█████████▋| 5895/6100 [8:54:25<18:30,  5.42s/it]
 97%|█████████▋| 5896/6100 [8:54:30<18:24,  5.42s/it]
 97%|█████████▋| 5897/6100 [8:54:36<18:20,  5.42s/it]
 97%|█████████▋| 5898/6100 [8:54:41<18:14,  5.42s/it]
 97%|█████████▋| 5899/6100 [8:54:46<18:09,  5.42s/it]
 97%|█████████▋| 5900/6100 [8:54:52<18:03,  5.42s/it]
                                                     
{'loss': 0.0653, 'learning_rate': 0.0001, 'epoch': 9.67}

 97%|█████████▋| 5900/6100 [8:54:52<18:03,  5.42s/it]
 97%|█████████▋| 5901/6100 [8:54:57<17:58,  5.42s/it]
 97%|█████████▋| 5902/6100 [8:55:03<17:52,  5.41s/it]
 97%|█████████▋| 5903/6100 [8:55:08<17:47,  5.42s/it]
 97%|█████████▋| 5904/6100 [8:55:14<17:41,  5.42s/it]
 97%|█████████▋| 5905/6100 [8:55:19<17:37,  5.42s/it]
                                                     
{'loss': 0.061, 'learning_rate': 0.0001, 'epoch': 9.68}

 97%|█████████▋| 5905/6100 [8:55:19<17:37,  5.42s/it]
 97%|█████████▋| 5906/6100 [8:55:24<17:31,  5.42s/it]
 97%|█████████▋| 5907/6100 [8:55:30<17:26,  5.42s/it]
 97%|█████████▋| 5908/6100 [8:55:35<17:20,  5.42s/it]
 97%|█████████▋| 5909/6100 [8:55:41<17:14,  5.42s/it]
 97%|█████████▋| 5910/6100 [8:55:46<17:09,  5.42s/it]
                                                     
{'loss': 0.0637, 'learning_rate': 0.0001, 'epoch': 9.69}

 97%|█████████▋| 5910/6100 [8:55:46<17:09,  5.42s/it]
 97%|█████████▋| 5911/6100 [8:55:51<17:04,  5.42s/it]
 97%|█████████▋| 5912/6100 [8:55:57<16:57,  5.41s/it]
 97%|█████████▋| 5913/6100 [8:56:02<16:52,  5.41s/it]
 97%|█████████▋| 5914/6100 [8:56:08<16:46,  5.41s/it]
 97%|█████████▋| 5915/6100 [8:56:13<16:42,  5.42s/it]
                                                     
{'loss': 0.0666, 'learning_rate': 0.0001, 'epoch': 9.7}

 97%|█████████▋| 5915/6100 [8:56:13<16:42,  5.42s/it]
 97%|█████████▋| 5916/6100 [8:56:19<16:37,  5.42s/it]
 97%|█████████▋| 5917/6100 [8:56:24<16:32,  5.43s/it]
 97%|█████████▋| 5918/6100 [8:56:29<16:28,  5.43s/it]
 97%|█████████▋| 5919/6100 [8:56:35<16:24,  5.44s/it]
 97%|█████████▋| 5920/6100 [8:56:40<16:18,  5.43s/it]
                                                     
{'loss': 0.069, 'learning_rate': 0.0001, 'epoch': 9.7}

 97%|█████████▋| 5920/6100 [8:56:40<16:18,  5.43s/it]
 97%|█████████▋| 5921/6100 [8:56:46<16:12,  5.43s/it]
 97%|█████████▋| 5922/6100 [8:56:51<16:06,  5.43s/it]
 97%|█████████▋| 5923/6100 [8:56:57<16:00,  5.43s/it]
 97%|█████████▋| 5924/6100 [8:57:02<15:55,  5.43s/it]
 97%|█████████▋| 5925/6100 [8:57:07<15:50,  5.43s/it]
                                                     
{'loss': 0.0682, 'learning_rate': 0.0001, 'epoch': 9.71}

 97%|█████████▋| 5925/6100 [8:57:07<15:50,  5.43s/it]
 97%|█████████▋| 5926/6100 [8:57:13<15:44,  5.43s/it]
 97%|█████████▋| 5927/6100 [8:57:18<15:40,  5.44s/it]
 97%|█████████▋| 5928/6100 [8:57:24<15:35,  5.44s/it]
 97%|█████████▋| 5929/6100 [8:57:29<15:28,  5.43s/it]
 97%|█████████▋| 5930/6100 [8:57:35<15:23,  5.43s/it]
                                                     
{'loss': 0.0679, 'learning_rate': 0.0001, 'epoch': 9.72}

 97%|█████████▋| 5930/6100 [8:57:35<15:23,  5.43s/it]
 97%|█████████▋| 5931/6100 [8:57:40<15:17,  5.43s/it]
 97%|█████████▋| 5932/6100 [8:57:45<15:11,  5.43s/it]
 97%|█████████▋| 5933/6100 [8:57:51<15:05,  5.42s/it]
 97%|█████████▋| 5934/6100 [8:57:56<14:59,  5.42s/it]
 97%|█████████▋| 5935/6100 [8:58:02<14:54,  5.42s/it]
                                                     
{'loss': 0.0648, 'learning_rate': 0.0001, 'epoch': 9.73}

 97%|█████████▋| 5935/6100 [8:58:02<14:54,  5.42s/it]
 97%|█████████▋| 5936/6100 [8:58:07<14:48,  5.42s/it]
 97%|█████████▋| 5937/6100 [8:58:13<14:43,  5.42s/it]
 97%|█████████▋| 5938/6100 [8:58:18<14:37,  5.42s/it]
 97%|█████████▋| 5939/6100 [8:58:23<14:32,  5.42s/it]
 97%|█████████▋| 5940/6100 [8:58:29<14:26,  5.42s/it]
                                                     
{'loss': 0.0646, 'learning_rate': 0.0001, 'epoch': 9.74}

 97%|█████████▋| 5940/6100 [8:58:29<14:26,  5.42s/it]
 97%|█████████▋| 5941/6100 [8:58:34<14:22,  5.42s/it]
 97%|█████████▋| 5942/6100 [8:58:40<14:17,  5.42s/it]
 97%|█████████▋| 5943/6100 [8:58:45<14:11,  5.42s/it]
 97%|█████████▋| 5944/6100 [8:58:50<14:05,  5.42s/it]
 97%|█████████▋| 5945/6100 [8:58:56<13:59,  5.41s/it]
                                                     
{'loss': 0.0643, 'learning_rate': 0.0001, 'epoch': 9.75}

 97%|█████████▋| 5945/6100 [8:58:56<13:59,  5.41s/it]
 97%|█████████▋| 5946/6100 [8:59:02<14:18,  5.57s/it]
 97%|█████████▋| 5947/6100 [8:59:07<14:05,  5.53s/it]
 98%|█████████▊| 5948/6100 [8:59:13<13:55,  5.50s/it]
 98%|█████████▊| 5949/6100 [8:59:18<13:46,  5.47s/it]
 98%|█████████▊| 5950/6100 [8:59:24<13:39,  5.46s/it]
                                                     
{'loss': 0.0707, 'learning_rate': 0.0001, 'epoch': 9.75}

 98%|█████████▊| 5950/6100 [8:59:24<13:39,  5.46s/it]
 98%|█████████▊| 5951/6100 [8:59:29<13:31,  5.45s/it]
 98%|█████████▊| 5952/6100 [8:59:34<13:25,  5.44s/it]
 98%|█████████▊| 5953/6100 [8:59:40<13:19,  5.44s/it]
 98%|█████████▊| 5954/6100 [8:59:45<13:13,  5.43s/it]
 98%|█████████▊| 5955/6100 [8:59:51<13:07,  5.43s/it]
                                                     
{'loss': 0.0626, 'learning_rate': 0.0001, 'epoch': 9.76}

 98%|█████████▊| 5955/6100 [8:59:51<13:07,  5.43s/it]
 98%|█████████▊| 5956/6100 [8:59:56<13:01,  5.43s/it]
 98%|█████████▊| 5957/6100 [9:00:02<12:56,  5.43s/it]
 98%|█████████▊| 5958/6100 [9:00:07<12:50,  5.43s/it]
 98%|█████████▊| 5959/6100 [9:00:12<12:45,  5.43s/it]
 98%|█████████▊| 5960/6100 [9:00:18<12:40,  5.43s/it]
                                                     
{'loss': 0.0675, 'learning_rate': 0.0001, 'epoch': 9.77}

 98%|█████████▊| 5960/6100 [9:00:18<12:40,  5.43s/it]
 98%|█████████▊| 5961/6100 [9:00:23<12:35,  5.43s/it]
 98%|█████████▊| 5962/6100 [9:00:29<12:29,  5.43s/it]
 98%|█████████▊| 5963/6100 [9:00:34<12:23,  5.43s/it]
 98%|█████████▊| 5964/6100 [9:00:40<12:18,  5.43s/it]
 98%|█████████▊| 5965/6100 [9:00:45<12:12,  5.42s/it]
                                                     
{'loss': 0.0642, 'learning_rate': 0.0001, 'epoch': 9.78}

 98%|█████████▊| 5965/6100 [9:00:45<12:12,  5.42s/it]
 98%|█████████▊| 5966/6100 [9:00:50<12:06,  5.42s/it]
 98%|█████████▊| 5967/6100 [9:00:56<12:01,  5.42s/it]
 98%|█████████▊| 5968/6100 [9:01:01<11:56,  5.43s/it]
 98%|█████████▊| 5969/6100 [9:01:07<11:51,  5.43s/it]
 98%|█████████▊| 5970/6100 [9:01:12<11:45,  5.43s/it]
                                                     
{'loss': 0.0659, 'learning_rate': 0.0001, 'epoch': 9.79}

 98%|█████████▊| 5970/6100 [9:01:12<11:45,  5.43s/it]
 98%|█████████▊| 5971/6100 [9:01:17<11:39,  5.43s/it]
 98%|█████████▊| 5972/6100 [9:01:23<11:34,  5.43s/it]
 98%|█████████▊| 5973/6100 [9:01:28<11:28,  5.43s/it]
 98%|█████████▊| 5974/6100 [9:01:34<11:23,  5.42s/it]
 98%|█████████▊| 5975/6100 [9:01:39<11:17,  5.42s/it]
                                                     
{'loss': 0.0632, 'learning_rate': 0.0001, 'epoch': 9.79}

 98%|█████████▊| 5975/6100 [9:01:39<11:17,  5.42s/it]
 98%|█████████▊| 5976/6100 [9:01:45<11:12,  5.43s/it]
 98%|█████████▊| 5977/6100 [9:01:50<11:07,  5.43s/it]
 98%|█████████▊| 5978/6100 [9:01:55<11:02,  5.43s/it]
 98%|█████████▊| 5979/6100 [9:02:01<10:56,  5.43s/it]
 98%|█████████▊| 5980/6100 [9:02:06<10:50,  5.42s/it]
                                                     
{'loss': 0.0647, 'learning_rate': 0.0001, 'epoch': 9.8}

 98%|█████████▊| 5980/6100 [9:02:06<10:50,  5.42s/it]
 98%|█████████▊| 5981/6100 [9:02:12<10:45,  5.42s/it]
 98%|█████████▊| 5982/6100 [9:02:17<10:40,  5.43s/it]
 98%|█████████▊| 5983/6100 [9:02:23<10:34,  5.42s/it]
 98%|█████████▊| 5984/6100 [9:02:28<10:29,  5.43s/it]
 98%|█████████▊| 5985/6100 [9:02:33<10:24,  5.43s/it]
                                                     
{'loss': 0.0623, 'learning_rate': 0.0001, 'epoch': 9.81}

 98%|█████████▊| 5985/6100 [9:02:33<10:24,  5.43s/it]
 98%|█████████▊| 5986/6100 [9:02:39<10:18,  5.42s/it]
 98%|█████████▊| 5987/6100 [9:02:44<10:12,  5.42s/it]
 98%|█████████▊| 5988/6100 [9:02:50<10:07,  5.42s/it]
 98%|█████████▊| 5989/6100 [9:02:55<10:02,  5.43s/it]
 98%|█████████▊| 5990/6100 [9:03:01<09:57,  5.43s/it]
                                                     
{'loss': 0.0645, 'learning_rate': 0.0001, 'epoch': 9.82}

 98%|█████████▊| 5990/6100 [9:03:01<09:57,  5.43s/it]
 98%|█████████▊| 5991/6100 [9:03:06<09:51,  5.43s/it]
 98%|█████████▊| 5992/6100 [9:03:11<09:46,  5.43s/it]
 98%|█████████▊| 5993/6100 [9:03:17<09:40,  5.43s/it]
 98%|█████████▊| 5994/6100 [9:03:22<09:34,  5.42s/it]
 98%|█████████▊| 5995/6100 [9:03:28<09:29,  5.42s/it]
                                                     
{'loss': 0.0688, 'learning_rate': 0.0001, 'epoch': 9.83}

 98%|█████████▊| 5995/6100 [9:03:28<09:29,  5.42s/it][2023-02-22 04:55:45,684] [INFO] [logging.py:75:log_dist] [Rank 0] step=6000, skipped=0, lr=[0.0001], mom=[[0.9, 0.999]]
[2023-02-22 04:55:45,787] [INFO] [timer.py:198:stop] epoch=0/micro_step=24000/global_step=6000, RunningAvgSamplesPerSec=35.870118145141674, CurrSamplesPerSec=35.85297999216521, MemAllocated=0.66GB, MaxMemAllocated=36.72GB

 98%|█████████▊| 5996/6100 [9:03:33<09:24,  5.43s/it]
 98%|█████████▊| 5997/6100 [9:03:39<09:18,  5.42s/it]
 98%|█████████▊| 5998/6100 [9:03:44<09:12,  5.42s/it]
 98%|█████████▊| 5999/6100 [9:03:49<09:08,  5.43s/it]
 98%|█████████▊| 6000/6100 [9:03:55<09:07,  5.47s/it]
                                                     
{'loss': 0.065, 'learning_rate': 0.0001, 'epoch': 9.84}

 98%|█████████▊| 6000/6100 [9:03:55<09:07,  5.47s/it]Saving model checkpoint to ./results/checkpoint-6000
Configuration saved in ./results/checkpoint-6000/config.json
Model weights saved in ./results/checkpoint-6000/pytorch_model.bin
tokenizer config file saved in ./results/checkpoint-6000/tokenizer_config.json
Special tokens file saved in ./results/checkpoint-6000/special_tokens_map.json
[2023-02-22 04:56:11,153] [INFO] [logging.py:75:log_dist] [Rank 0] [Torch] Checkpoint global_step6004 is begin to save!
[2023-02-22 04:56:11,156] [INFO] [logging.py:75:log_dist] [Rank 0] Saving model checkpoint: ./results/checkpoint-6000/global_step6004/mp_rank_00_model_states.pt
[2023-02-22 04:56:11,156] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-6000/global_step6004/mp_rank_00_model_states.pt...
[2023-02-22 04:56:11,979] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-6000/global_step6004/mp_rank_00_model_states.pt.
[2023-02-22 04:56:11,981] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./results/checkpoint-6000/global_step6004/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-02-22 04:56:12,154] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./results/checkpoint-6000/global_step6004/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-02-22 04:56:12,154] [INFO] [engine.py:3407:_save_zero_checkpoint] zero checkpoint saved ./results/checkpoint-6000/global_step6004/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2023-02-22 04:56:12,154] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6004 is ready now!
Deleting older checkpoint [results/checkpoint-5850] due to args.save_total_limit

 98%|█████████▊| 6001/6100 [9:04:03<10:04,  6.10s/it]
 98%|█████████▊| 6002/6100 [9:04:08<09:37,  5.90s/it]
 98%|█████████▊| 6003/6100 [9:04:13<09:17,  5.75s/it]
 98%|█████████▊| 6004/6100 [9:04:19<09:02,  5.65s/it]
 98%|█████████▊| 6005/6100 [9:04:24<08:50,  5.58s/it]
                                                     
{'loss': 0.0658, 'learning_rate': 0.0001, 'epoch': 9.84}

 98%|█████████▊| 6005/6100 [9:04:24<08:50,  5.58s/it]
 98%|█████████▊| 6006/6100 [9:04:30<08:40,  5.54s/it]
 98%|█████████▊| 6007/6100 [9:04:35<08:31,  5.50s/it]
 98%|█████████▊| 6008/6100 [9:04:40<08:23,  5.48s/it]
 99%|█████████▊| 6009/6100 [9:04:46<08:17,  5.46s/it]
 99%|█████████▊| 6010/6100 [9:04:51<08:11,  5.46s/it]
                                                     
{'loss': 0.0658, 'learning_rate': 0.0001, 'epoch': 9.85}

 99%|█████████▊| 6010/6100 [9:04:51<08:11,  5.46s/it]
 99%|█████████▊| 6011/6100 [9:04:57<08:04,  5.45s/it]
 99%|█████████▊| 6012/6100 [9:05:02<07:58,  5.44s/it]
 99%|█████████▊| 6013/6100 [9:05:08<07:52,  5.43s/it]
 99%|█████████▊| 6014/6100 [9:05:13<07:47,  5.43s/it]
 99%|█████████▊| 6015/6100 [9:05:18<07:41,  5.43s/it]
                                                     
{'loss': 0.0613, 'learning_rate': 0.0001, 'epoch': 9.86}

 99%|█████████▊| 6015/6100 [9:05:18<07:41,  5.43s/it]
 99%|█████████▊| 6016/6100 [9:05:24<07:36,  5.43s/it]
 99%|█████████▊| 6017/6100 [9:05:29<07:30,  5.42s/it]
 99%|█████████▊| 6018/6100 [9:05:35<07:25,  5.43s/it]
 99%|█████████▊| 6019/6100 [9:05:40<07:19,  5.42s/it]
 99%|█████████▊| 6020/6100 [9:05:46<07:13,  5.42s/it]
                                                     
{'loss': 0.065, 'learning_rate': 0.0001, 'epoch': 9.87}

 99%|█████████▊| 6020/6100 [9:05:46<07:13,  5.42s/it]
 99%|█████████▊| 6021/6100 [9:05:51<07:08,  5.43s/it]
 99%|█████████▊| 6022/6100 [9:05:56<07:03,  5.43s/it]
 99%|█████████▊| 6023/6100 [9:06:02<06:58,  5.43s/it]
 99%|█████████▉| 6024/6100 [9:06:07<06:53,  5.43s/it]
 99%|█████████▉| 6025/6100 [9:06:13<06:47,  5.44s/it]
                                                     
{'loss': 0.0651, 'learning_rate': 0.0001, 'epoch': 9.88}

 99%|█████████▉| 6025/6100 [9:06:13<06:47,  5.44s/it]
 99%|█████████▉| 6026/6100 [9:06:18<06:42,  5.44s/it]
 99%|█████████▉| 6027/6100 [9:06:24<06:36,  5.43s/it]
 99%|█████████▉| 6028/6100 [9:06:29<06:30,  5.43s/it]
 99%|█████████▉| 6029/6100 [9:06:35<06:25,  5.43s/it]
 99%|█████████▉| 6030/6100 [9:06:40<06:20,  5.43s/it]
                                                     
{'loss': 0.0646, 'learning_rate': 0.0001, 'epoch': 9.88}

 99%|█████████▉| 6030/6100 [9:06:40<06:20,  5.43s/it]
 99%|█████████▉| 6031/6100 [9:06:45<06:15,  5.44s/it]
 99%|█████████▉| 6032/6100 [9:06:51<06:09,  5.44s/it]
 99%|█████████▉| 6033/6100 [9:06:56<06:04,  5.44s/it]
 99%|█████████▉| 6034/6100 [9:07:02<05:58,  5.44s/it]
 99%|█████████▉| 6035/6100 [9:07:07<05:53,  5.43s/it]
                                                     
{'loss': 0.0631, 'learning_rate': 0.0001, 'epoch': 9.89}

 99%|█████████▉| 6035/6100 [9:07:07<05:53,  5.43s/it]
 99%|█████████▉| 6036/6100 [9:07:13<05:47,  5.43s/it]
 99%|█████████▉| 6037/6100 [9:07:18<05:42,  5.43s/it]
 99%|█████████▉| 6038/6100 [9:07:23<05:36,  5.43s/it]
 99%|█████████▉| 6039/6100 [9:07:29<05:31,  5.43s/it]
 99%|█████████▉| 6040/6100 [9:07:34<05:25,  5.43s/it]
                                                     
{'loss': 0.0662, 'learning_rate': 0.0001, 'epoch': 9.9}

 99%|█████████▉| 6040/6100 [9:07:34<05:25,  5.43s/it]
 99%|█████████▉| 6041/6100 [9:07:40<05:20,  5.43s/it]
 99%|█████████▉| 6042/6100 [9:07:45<05:14,  5.42s/it]
 99%|█████████▉| 6043/6100 [9:07:51<05:09,  5.43s/it]
 99%|█████████▉| 6044/6100 [9:07:56<05:03,  5.43s/it]
 99%|█████████▉| 6045/6100 [9:08:01<04:58,  5.43s/it]
                                                     
{'loss': 0.0643, 'learning_rate': 0.0001, 'epoch': 9.91}

 99%|█████████▉| 6045/6100 [9:08:01<04:58,  5.43s/it]
 99%|█████████▉| 6046/6100 [9:08:07<04:53,  5.43s/it]
 99%|█████████▉| 6047/6100 [9:08:12<04:47,  5.43s/it]
 99%|█████████▉| 6048/6100 [9:08:18<04:42,  5.43s/it]
 99%|█████████▉| 6049/6100 [9:08:23<04:36,  5.42s/it]
 99%|█████████▉| 6050/6100 [9:08:29<04:31,  5.42s/it]
                                                     
{'loss': 0.0653, 'learning_rate': 0.0001, 'epoch': 9.92}

 99%|█████████▉| 6050/6100 [9:08:29<04:31,  5.42s/it]
 99%|█████████▉| 6051/6100 [9:08:34<04:25,  5.42s/it]
 99%|█████████▉| 6052/6100 [9:08:39<04:20,  5.43s/it]
 99%|█████████▉| 6053/6100 [9:08:45<04:15,  5.43s/it]
 99%|█████████▉| 6054/6100 [9:08:50<04:09,  5.43s/it]
 99%|█████████▉| 6055/6100 [9:08:56<04:04,  5.43s/it]
                                                     
{'loss': 0.0627, 'learning_rate': 0.0001, 'epoch': 9.93}

 99%|█████████▉| 6055/6100 [9:08:56<04:04,  5.43s/it]
 99%|█████████▉| 6056/6100 [9:09:01<03:58,  5.43s/it]
 99%|█████████▉| 6057/6100 [9:09:07<03:53,  5.43s/it]
 99%|█████████▉| 6058/6100 [9:09:12<03:47,  5.43s/it]
 99%|█████████▉| 6059/6100 [9:09:17<03:42,  5.43s/it]
 99%|█████████▉| 6060/6100 [9:09:23<03:37,  5.44s/it]
                                                     
{'loss': 0.0616, 'learning_rate': 0.0001, 'epoch': 9.93}

 99%|█████████▉| 6060/6100 [9:09:23<03:37,  5.44s/it]
 99%|█████████▉| 6061/6100 [9:09:28<03:32,  5.44s/it]
 99%|█████████▉| 6062/6100 [9:09:34<03:26,  5.43s/it]
 99%|█████████▉| 6063/6100 [9:09:39<03:20,  5.43s/it]
 99%|█████████▉| 6064/6100 [9:09:45<03:15,  5.43s/it]
 99%|█████████▉| 6065/6100 [9:09:50<03:10,  5.43s/it]
                                                     
{'loss': 0.0651, 'learning_rate': 0.0001, 'epoch': 9.94}

 99%|█████████▉| 6065/6100 [9:09:50<03:10,  5.43s/it]
 99%|█████████▉| 6066/6100 [9:09:56<03:05,  5.47s/it]
 99%|█████████▉| 6067/6100 [9:10:01<02:59,  5.45s/it]
 99%|█████████▉| 6068/6100 [9:10:06<02:54,  5.45s/it]
 99%|█████████▉| 6069/6100 [9:10:12<02:48,  5.45s/it]
100%|█████████▉| 6070/6100 [9:10:17<02:43,  5.43s/it]
                                                     
{'loss': 0.0681, 'learning_rate': 0.0001, 'epoch': 9.95}

100%|█████████▉| 6070/6100 [9:10:17<02:43,  5.43s/it]
100%|█████████▉| 6071/6100 [9:10:23<02:37,  5.43s/it]
100%|█████████▉| 6072/6100 [9:10:28<02:31,  5.43s/it]
100%|█████████▉| 6073/6100 [9:10:34<02:26,  5.42s/it]
100%|█████████▉| 6074/6100 [9:10:39<02:21,  5.42s/it]
100%|█████████▉| 6075/6100 [9:10:44<02:15,  5.43s/it]
                                                     
{'loss': 0.0657, 'learning_rate': 0.0001, 'epoch': 9.96}

100%|█████████▉| 6075/6100 [9:10:44<02:15,  5.43s/it]
100%|█████████▉| 6076/6100 [9:10:50<02:10,  5.43s/it]
100%|█████████▉| 6077/6100 [9:10:55<02:04,  5.43s/it]
100%|█████████▉| 6078/6100 [9:11:01<01:59,  5.43s/it]
100%|█████████▉| 6079/6100 [9:11:06<01:53,  5.42s/it]
100%|█████████▉| 6080/6100 [9:11:12<01:48,  5.42s/it]
                                                     
{'loss': 0.0653, 'learning_rate': 0.0001, 'epoch': 9.97}

100%|█████████▉| 6080/6100 [9:11:12<01:48,  5.42s/it]
100%|█████████▉| 6081/6100 [9:11:17<01:43,  5.42s/it]
100%|█████████▉| 6082/6100 [9:11:22<01:37,  5.42s/it]
100%|█████████▉| 6083/6100 [9:11:28<01:33,  5.48s/it]
100%|█████████▉| 6084/6100 [9:11:33<01:27,  5.47s/it]
100%|█████████▉| 6085/6100 [9:11:39<01:21,  5.46s/it]
                                                     
{'loss': 0.0635, 'learning_rate': 0.0001, 'epoch': 9.97}

100%|█████████▉| 6085/6100 [9:11:39<01:21,  5.46s/it]
100%|█████████▉| 6086/6100 [9:11:44<01:16,  5.44s/it]
100%|█████████▉| 6087/6100 [9:11:50<01:10,  5.44s/it]
100%|█████████▉| 6088/6100 [9:11:55<01:05,  5.44s/it]
100%|█████████▉| 6089/6100 [9:12:01<00:59,  5.44s/it]
100%|█████████▉| 6090/6100 [9:12:06<00:54,  5.43s/it]
                                                     
{'loss': 0.0648, 'learning_rate': 0.0001, 'epoch': 9.98}

100%|█████████▉| 6090/6100 [9:12:06<00:54,  5.43s/it]
100%|█████████▉| 6091/6100 [9:12:11<00:48,  5.43s/it]
100%|█████████▉| 6092/6100 [9:12:17<00:43,  5.43s/it]
100%|█████████▉| 6093/6100 [9:12:22<00:37,  5.43s/it]
100%|█████████▉| 6094/6100 [9:12:28<00:32,  5.42s/it]
100%|█████████▉| 6095/6100 [9:12:33<00:27,  5.42s/it]
                                                     
{'loss': 0.0669, 'learning_rate': 0.0001, 'epoch': 9.99}

100%|█████████▉| 6095/6100 [9:12:33<00:27,  5.42s/it]
100%|█████████▉| 6096/6100 [9:12:39<00:21,  5.42s/it]
100%|█████████▉| 6097/6100 [9:12:44<00:16,  5.43s/it]
100%|█████████▉| 6098/6100 [9:12:49<00:10,  5.43s/it]
100%|█████████▉| 6099/6100 [9:12:55<00:05,  5.43s/it]
100%|██████████| 6100/6100 [9:13:00<00:00,  5.43s/it]
                                                     
{'loss': 0.0651, 'learning_rate': 0.0001, 'epoch': 10.0}

100%|██████████| 6100/6100 [9:13:00<00:00,  5.43s/it]

Training completed. Do not forget to share your model on huggingface.co/models =)


Time: 33180.99Time: 33181.01Time: 33181.06
Time: 33180.98Time: 33181.18

Samples/second: 35.33Samples/second: 35.33
Time: 33181.00

Samples/second: 35.33
Samples/second: 35.33
Samples/second: 35.33

Samples/second: 35.33

Time: 33180.96
Samples/second: 35.33
GPU memory occupied: 45195 MB.

                                                     
{'train_runtime': 33180.7459, 'train_samples_per_second': 35.331, 'train_steps_per_second': 0.184, 'train_loss': 0.1120870843480845, 'epoch': 10.0}GPU memory occupied: 45195 MB.

GPU memory occupied: 45195 MB.
GPU memory occupied: 45195 MB.
GPU memory occupied: 45195 MB.
GPU memory occupied: 45195 MB.
GPU memory occupied: 45195 MB.

100%|██████████| 6100/6100 [9:13:00<00:00,  5.43s/it]
100%|██████████| 6100/6100 [9:13:00<00:00,  5.44s/it]
Time: 33180.75
Samples/second: 35.33
GPU memory occupied: 45195 MB.
Current Path:  /home/code-generation-gpt-models
Configuration saved in experiments/2023-02-22-5f8754af240241f4c78ba924918b0bfb26f26a17b0e1799c8a046eb3f6dd6e94/final_checkpoint/config.json
Model weights saved in experiments/2023-02-22-5f8754af240241f4c78ba924918b0bfb26f26a17b0e1799c8a046eb3f6dd6e94/final_checkpoint/pytorch_model.bin
tokenizer config file saved in experiments/2023-02-22-5f8754af240241f4c78ba924918b0bfb26f26a17b0e1799c8a046eb3f6dd6e94/final_checkpoint/tokenizer/tokenizer_config.json
Special tokens file saved in experiments/2023-02-22-5f8754af240241f4c78ba924918b0bfb26f26a17b0e1799c8a046eb3f6dd6e94/final_checkpoint/tokenizer/special_tokens_map.json
Saving model checkpoint to experiments/2023-02-22-5f8754af240241f4c78ba924918b0bfb26f26a17b0e1799c8a046eb3f6dd6e94/trainer_final_checkpoint
Configuration saved in experiments/2023-02-22-5f8754af240241f4c78ba924918b0bfb26f26a17b0e1799c8a046eb3f6dd6e94/trainer_final_checkpoint/config.json
Model weights saved in experiments/2023-02-22-5f8754af240241f4c78ba924918b0bfb26f26a17b0e1799c8a046eb3f6dd6e94/trainer_final_checkpoint/pytorch_model.bin
tokenizer config file saved in experiments/2023-02-22-5f8754af240241f4c78ba924918b0bfb26f26a17b0e1799c8a046eb3f6dd6e94/trainer_final_checkpoint/tokenizer_config.json
Special tokens file saved in experiments/2023-02-22-5f8754af240241f4c78ba924918b0bfb26f26a17b0e1799c8a046eb3f6dd6e94/trainer_final_checkpoint/special_tokens_map.json
Traceback (most recent call last):
  File "tune_gpt.py", line 228, in <module>
    trainer.save_state(trainer_save_dir)
TypeError: save_state() takes 1 positional argument but 2 were given
[2023-02-22 05:05:17,992] [INFO] [launch.py:350:main] Process 11659 exits successfully.
[2023-02-22 05:05:17,993] [INFO] [launch.py:350:main] Process 11661 exits successfully.
[2023-02-22 05:05:17,994] [INFO] [launch.py:350:main] Process 11655 exits successfully.
[2023-02-22 05:05:18,995] [INFO] [launch.py:350:main] Process 11654 exits successfully.
[2023-02-22 05:05:18,996] [INFO] [launch.py:350:main] Process 11656 exits successfully.
[2023-02-22 05:05:18,996] [INFO] [launch.py:350:main] Process 11653 exits successfully.
[2023-02-22 05:05:18,997] [INFO] [launch.py:350:main] Process 11657 exits successfully.
[2023-02-22 05:05:19,998] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 11652
[2023-02-22 05:05:20,000] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 11653
[2023-02-22 05:05:20,001] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 11654
[2023-02-22 05:05:20,001] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 11655
[2023-02-22 05:05:20,001] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 11656
[2023-02-22 05:05:20,001] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 11657
[2023-02-22 05:05:20,001] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 11659
[2023-02-22 05:05:20,001] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 11661
[2023-02-22 05:05:20,002] [ERROR] [launch.py:324:sigkill_handler] ['/opt/conda/bin/python3', '-u', 'tune_gpt.py', '--local_rank=7', '--deepspeed', 'deepspeed.json', '--upload-experiment'] exits with return code = 1
/opt/conda/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  from pandas import MultiIndex, Int64Index