Model save

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ecd8b0daf78e11f6580bc81ddb2d90bb94e3225d215d9e89eba3e80edbc77004
+oid sha256:0d745968c412da9bc113cddbf19c5ec4b4d3eccb04366c369874f720414d7357
 size 3086784957

run.log

@@ -6259,3 +6259,199 @@ Time to load utils op: 0.0003571510314941406 seconds
 [2022-12-19 17:51:17,444] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./checkpoint-5000/global_step5018/zero_pp_rank_0_mp_rank_00_optim_states.pt.
 [2022-12-19 17:51:17,444] [INFO] [engine.py:3269:_save_zero_checkpoint] zero checkpoint saved ./checkpoint-5000/global_step5018/zero_pp_rank_0_mp_rank_00_optim_states.pt
 [2022-12-19 17:51:17,444] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5018 is ready now!
+[2022-12-19 17:53:07,628] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed info: version=0.7.7, git-hash=unknown, git-branch=unknown
+[2022-12-19 17:53:07,690] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
+[2022-12-19 17:53:09,125] [WARNING] [cpu_adam.py:83:__init__] FP16 params for CPUAdam may not work on AMD CPUs
+Installed CUDA version 11.6 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
+Time to load cpu_adam op: 3.2118442058563232 seconds
+Adam Optimizer #1 is created with AVX2 arithmetic capability.
+Config: alpha=0.000010, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_w=1
+[2022-12-19 17:53:12,976] [INFO] [logging.py:68:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer
+[2022-12-19 17:53:13,280] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
+[2022-12-19 17:53:13,280] [INFO] [utils.py:52:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
+[2022-12-19 17:53:13,280] [INFO] [logging.py:68:log_dist] [Rank 0] Creating fp16 ZeRO stage 2 optimizer
+[2022-12-19 17:53:13,280] [INFO] [stage_1_and_2.py:140:__init__] Reduce bucket size 200000000
+[2022-12-19 17:53:13,280] [INFO] [stage_1_and_2.py:141:__init__] Allgather bucket size 200000000
+[2022-12-19 17:53:13,280] [INFO] [stage_1_and_2.py:142:__init__] CPU Offload: True
+[2022-12-19 17:53:13,280] [INFO] [stage_1_and_2.py:143:__init__] Round robin gradient partitioning: False
+Time to load utils op: 0.0003826618194580078 seconds
+Rank: 0 partition count [1] and sizes[(1543304960, False)] 
+[2022-12-19 17:53:16,589] [INFO] [utils.py:827:see_memory_usage] Before initializing optimizer states
+[2022-12-19 17:53:16,590] [INFO] [utils.py:828:see_memory_usage] MA 6.0 GB         Max_MA 19.53 GB         CA 29.61 GB         Max_CA 30 GB 
+[2022-12-19 17:53:16,590] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 48.69 GB, percent = 24.8%
+[2022-12-19 17:53:20,495] [INFO] [utils.py:827:see_memory_usage] After initializing optimizer states
+[2022-12-19 17:53:20,496] [INFO] [utils.py:828:see_memory_usage] MA 6.0 GB         Max_MA 6.0 GB         CA 29.61 GB         Max_CA 30 GB 
+[2022-12-19 17:53:20,496] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 68.29 GB, percent = 34.7%
+[2022-12-19 17:53:20,496] [INFO] [stage_1_and_2.py:525:__init__] optimizer state initialized
+[2022-12-19 17:53:20,573] [INFO] [utils.py:827:see_memory_usage] After initializing ZeRO optimizer
+[2022-12-19 17:53:20,574] [INFO] [utils.py:828:see_memory_usage] MA 6.0 GB         Max_MA 6.0 GB         CA 29.61 GB         Max_CA 30 GB 
+[2022-12-19 17:53:20,574] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 68.3 GB, percent = 34.7%
+[2022-12-19 17:53:20,603] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw
+[2022-12-19 17:53:20,604] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed using configured LR scheduler = WarmupDecayLR
+[2022-12-19 17:53:20,604] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed LR Scheduler = <deepspeed.runtime.lr_schedules.WarmupDecayLR object at 0x7f72490c8c40>
+[2022-12-19 17:53:20,604] [INFO] [logging.py:68:log_dist] [Rank 0] step=0, skipped=0, lr=[1e-05], mom=[[0.9, 0.999]]
+[2022-12-19 17:53:20,605] [INFO] [config.py:1020:print] DeepSpeedEngine configuration:
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   activation_checkpointing_config  {
+    "partition_activations": false, 
+    "contiguous_memory_optimization": false, 
+    "cpu_checkpointing": false, 
+    "number_checkpoints": null, 
+    "synchronize_checkpoint_boundary": false, 
+    "profile": false
+}
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   amp_enabled .................. False
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   amp_params ................... False
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   autotuning_config ............ {
+    "enabled": false, 
+    "start_step": null, 
+    "end_step": null, 
+    "metric_path": null, 
+    "arg_mappings": null, 
+    "metric": "throughput", 
+    "model_info": null, 
+    "results_dir": "autotuning_results", 
+    "exps_dir": "autotuning_exps", 
+    "overwrite": true, 
+    "fast": true, 
+    "start_profile_step": 3, 
+    "end_profile_step": 5, 
+    "tuner_type": "gridsearch", 
+    "tuner_early_stopping": 5, 
+    "tuner_num_trials": 50, 
+    "model_info_path": null, 
+    "mp_size": 1, 
+    "max_train_batch_size": null, 
+    "min_train_batch_size": 1, 
+    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
+    "min_train_micro_batch_size_per_gpu": 1, 
+    "num_tuning_micro_batch_sizes": 3
+}
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   bfloat16_enabled ............. False
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   checkpoint_parallel_write_pipeline  False
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   checkpoint_tag_validation_enabled  True
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   checkpoint_tag_validation_fail  False
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7f725294b3a0>
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   communication_data_type ...... None
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   curriculum_enabled ........... False
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   curriculum_params ............ False
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   dataloader_drop_last ......... False
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   disable_allgather ............ False
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   dump_state ................... False
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   dynamic_loss_scale_args ...... {'init_scale': 65536, 'scale_window': 1000, 'delayed_shift': 2, 'min_scale': 1}
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   eigenvalue_enabled ........... False
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   eigenvalue_gas_boundary_resolution  1
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   eigenvalue_layer_name ........ bert.encoder.layer
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   eigenvalue_layer_num ......... 0
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   eigenvalue_max_iter .......... 100
+[2022-12-19 17:53:20,606] [INFO] [config.py:1024:print]   eigenvalue_stability ......... 1e-06
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   eigenvalue_tol ............... 0.01
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   eigenvalue_verbose ........... False
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   elasticity_enabled ........... False
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   flops_profiler_config ........ {
+    "enabled": false, 
+    "profile_step": 1, 
+    "module_depth": -1, 
+    "top_modules": 1, 
+    "detailed": true, 
+    "output_file": null
+}
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   fp16_auto_cast ............... False
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   fp16_enabled ................. True
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   fp16_master_weights_and_gradients  False
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   global_rank .................. 0
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   grad_accum_dtype ............. None
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   gradient_accumulation_steps .. 2
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   gradient_clipping ............ 1.0
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   gradient_predivide_factor .... 1.0
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   initial_dynamic_scale ........ 65536
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   load_universal_checkpoint .... False
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   loss_scale ................... 0
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   memory_breakdown ............. False
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   monitor_config ............... <deepspeed.monitor.config.DeepSpeedMonitorConfig object at 0x7f725294b2e0>
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   nebula_config ................ {
+    "enabled": false, 
+    "persistent_storage_path": null, 
+    "persistent_time_interval": 100, 
+    "num_of_version_in_retention": 2, 
+    "enable_nebula_load": true, 
+    "load_path": null
+}
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   optimizer_legacy_fusion ...... False
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   optimizer_name ............... adamw
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   optimizer_params ............. {'lr': 1e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.0}
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   pld_enabled .................. False
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   pld_params ................... False
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   prescale_gradients ........... False
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   scheduler_name ............... WarmupDecayLR
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   scheduler_params ............. {'last_batch_iteration': -1, 'total_num_steps': 5000, 'warmup_min_lr': 0, 'warmup_max_lr': 1e-05, 'warmup_num_steps': 500}
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   sparse_attention ............. None
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   sparse_gradients_enabled ..... False
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   steps_per_print .............. 10
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   train_batch_size ............. 64
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   train_micro_batch_size_per_gpu  32
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   use_node_local_storage ....... False
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   wall_clock_breakdown ......... False
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   world_size ................... 1
+[2022-12-19 17:53:20,607] [INFO] [config.py:1024:print]   zero_allow_untested_optimizer  False
+[2022-12-19 17:53:20,608] [INFO] [config.py:1024:print]   zero_config .................. stage=2 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=200000000 allgather_partitions=True allgather_bucket_size=200000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='cpu', nvme_path=None, buffer_count=4, pin_memory=True, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False) sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False
+[2022-12-19 17:53:20,608] [INFO] [config.py:1024:print]   zero_enabled ................. True
+[2022-12-19 17:53:20,608] [INFO] [config.py:1024:print]   zero_optimization_stage ...... 2
+[2022-12-19 17:53:20,608] [INFO] [config.py:1009:print_user_config]   json = {
+    "fp16": {
+        "enabled": true, 
+        "loss_scale": 0, 
+        "loss_scale_window": 1000, 
+        "initial_scale_power": 16, 
+        "hysteresis": 2, 
+        "min_loss_scale": 1
+    }, 
+    "optimizer": {
+        "type": "AdamW", 
+        "params": {
+            "lr": 1e-05, 
+            "betas": [0.9, 0.999], 
+            "eps": 1e-08, 
+            "weight_decay": 0.0
+        }
+    }, 
+    "scheduler": {
+        "type": "WarmupDecayLR", 
+        "params": {
+            "last_batch_iteration": -1, 
+            "total_num_steps": 5.000000e+03, 
+            "warmup_min_lr": 0, 
+            "warmup_max_lr": 1e-05, 
+            "warmup_num_steps": 500
+        }
+    }, 
+    "zero_optimization": {
+        "stage": 2, 
+        "offload_optimizer": {
+            "device": "cpu", 
+            "pin_memory": true
+        }, 
+        "allgather_partitions": true, 
+        "allgather_bucket_size": 2.000000e+08, 
+        "overlap_comm": true, 
+        "reduce_scatter": true, 
+        "reduce_bucket_size": 2.000000e+08, 
+        "contiguous_gradients": true
+    }, 
+    "gradient_accumulation_steps": 2, 
+    "gradient_clipping": 1.0, 
+    "train_batch_size": 64, 
+    "train_micro_batch_size_per_gpu": 32
+}
+Time to load utils op: 0.0003173351287841797 seconds
+[2022-12-19 17:53:20,609] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from ./checkpoint-4000/global_step4015/mp_rank_00_model_states.pt...
+[2022-12-19 17:53:21,616] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from ./checkpoint-4000/global_step4015/mp_rank_00_model_states.pt.
+[2022-12-19 17:53:21,742] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from ./checkpoint-4000/global_step4015/mp_rank_00_model_states.pt...
+[2022-12-19 17:53:22,870] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from ./checkpoint-4000/global_step4015/mp_rank_00_model_states.pt.
+[2022-12-19 17:53:23,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from ./checkpoint-4000/global_step4015/zero_pp_rank_0_mp_rank_00_optim_states.pt...
+[2022-12-19 17:53:29,120] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from ./checkpoint-4000/global_step4015/zero_pp_rank_0_mp_rank_00_optim_states.pt.
+[2022-12-19 17:53:29,120] [INFO] [engine.py:2900:_get_all_zero_checkpoint_state_dicts] successfully read 1 ZeRO state_dicts for rank 0
+[2022-12-19 17:53:30,350] [INFO] [engine.py:2840:_load_zero_checkpoint] loading 1 zero partition checkpoints for rank 0
+{'train_runtime': 64619.8702, 'train_samples_per_second': 4.952, 'train_steps_per_second': 0.077, 'train_loss': 0.01643497463464737, 'epoch': 37.59}

runs/Dec18_23-21-30_129-146-123-136/events.out.tfevents.1671407791.129-146-123-136.1701007.0

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:83c8f3ae9456db2cea61dd231e8ca0da01fb77066a4bb46d6ff0367742a9e0e9
-size 37257
+oid sha256:e5ab7d6f2245124021d2524cec318217a410cc12580935f564208ccf36be80cc
+size 37611