W1205 13:38:14.913000 139671368247104 torch/distributed/run.py:779] 
W1205 13:38:14.913000 139671368247104 torch/distributed/run.py:779] *****************************************
W1205 13:38:14.913000 139671368247104 torch/distributed/run.py:779] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W1205 13:38:14.913000 139671368247104 torch/distributed/run.py:779] *****************************************
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/_distutils_hack/__init__.py:55: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
  warnings.warn(
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/_distutils_hack/__init__.py:55: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
  warnings.warn(
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/_distutils_hack/__init__.py:55: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
  warnings.warn(
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/_distutils_hack/__init__.py:55: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
  warnings.warn(
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/_distutils_hack/__init__.py:55: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
  warnings.warn(
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/_distutils_hack/__init__.py:55: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
  warnings.warn(
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/_distutils_hack/__init__.py:55: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
  warnings.warn(
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/_distutils_hack/__init__.py:55: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
  warnings.warn(
[2024-12-05 13:38:39,939] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-12-05 13:38:39,939] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-12-05 13:38:39,939] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-12-05 13:38:39,939] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-12-05 13:38:39,939] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-12-05 13:38:39,939] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-12-05 13:38:39,939] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-12-05 13:38:39,939] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
df: df: /root/.triton/autotune/root/.triton/autotunedf: /root/.triton/autotune: 没有那个文件或目录
: 没有那个文件或目录
: 没有那个文件或目录
df: /root/.triton/autotune: 没有那个文件或目录
df: /root/.triton/autotune: 没有那个文件或目录
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/transformers/training_args.py:1733: FutureWarning: Using `--dispatch_batches` is deprecated and will be removed in version 4.41 of 🤗 Transformers. Use `--accelerator_config {'dispatch_batches':VALUE} instead
  warnings.warn(
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/transformers/training_args.py:1733: FutureWarning: Using `--dispatch_batches` is deprecated and will be removed in version 4.41 of 🤗 Transformers. Use `--accelerator_config {'dispatch_batches':VALUE} instead
  warnings.warn(
[2024-12-05 13:38:47,217] [INFO] [comm.py:652:init_distributed] cdb=None
[2024-12-05 13:38:47,217] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[2024-12-05 13:38:47,218] [INFO] [comm.py:652:init_distributed] cdb=None
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/transformers/training_args.py:1733: FutureWarning: Using `--dispatch_batches` is deprecated and will be removed in version 4.41 of 🤗 Transformers. Use `--accelerator_config {'dispatch_batches':VALUE} instead
  warnings.warn(
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/transformers/training_args.py:1733: FutureWarning: Using `--dispatch_batches` is deprecated and will be removed in version 4.41 of 🤗 Transformers. Use `--accelerator_config {'dispatch_batches':VALUE} instead
  warnings.warn(
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/transformers/training_args.py:1733: FutureWarning: Using `--dispatch_batches` is deprecated and will be removed in version 4.41 of 🤗 Transformers. Use `--accelerator_config {'dispatch_batches':VALUE} instead
  warnings.warn(
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/transformers/training_args.py:1733: FutureWarning: Using `--dispatch_batches` is deprecated and will be removed in version 4.41 of 🤗 Transformers. Use `--accelerator_config {'dispatch_batches':VALUE} instead
  warnings.warn(
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/transformers/training_args.py:1733: FutureWarning: Using `--dispatch_batches` is deprecated and will be removed in version 4.41 of 🤗 Transformers. Use `--accelerator_config {'dispatch_batches':VALUE} instead
  warnings.warn(
[2024-12-05 13:38:47,229] [INFO] [comm.py:652:init_distributed] cdb=None
[2024-12-05 13:38:47,230] [INFO] [comm.py:652:init_distributed] cdb=None
[2024-12-05 13:38:47,231] [INFO] [comm.py:652:init_distributed] cdb=None
[2024-12-05 13:38:47,231] [INFO] [comm.py:652:init_distributed] cdb=None
[2024-12-05 13:38:47,231] [INFO] [comm.py:652:init_distributed] cdb=None
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/transformers/training_args.py:1733: FutureWarning: Using `--dispatch_batches` is deprecated and will be removed in version 4.41 of 🤗 Transformers. Use `--accelerator_config {'dispatch_batches':VALUE} instead
  warnings.warn(
[2024-12-05 13:38:47,237] [INFO] [comm.py:652:init_distributed] cdb=None
12/05/2024 13:38:47 - INFO - llamafactory.hparams.parser - Process rank: 0, device: cuda:0, n_gpu: 1, distributed training: True, compute dtype: torch.bfloat16
[INFO|configuration_utils.py:670] 2024-12-05 13:38:48,151 >> loading configuration file /nas/shared/NLP_A100/wuzhenyu/LLMs/Qwen2-VL-7B-Instruct/config.json
[WARNING|modeling_rope_utils.py:379] 2024-12-05 13:38:48,155 >> Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}
[INFO|configuration_utils.py:739] 2024-12-05 13:38:48,156 >> Model config Qwen2VLConfig {
  "_name_or_path": "/nas/shared/NLP_A100/wuzhenyu/LLMs/Qwen2-VL-7B-Instruct",
  "architectures": [
    "Qwen2VLForConditionalGeneration"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "hidden_act": "silu",
  "hidden_size": 3584,
  "image_token_id": 151655,
  "initializer_range": 0.02,
  "intermediate_size": 18944,
  "max_position_embeddings": 32768,
  "max_window_layers": 28,
  "model_type": "qwen2_vl",
  "num_attention_heads": 28,
  "num_hidden_layers": 28,
  "num_key_value_heads": 4,
  "rms_norm_eps": 1e-06,
  "rope_scaling": {
    "mrope_section": [
      16,
      24,
      24
    ],
    "rope_type": "default",
    "type": "default"
  },
  "rope_theta": 1000000.0,
  "sliding_window": 32768,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.45.0",
  "use_cache": true,
  "use_sliding_window": false,
  "video_token_id": 151656,
  "vision_config": {
    "in_chans": 3,
    "model_type": "qwen2_vl",
    "spatial_patch_size": 14
  },
  "vision_end_token_id": 151653,
  "vision_start_token_id": 151652,
  "vision_token_id": 151654,
  "vocab_size": 152064
}

[INFO|tokenization_utils_base.py:2212] 2024-12-05 13:38:48,288 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2212] 2024-12-05 13:38:48,288 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2212] 2024-12-05 13:38:48,288 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2212] 2024-12-05 13:38:48,288 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2212] 2024-12-05 13:38:48,288 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2212] 2024-12-05 13:38:48,288 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2478] 2024-12-05 13:38:48,699 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[INFO|image_processing_base.py:373] 2024-12-05 13:38:48,771 >> loading configuration file /nas/shared/NLP_A100/wuzhenyu/LLMs/Qwen2-VL-7B-Instruct/preprocessor_config.json
[INFO|image_processing_base.py:373] 2024-12-05 13:38:48,793 >> loading configuration file /nas/shared/NLP_A100/wuzhenyu/LLMs/Qwen2-VL-7B-Instruct/preprocessor_config.json
[INFO|image_processing_base.py:429] 2024-12-05 13:38:48,793 >> Image processor Qwen2VLImageProcessor {
  "do_convert_rgb": true,
  "do_normalize": true,
  "do_rescale": true,
  "do_resize": true,
  "image_mean": [
    0.48145466,
    0.4578275,
    0.40821073
  ],
  "image_processor_type": "Qwen2VLImageProcessor",
  "image_std": [
    0.26862954,
    0.26130258,
    0.27577711
  ],
  "max_pixels": 12845056,
  "merge_size": 2,
  "min_pixels": 3136,
  "patch_size": 14,
  "processor_class": "Qwen2VLProcessor",
  "resample": 3,
  "rescale_factor": 0.00392156862745098,
  "size": {
    "max_pixels": 12845056,
    "min_pixels": 3136
  },
  "temporal_patch_size": 2
}

[INFO|tokenization_utils_base.py:2212] 2024-12-05 13:38:48,805 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2212] 2024-12-05 13:38:48,806 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2212] 2024-12-05 13:38:48,806 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2212] 2024-12-05 13:38:48,806 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2212] 2024-12-05 13:38:48,806 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2212] 2024-12-05 13:38:48,806 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2478] 2024-12-05 13:38:49,114 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[INFO|processing_utils.py:744] 2024-12-05 13:38:49,684 >> Processor Qwen2VLProcessor:
- image_processor: Qwen2VLImageProcessor {
  "do_convert_rgb": true,
  "do_normalize": true,
  "do_rescale": true,
  "do_resize": true,
  "image_mean": [
    0.48145466,
    0.4578275,
    0.40821073
  ],
  "image_processor_type": "Qwen2VLImageProcessor",
  "image_std": [
    0.26862954,
    0.26130258,
    0.27577711
  ],
  "max_pixels": 12845056,
  "merge_size": 2,
  "min_pixels": 3136,
  "patch_size": 14,
  "processor_class": "Qwen2VLProcessor",
  "resample": 3,
  "rescale_factor": 0.00392156862745098,
  "size": {
    "max_pixels": 12845056,
    "min_pixels": 3136
  },
  "temporal_patch_size": 2
}

- tokenizer: Qwen2TokenizerFast(name_or_path='/nas/shared/NLP_A100/wuzhenyu/LLMs/Qwen2-VL-7B-Instruct', vocab_size=151643, model_max_length=32768, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>', 'additional_special_tokens': ['<|im_start|>', '<|im_end|>', '<|object_ref_start|>', '<|object_ref_end|>', '<|box_start|>', '<|box_end|>', '<|quad_start|>', '<|quad_end|>', '<|vision_start|>', '<|vision_end|>', '<|vision_pad|>', '<|image_pad|>', '<|video_pad|>']}, clean_up_tokenization_spaces=False),  added_tokens_decoder={
	151643: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	151644: AddedToken("<|im_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	151645: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	151646: AddedToken("<|object_ref_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	151647: AddedToken("<|object_ref_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	151648: AddedToken("<|box_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	151649: AddedToken("<|box_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	151650: AddedToken("<|quad_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	151651: AddedToken("<|quad_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	151652: AddedToken("<|vision_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	151653: AddedToken("<|vision_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	151654: AddedToken("<|vision_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	151655: AddedToken("<|image_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	151656: AddedToken("<|video_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}

{
  "processor_class": "Qwen2VLProcessor"
}

12/05/2024 13:38:49 - INFO - llamafactory.data.template - Replace eos token: <|im_end|>
12/05/2024 13:38:49 - INFO - llamafactory.data.loader - Loading dataset sim/1205/1205_aw_24k.jsonl...
12/05/2024 13:38:50 - INFO - llamafactory.hparams.parser - Process rank: 6, device: cuda:6, n_gpu: 1, distributed training: True, compute dtype: torch.bfloat16
12/05/2024 13:38:50 - INFO - llamafactory.hparams.parser - Process rank: 2, device: cuda:2, n_gpu: 1, distributed training: True, compute dtype: torch.bfloat16
12/05/2024 13:38:50 - INFO - llamafactory.hparams.parser - Process rank: 3, device: cuda:3, n_gpu: 1, distributed training: True, compute dtype: torch.bfloat16
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}
12/05/2024 13:38:50 - INFO - llamafactory.hparams.parser - Process rank: 4, device: cuda:4, n_gpu: 1, distributed training: True, compute dtype: torch.bfloat16
12/05/2024 13:38:50 - INFO - llamafactory.hparams.parser - Process rank: 7, device: cuda:7, n_gpu: 1, distributed training: True, compute dtype: torch.bfloat16
12/05/2024 13:38:50 - INFO - llamafactory.hparams.parser - Process rank: 1, device: cuda:1, n_gpu: 1, distributed training: True, compute dtype: torch.bfloat16
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}
12/05/2024 13:38:50 - INFO - llamafactory.hparams.parser - Process rank: 5, device: cuda:5, n_gpu: 1, distributed training: True, compute dtype: torch.bfloat16
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}

Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 5754 examples [00:00, 43710.45 examples/s]
Generating train split: 23931 examples [00:00, 83108.86 examples/s]
Generating train split: 24618 examples [00:00, 78770.57 examples/s]
12/05/2024 13:38:51 - INFO - llamafactory.data.template - Replace eos token: <|im_end|>
12/05/2024 13:38:51 - INFO - llamafactory.data.template - Replace eos token: <|im_end|>
12/05/2024 13:38:51 - INFO - llamafactory.data.template - Replace eos token: <|im_end|>
12/05/2024 13:38:51 - INFO - llamafactory.data.template - Replace eos token: <|im_end|>
12/05/2024 13:38:51 - INFO - llamafactory.data.template - Replace eos token: <|im_end|>
12/05/2024 13:38:51 - INFO - llamafactory.data.template - Replace eos token: <|im_end|>
12/05/2024 13:38:52 - INFO - llamafactory.data.template - Replace eos token: <|im_end|>

Converting format of dataset (num_proc=320):   0%|          | 0/24618 [00:00<?, ? examples/s]
Converting format of dataset (num_proc=320):   0%|          | 1/24618 [00:00<2:08:52,  3.18 examples/s]
Converting format of dataset (num_proc=320):   1%|          | 201/24618 [00:00<00:39, 617.70 examples/s]
Converting format of dataset (num_proc=320):   1%|▏         | 315/24618 [00:00<00:33, 719.00 examples/s]
Converting format of dataset (num_proc=320):   3%|▎         | 730/24618 [00:00<00:14, 1704.55 examples/s]
Converting format of dataset (num_proc=320):   4%|▍         | 1102/24618 [00:00<00:10, 2241.07 examples/s]
Converting format of dataset (num_proc=320):   6%|▌         | 1371/24618 [00:00<00:09, 2327.84 examples/s]
Converting format of dataset (num_proc=320):   8%|▊         | 2035/24618 [00:00<00:06, 3569.43 examples/s]
Converting format of dataset (num_proc=320):  11%|█▏        | 2785/24618 [00:01<00:04, 4706.54 examples/s]
Converting format of dataset (num_proc=320):  15%|█▍        | 3672/24618 [00:01<00:03, 5928.04 examples/s]
Converting format of dataset (num_proc=320):  18%|█▊        | 4441/24618 [00:01<00:03, 6441.55 examples/s]
Converting format of dataset (num_proc=320):  22%|██▏       | 5314/24618 [00:01<00:02, 7115.35 examples/s]
Converting format of dataset (num_proc=320):  25%|██▌       | 6169/24618 [00:01<00:02, 7511.75 examples/s]
Converting format of dataset (num_proc=320):  28%|██▊       | 6993/24618 [00:01<00:02, 7726.24 examples/s]
Converting format of dataset (num_proc=320):  33%|███▎      | 8024/24618 [00:01<00:01, 8496.09 examples/s]
Converting format of dataset (num_proc=320):  36%|███▌      | 8890/24618 [00:01<00:01, 8542.95 examples/s]
Converting format of dataset (num_proc=320):  40%|████      | 9848/24618 [00:01<00:01, 8835.43 examples/s]
Converting format of dataset (num_proc=320):  44%|████▎     | 10756/24618 [00:01<00:01, 8843.51 examples/s]
Converting format of dataset (num_proc=320):  47%|████▋     | 11645/24618 [00:02<00:01, 8365.98 examples/s]
Converting format of dataset (num_proc=320):  51%|█████     | 12504/24618 [00:02<00:01, 7736.55 examples/s]
Converting format of dataset (num_proc=320):  54%|█████▍    | 13291/24618 [00:02<00:01, 7751.50 examples/s]
Converting format of dataset (num_proc=320):  58%|█████▊    | 14350/24618 [00:02<00:01, 8544.66 examples/s]
Converting format of dataset (num_proc=320):  64%|██████▍   | 15708/24618 [00:02<00:00, 9979.10 examples/s]
Converting format of dataset (num_proc=320):  69%|██████▉   | 16966/24618 [00:02<00:00, 10729.11 examples/s]
Converting format of dataset (num_proc=320):  73%|███████▎  | 18054/24618 [00:02<00:00, 10737.86 examples/s]
Converting format of dataset (num_proc=320):  78%|███████▊  | 19138/24618 [00:02<00:00, 10548.47 examples/s]
Converting format of dataset (num_proc=320):  82%|████████▏ | 20201/24618 [00:02<00:00, 10280.59 examples/s]
Converting format of dataset (num_proc=320):  87%|████████▋ | 21459/24618 [00:03<00:00, 10942.27 examples/s]
Converting format of dataset (num_proc=320):  92%|█████████▏| 22565/24618 [00:03<00:00, 8700.53 examples/s] 
Converting format of dataset (num_proc=320):  96%|█████████▌| 23511/24618 [00:03<00:00, 8644.19 examples/s]
Converting format of dataset (num_proc=320):  99%|█████████▉| 24430/24618 [00:03<00:00, 5924.34 examples/s]
Converting format of dataset (num_proc=320): 100%|██████████| 24618/24618 [00:03<00:00, 6554.46 examples/s]
dlc1o8747bm15inu-master-0:71:71 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
dlc1o8747bm15inu-master-0:71:71 [0] NCCL INFO Bootstrap : Using eth0:22.8.20.90<0>
dlc1o8747bm15inu-master-0:71:71 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so
dlc1o8747bm15inu-master-0:71:71 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation
dlc1o8747bm15inu-master-0:71:71 [0] NCCL INFO cudaDriverVersion 12010
NCCL version 2.20.5+cuda12.4
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO NCCL_IB_HCA set to mlx5
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:22.8.20.90<0>
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Using non-device net plugin version 0
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Using network IB
dlc1o8747bm15inu-master-0:76:76 [5] NCCL INFO cudaDriverVersion 12010
dlc1o8747bm15inu-master-0:76:76 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
dlc1o8747bm15inu-master-0:76:76 [5] NCCL INFO Bootstrap : Using eth0:22.8.20.90<0>
dlc1o8747bm15inu-master-0:76:76 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so
dlc1o8747bm15inu-master-0:75:75 [4] NCCL INFO cudaDriverVersion 12010
dlc1o8747bm15inu-master-0:75:75 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
dlc1o8747bm15inu-master-0:75:75 [4] NCCL INFO Bootstrap : Using eth0:22.8.20.90<0>
dlc1o8747bm15inu-master-0:75:75 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so
dlc1o8747bm15inu-master-0:72:72 [1] NCCL INFO cudaDriverVersion 12010
dlc1o8747bm15inu-master-0:74:74 [3] NCCL INFO cudaDriverVersion 12010
dlc1o8747bm15inu-master-0:72:72 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
dlc1o8747bm15inu-master-0:74:74 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
dlc1o8747bm15inu-master-0:73:73 [2] NCCL INFO cudaDriverVersion 12010
dlc1o8747bm15inu-master-0:72:72 [1] NCCL INFO Bootstrap : Using eth0:22.8.20.90<0>
dlc1o8747bm15inu-master-0:74:74 [3] NCCL INFO Bootstrap : Using eth0:22.8.20.90<0>
dlc1o8747bm15inu-master-0:72:72 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so
dlc1o8747bm15inu-master-0:74:74 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so
dlc1o8747bm15inu-master-0:73:73 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
dlc1o8747bm15inu-master-0:73:73 [2] NCCL INFO Bootstrap : Using eth0:22.8.20.90<0>
dlc1o8747bm15inu-master-0:73:73 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so
dlc1o8747bm15inu-master-0:78:78 [7] NCCL INFO cudaDriverVersion 12010
dlc1o8747bm15inu-master-0:78:78 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
dlc1o8747bm15inu-master-0:78:78 [7] NCCL INFO Bootstrap : Using eth0:22.8.20.90<0>
dlc1o8747bm15inu-master-0:78:78 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so
dlc1o8747bm15inu-master-0:76:76 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation
dlc1o8747bm15inu-master-0:75:75 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation
dlc1o8747bm15inu-master-0:72:72 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation
dlc1o8747bm15inu-master-0:74:74 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation
dlc1o8747bm15inu-master-0:73:73 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation
dlc1o8747bm15inu-master-0:77:77 [6] NCCL INFO cudaDriverVersion 12010
dlc1o8747bm15inu-master-0:77:77 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
dlc1o8747bm15inu-master-0:78:78 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation
dlc1o8747bm15inu-master-0:77:77 [6] NCCL INFO Bootstrap : Using eth0:22.8.20.90<0>
dlc1o8747bm15inu-master-0:77:77 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so
dlc1o8747bm15inu-master-0:77:77 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO NCCL_IB_HCA set to mlx5
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO NCCL_IB_HCA set to mlx5
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO NCCL_IB_HCA set to mlx5
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO NCCL_IB_HCA set to mlx5
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO NCCL_IB_HCA set to mlx5
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO NCCL_IB_HCA set to mlx5
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO NCCL_IB_HCA set to mlx5
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:22.8.20.90<0>
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Using non-device net plugin version 0
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Using network IB
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:22.8.20.90<0>
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:22.8.20.90<0>
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Using non-device net plugin version 0
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Using network IB
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:22.8.20.90<0>
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Using non-device net plugin version 0
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Using network IB
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:22.8.20.90<0>
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:22.8.20.90<0>
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:22.8.20.90<0>
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Using non-device net plugin version 0
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Using network IB
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Using non-device net plugin version 0
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Using network IB
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Using non-device net plugin version 0
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Using network IB
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Using non-device net plugin version 0
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Using network IB
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO comm 0xa547c690 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId 80 commId 0x28a6667510b10773 - Init START
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO comm 0xa4c31ca0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 20 commId 0x28a6667510b10773 - Init START
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO comm 0xa6df0650 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId 30 commId 0x28a6667510b10773 - Init START
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO comm 0xa55b37a0 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId 70 commId 0x28a6667510b10773 - Init START
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO comm 0xaf287310 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId 50 commId 0x28a6667510b10773 - Init START
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO comm 0xa4b2bc20 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId 60 commId 0x28a6667510b10773 - Init START
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO comm 0xa75046a0 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId 40 commId 0x28a6667510b10773 - Init START
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO comm 0xc0177130 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 10 commId 0x28a6667510b10773 - Init START
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ffffffff,ffffffff
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO NVLS multicast support is not available on dev 1
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO NVLS multicast support is not available on dev 7
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Setting affinity for GPU 2 to ffff,ffffffff,ffffffff
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO NVLS multicast support is not available on dev 2
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO NVLS multicast support is not available on dev 4
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO NVLS multicast support is not available on dev 6
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Setting affinity for GPU 0 to ffff,ffffffff,ffffffff
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO NVLS multicast support is not available on dev 0
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Setting affinity for GPU 3 to ffff,ffffffff,ffffffff
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO NVLS multicast support is not available on dev 3
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO NVLS multicast support is not available on dev 5
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO comm 0xaf287310 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO comm 0xa75046a0 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO NCCL_MIN_NCHANNELS set by environment to 4.
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO comm 0xa6df0650 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO NCCL_MIN_NCHANNELS set by environment to 4.
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO comm 0xa547c690 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO NCCL_MIN_NCHANNELS set by environment to 4.
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO comm 0xa4c31ca0 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO comm 0xa55b37a0 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO comm 0xa4b2bc20 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO NCCL_MIN_NCHANNELS set by environment to 4.
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO comm 0xc0177130 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO NCCL_MIN_NCHANNELS set by environment to 4.
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO NCCL_MIN_NCHANNELS set by environment to 4.
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO NCCL_MIN_NCHANNELS set by environment to 4.
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO NCCL_MIN_NCHANNELS set by environment to 4.
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO P2P Chunksize set to 524288
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 00/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 01/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO P2P Chunksize set to 524288
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 02/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO P2P Chunksize set to 524288
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO P2P Chunksize set to 524288
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 03/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO P2P Chunksize set to 524288
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO P2P Chunksize set to 524288
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 04/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 05/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO P2P Chunksize set to 524288
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 06/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 07/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 08/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 09/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 10/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 11/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 12/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 13/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 14/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 15/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 16/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 17/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 18/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 19/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 20/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 21/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 22/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 23/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO P2P Chunksize set to 524288
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 00/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 01/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 02/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 03/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 04/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 04/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 04/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 04/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 05/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 05/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 05/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 05/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 05/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 05/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 06/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 06/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 06/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 06/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 06/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 06/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 07/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 07/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 07/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 07/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 07/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 08/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 08/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 08/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 08/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 09/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 09/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 09/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 09/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 09/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 10/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 10/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 10/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 10/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 10/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 10/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 11/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 11/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 11/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 11/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 11/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 12/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 12/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 12/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 12/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 13/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 13/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 13/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 13/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 13/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 13/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 14/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 14/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 14/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 14/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 14/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 14/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 15/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 15/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 15/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 15/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 15/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 16/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 16/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 16/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 16/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 16/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 16/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 17/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 17/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 17/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 17/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 17/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 17/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 18/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 18/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 18/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 18/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 18/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 18/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 19/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 19/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 19/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 19/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 19/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 19/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 20/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 20/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 20/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 20/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 20/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 20/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 21/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 21/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 21/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 21/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 21/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 21/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 22/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 22/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 22/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 22/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 22/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 22/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 23/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 23/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 23/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 23/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 23/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 23/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Connected all rings
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Connected all rings
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Connected all rings
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Connected all rings
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Connected all rings
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 05/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Connected all rings
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 00/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Connected all rings
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Connected all rings
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 01/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 02/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 11/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 03/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 04/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 13/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 05/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 06/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 00/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 07/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 01/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 16/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 08/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 02/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 17/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 09/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 03/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 18/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 10/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 04/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 19/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 11/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 05/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 20/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 12/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 06/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 21/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 13/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 07/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 22/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 14/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 08/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Channel 23/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 15/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 09/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 16/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 10/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 17/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 11/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 18/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 12/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 19/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 13/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 20/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 14/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 21/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 15/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 22/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 16/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Channel 23/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 00/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 17/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 01/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 18/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 01/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 02/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 19/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 03/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 20/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 03/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 04/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 21/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 04/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 05/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 22/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 05/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 06/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Channel 23/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 06/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 07/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 07/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 08/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 08/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 09/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 09/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 10/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 10/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 11/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 11/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 12/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 12/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 13/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 13/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 14/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 14/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 15/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 15/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 16/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 16/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 17/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 17/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 18/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 18/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 19/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 19/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 20/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 20/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 21/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 21/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 22/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 22/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Channel 23/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Channel 23/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO Connected all trees
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO Connected all trees
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO Connected all trees
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO Connected all trees
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO Connected all trees
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO Connected all trees
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO Connected all trees
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO Connected all trees
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dlc1o8747bm15inu-master-0:74:1649 [3] NCCL INFO comm 0xa75046a0 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId 40 commId 0x28a6667510b10773 - Init COMPLETE
dlc1o8747bm15inu-master-0:76:1646 [5] NCCL INFO comm 0xa4b2bc20 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId 60 commId 0x28a6667510b10773 - Init COMPLETE
dlc1o8747bm15inu-master-0:77:1652 [6] NCCL INFO comm 0xa55b37a0 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId 70 commId 0x28a6667510b10773 - Init COMPLETE
dlc1o8747bm15inu-master-0:72:1648 [1] NCCL INFO comm 0xa4c31ca0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 20 commId 0x28a6667510b10773 - Init COMPLETE
dlc1o8747bm15inu-master-0:78:1651 [7] NCCL INFO comm 0xa547c690 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId 80 commId 0x28a6667510b10773 - Init COMPLETE
dlc1o8747bm15inu-master-0:75:1647 [4] NCCL INFO comm 0xaf287310 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId 50 commId 0x28a6667510b10773 - Init COMPLETE
dlc1o8747bm15inu-master-0:71:1641 [0] NCCL INFO comm 0xc0177130 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 10 commId 0x28a6667510b10773 - Init COMPLETE
dlc1o8747bm15inu-master-0:73:1650 [2] NCCL INFO comm 0xa6df0650 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId 30 commId 0x28a6667510b10773 - Init COMPLETE
12/05/2024 13:39:03 - INFO - llamafactory.data.loader - Loading dataset sim/1205/1205_aw_24k.jsonl...
12/05/2024 13:39:03 - INFO - llamafactory.data.loader - Loading dataset sim/1205/1205_aw_24k.jsonl...
12/05/2024 13:39:03 - INFO - llamafactory.data.loader - Loading dataset sim/1205/1205_aw_24k.jsonl...
12/05/2024 13:39:03 - INFO - llamafactory.data.loader - Loading dataset sim/1205/1205_aw_24k.jsonl...
12/05/2024 13:39:03 - INFO - llamafactory.data.loader - Loading dataset sim/1205/1205_aw_24k.jsonl...
12/05/2024 13:39:03 - INFO - llamafactory.data.loader - Loading dataset sim/1205/1205_aw_24k.jsonl...
12/05/2024 13:39:03 - INFO - llamafactory.data.loader - Loading dataset sim/1205/1205_aw_24k.jsonl...

Running tokenizer on dataset (num_proc=320):   0%|          | 0/24618 [00:00<?, ? examples/s]
Running tokenizer on dataset (num_proc=320):   0%|          | 77/24618 [00:08<45:57,  8.90 examples/s]
Running tokenizer on dataset (num_proc=320):   1%|          | 154/24618 [00:08<19:47, 20.60 examples/s]
Running tokenizer on dataset (num_proc=320):   1%|▏         | 308/24618 [00:09<07:35, 53.38 examples/s]
Running tokenizer on dataset (num_proc=320):   2%|▏         | 462/24618 [00:09<04:20, 92.84 examples/s]
Running tokenizer on dataset (num_proc=320):   2%|▏         | 539/24618 [00:09<03:44, 107.21 examples/s]
Running tokenizer on dataset (num_proc=320):   3%|▎         | 770/24618 [00:09<01:57, 203.34 examples/s]
Running tokenizer on dataset (num_proc=320):   3%|▎         | 847/24618 [00:10<01:44, 226.41 examples/s]
Running tokenizer on dataset (num_proc=320):   4%|▍         | 924/24618 [00:10<01:41, 232.44 examples/s]
Running tokenizer on dataset (num_proc=320):   5%|▍         | 1155/24618 [00:10<00:59, 397.61 examples/s]
Running tokenizer on dataset (num_proc=320):   5%|▌         | 1309/24618 [00:11<01:03, 365.93 examples/s]
Running tokenizer on dataset (num_proc=320):   6%|▌         | 1386/24618 [00:11<01:18, 296.80 examples/s]
Running tokenizer on dataset (num_proc=320):   6%|▌         | 1463/24618 [00:13<02:41, 143.62 examples/s]
Running tokenizer on dataset (num_proc=320):   6%|▋         | 1540/24618 [00:13<02:17, 168.17 examples/s]
Running tokenizer on dataset (num_proc=320):   7%|▋         | 1617/24618 [00:13<01:57, 196.46 examples/s]
Running tokenizer on dataset (num_proc=320):   7%|▋         | 1694/24618 [00:13<01:42, 223.85 examples/s]
Running tokenizer on dataset (num_proc=320):   8%|▊         | 1848/24618 [00:13<01:12, 313.11 examples/s]
Running tokenizer on dataset (num_proc=320):   8%|▊         | 2002/24618 [00:14<00:58, 385.77 examples/s]
Running tokenizer on dataset (num_proc=320):   8%|▊         | 2079/24618 [00:14<01:13, 307.49 examples/s]
Running tokenizer on dataset (num_proc=320):   9%|▉         | 2233/24618 [00:14<00:56, 394.17 examples/s]
Running tokenizer on dataset (num_proc=320):   9%|▉         | 2310/24618 [00:15<00:57, 385.85 examples/s]
Running tokenizer on dataset (num_proc=320):  10%|▉         | 2387/24618 [00:15<00:55, 400.65 examples/s]
Running tokenizer on dataset (num_proc=320):  11%|█         | 2618/24618 [00:15<00:36, 609.69 examples/s]
Running tokenizer on dataset (num_proc=320):  11%|█         | 2695/24618 [00:15<00:38, 576.82 examples/s]
Running tokenizer on dataset (num_proc=320):  11%|█▏        | 2772/24618 [00:15<00:43, 502.48 examples/s]
Running tokenizer on dataset (num_proc=320):  12%|█▏        | 2849/24618 [00:16<00:46, 470.06 examples/s]
Running tokenizer on dataset (num_proc=320):  12%|█▏        | 3003/24618 [00:16<00:49, 436.87 examples/s]
Running tokenizer on dataset (num_proc=320):  13%|█▎        | 3157/24618 [00:16<00:53, 402.71 examples/s]
Running tokenizer on dataset (num_proc=320):  13%|█▎        | 3311/24618 [00:17<00:46, 458.55 examples/s]
Running tokenizer on dataset (num_proc=320):  14%|█▍        | 3465/24618 [00:17<00:39, 530.23 examples/s]
Running tokenizer on dataset (num_proc=320):  15%|█▍        | 3619/24618 [00:17<00:33, 621.93 examples/s]
Running tokenizer on dataset (num_proc=320):  15%|█▌        | 3696/24618 [00:17<00:39, 536.04 examples/s]
Running tokenizer on dataset (num_proc=320):  15%|█▌        | 3773/24618 [00:17<00:43, 480.41 examples/s]
Running tokenizer on dataset (num_proc=320):  16%|█▋        | 4004/24618 [00:18<00:33, 623.13 examples/s]
Running tokenizer on dataset (num_proc=320):  17%|█▋        | 4081/24618 [00:18<00:37, 540.72 examples/s]
Running tokenizer on dataset (num_proc=320):  17%|█▋        | 4158/24618 [00:18<00:41, 489.17 examples/s]
Running tokenizer on dataset (num_proc=320):  18%|█▊        | 4466/24618 [00:18<00:25, 800.35 examples/s]
Running tokenizer on dataset (num_proc=320):  19%|█▉        | 4697/24618 [00:18<00:21, 923.35 examples/s]
Running tokenizer on dataset (num_proc=320):  20%|█▉        | 4851/24618 [00:19<00:34, 570.36 examples/s]
Running tokenizer on dataset (num_proc=320):  20%|██        | 4928/24618 [00:19<00:44, 447.07 examples/s]
Running tokenizer on dataset (num_proc=320):  21%|██        | 5082/24618 [00:20<00:39, 493.59 examples/s]
Running tokenizer on dataset (num_proc=320):  21%|██▏       | 5236/24618 [00:20<00:33, 570.83 examples/s]
Running tokenizer on dataset (num_proc=320):  22%|██▏       | 5390/24618 [00:20<00:29, 642.60 examples/s]
Running tokenizer on dataset (num_proc=320):  22%|██▏       | 5467/24618 [00:20<00:35, 544.17 examples/s]
Running tokenizer on dataset (num_proc=320):  23%|██▎       | 5775/24618 [00:20<00:23, 786.69 examples/s]
Running tokenizer on dataset (num_proc=320):  24%|██▍       | 5929/24618 [00:21<00:31, 589.77 examples/s]
Running tokenizer on dataset (num_proc=320):  25%|██▌       | 6160/24618 [00:21<00:26, 692.31 examples/s]
Running tokenizer on dataset (num_proc=320):  27%|██▋       | 6545/24618 [00:21<00:18, 966.42 examples/s]
Running tokenizer on dataset (num_proc=320):  27%|██▋       | 6699/24618 [00:22<00:20, 885.68 examples/s]
Running tokenizer on dataset (num_proc=320):  28%|██▊       | 6853/24618 [00:22<00:21, 842.18 examples/s]
Running tokenizer on dataset (num_proc=320):  28%|██▊       | 7007/24618 [00:22<00:33, 523.47 examples/s]
Running tokenizer on dataset (num_proc=320):  29%|██▉       | 7084/24618 [00:23<00:35, 496.76 examples/s]
Running tokenizer on dataset (num_proc=320):  29%|██▉       | 7238/24618 [00:23<00:32, 541.36 examples/s]
Running tokenizer on dataset (num_proc=320):  30%|██▉       | 7315/24618 [00:23<00:34, 500.38 examples/s]
Running tokenizer on dataset (num_proc=320):  30%|███       | 7392/24618 [00:24<00:50, 344.49 examples/s]
Running tokenizer on dataset (num_proc=320):  31%|███       | 7546/24618 [00:24<00:38, 446.84 examples/s]
Running tokenizer on dataset (num_proc=320):  31%|███▏      | 7700/24618 [00:24<00:43, 390.36 examples/s]
Running tokenizer on dataset (num_proc=320):  32%|███▏      | 7854/24618 [00:25<00:40, 412.54 examples/s]
Running tokenizer on dataset (num_proc=320):  33%|███▎      | 8008/24618 [00:25<00:37, 445.45 examples/s]
Running tokenizer on dataset (num_proc=320):  33%|███▎      | 8239/24618 [00:25<00:27, 594.45 examples/s]
Running tokenizer on dataset (num_proc=320):  34%|███▍      | 8393/24618 [00:25<00:27, 582.29 examples/s]
Running tokenizer on dataset (num_proc=320):  34%|███▍      | 8470/24618 [00:25<00:28, 568.15 examples/s]
Running tokenizer on dataset (num_proc=320):  35%|███▍      | 8547/24618 [00:26<00:40, 394.29 examples/s]
Running tokenizer on dataset (num_proc=320):  35%|███▌      | 8624/24618 [00:26<00:39, 400.22 examples/s]
Running tokenizer on dataset (num_proc=320):  36%|███▌      | 8855/24618 [00:26<00:26, 605.29 examples/s]
Running tokenizer on dataset (num_proc=320):  36%|███▋      | 8932/24618 [00:26<00:26, 593.47 examples/s]
Running tokenizer on dataset (num_proc=320):  37%|███▋      | 9086/24618 [00:27<00:27, 573.18 examples/s]
Running tokenizer on dataset (num_proc=320):  38%|███▊      | 9240/24618 [00:27<00:22, 695.81 examples/s]
Running tokenizer on dataset (num_proc=320):  38%|███▊      | 9394/24618 [00:27<00:20, 740.69 examples/s]
Running tokenizer on dataset (num_proc=320):  39%|███▉      | 9548/24618 [00:27<00:19, 763.19 examples/s]
Running tokenizer on dataset (num_proc=320):  39%|███▉      | 9702/24618 [00:28<00:32, 465.45 examples/s]
Running tokenizer on dataset (num_proc=320):  40%|███▉      | 9779/24618 [00:28<00:31, 472.35 examples/s]
Running tokenizer on dataset (num_proc=320):  40%|████      | 9933/24618 [00:28<00:24, 589.06 examples/s]
Running tokenizer on dataset (num_proc=320):  41%|████      | 10010/24618 [00:28<00:29, 492.53 examples/s]
Running tokenizer on dataset (num_proc=320):  41%|████▏     | 10164/24618 [00:29<00:27, 526.31 examples/s]
Running tokenizer on dataset (num_proc=320):  42%|████▏     | 10241/24618 [00:29<00:52, 274.49 examples/s]
Running tokenizer on dataset (num_proc=320):  42%|████▏     | 10395/24618 [00:30<00:37, 380.84 examples/s]
Running tokenizer on dataset (num_proc=320):  43%|████▎     | 10472/24618 [00:30<00:45, 313.58 examples/s]
Running tokenizer on dataset (num_proc=320):  43%|████▎     | 10549/24618 [00:30<00:39, 351.91 examples/s]
Running tokenizer on dataset (num_proc=320):  43%|████▎     | 10626/24618 [00:30<00:37, 369.96 examples/s]
Running tokenizer on dataset (num_proc=320):  44%|████▍     | 10780/24618 [00:31<00:33, 414.39 examples/s]
Running tokenizer on dataset (num_proc=320):  44%|████▍     | 10857/24618 [00:31<00:49, 279.56 examples/s]
Running tokenizer on dataset (num_proc=320):  44%|████▍     | 10934/24618 [00:32<01:31, 149.79 examples/s]
Running tokenizer on dataset (num_proc=320):  45%|████▍     | 11011/24618 [00:33<01:36, 140.59 examples/s]
Running tokenizer on dataset (num_proc=320):  45%|████▌     | 11165/24618 [00:33<01:07, 200.66 examples/s]
Running tokenizer on dataset (num_proc=320):  46%|████▌     | 11319/24618 [00:34<00:57, 233.11 examples/s]
Running tokenizer on dataset (num_proc=320):  47%|████▋     | 11473/24618 [00:34<00:41, 316.47 examples/s]
Running tokenizer on dataset (num_proc=320):  47%|████▋     | 11627/24618 [00:35<00:52, 246.87 examples/s]
Running tokenizer on dataset (num_proc=320):  48%|████▊     | 11704/24618 [00:35<00:49, 262.04 examples/s]
Running tokenizer on dataset (num_proc=320):  48%|████▊     | 11781/24618 [00:35<00:45, 279.11 examples/s]
Running tokenizer on dataset (num_proc=320):  48%|████▊     | 11858/24618 [00:36<00:42, 300.34 examples/s]
Running tokenizer on dataset (num_proc=320):  49%|████▉     | 12089/24618 [00:36<00:26, 467.01 examples/s]
Running tokenizer on dataset (num_proc=320):  50%|████▉     | 12243/24618 [00:36<00:24, 513.74 examples/s]
Running tokenizer on dataset (num_proc=320):  50%|█████     | 12320/24618 [00:36<00:30, 401.26 examples/s]
Running tokenizer on dataset (num_proc=320):  50%|█████     | 12397/24618 [00:37<00:36, 338.02 examples/s]
Running tokenizer on dataset (num_proc=320):  51%|█████     | 12474/24618 [00:37<00:43, 277.35 examples/s]
Running tokenizer on dataset (num_proc=320):  51%|█████▏    | 12628/24618 [00:37<00:32, 373.67 examples/s]
Running tokenizer on dataset (num_proc=320):  52%|█████▏    | 12782/24618 [00:38<00:25, 464.25 examples/s]
Running tokenizer on dataset (num_proc=320):  52%|█████▏    | 12859/24618 [00:38<00:25, 453.42 examples/s]
Running tokenizer on dataset (num_proc=320):  53%|█████▎    | 13013/24618 [00:38<00:30, 385.00 examples/s]
Running tokenizer on dataset (num_proc=320):  54%|█████▍    | 13244/24618 [00:39<00:25, 453.25 examples/s]
Running tokenizer on dataset (num_proc=320):  54%|█████▍    | 13321/24618 [00:39<00:35, 322.16 examples/s]
Running tokenizer on dataset (num_proc=320):  55%|█████▌    | 13552/24618 [00:39<00:23, 461.48 examples/s]
Running tokenizer on dataset (num_proc=320):  55%|█████▌    | 13629/24618 [00:40<00:24, 444.71 examples/s]
Running tokenizer on dataset (num_proc=320):  56%|█████▌    | 13706/24618 [00:40<00:25, 427.42 examples/s]
Running tokenizer on dataset (num_proc=320):  56%|█████▌    | 13783/24618 [00:40<00:25, 425.38 examples/s]
Running tokenizer on dataset (num_proc=320):  56%|█████▋    | 13860/24618 [00:40<00:26, 399.57 examples/s]
Running tokenizer on dataset (num_proc=320):  57%|█████▋    | 14014/24618 [00:40<00:20, 508.99 examples/s]
Running tokenizer on dataset (num_proc=320):  57%|█████▋    | 14091/24618 [00:41<00:21, 494.54 examples/s]
Running tokenizer on dataset (num_proc=320):  58%|█████▊    | 14322/24618 [00:41<00:14, 691.49 examples/s]
Running tokenizer on dataset (num_proc=320):  59%|█████▉    | 14553/24618 [00:41<00:11, 867.67 examples/s]
Running tokenizer on dataset (num_proc=320):  60%|██████    | 14784/24618 [00:41<00:09, 1000.76 examples/s]
Running tokenizer on dataset (num_proc=320):  61%|██████▏   | 15092/24618 [00:41<00:08, 1138.50 examples/s]
Running tokenizer on dataset (num_proc=320):  62%|██████▏   | 15246/24618 [00:41<00:08, 1093.05 examples/s]
Running tokenizer on dataset (num_proc=320):  63%|██████▎   | 15477/24618 [00:42<00:08, 1040.42 examples/s]
Running tokenizer on dataset (num_proc=320):  64%|██████▍   | 15708/24618 [00:42<00:08, 1043.01 examples/s]
Running tokenizer on dataset (num_proc=320):  64%|██████▍   | 15862/24618 [00:42<00:12, 729.63 examples/s] 
Running tokenizer on dataset (num_proc=320):  65%|██████▌   | 16016/24618 [00:43<00:12, 715.97 examples/s]
Running tokenizer on dataset (num_proc=320):  66%|██████▌   | 16170/24618 [00:43<00:17, 491.67 examples/s]
Running tokenizer on dataset (num_proc=320):  66%|██████▋   | 16324/24618 [00:43<00:14, 561.04 examples/s]
Running tokenizer on dataset (num_proc=320):  67%|██████▋   | 16478/24618 [00:44<00:17, 475.56 examples/s]
Running tokenizer on dataset (num_proc=320):  68%|██████▊   | 16632/24618 [00:44<00:14, 554.16 examples/s]
Running tokenizer on dataset (num_proc=320):  68%|██████▊   | 16786/24618 [00:44<00:12, 619.65 examples/s]
Running tokenizer on dataset (num_proc=320):  68%|██████▊   | 16863/24618 [00:44<00:12, 605.06 examples/s]
Running tokenizer on dataset (num_proc=320):  69%|██████▉   | 17017/24618 [00:44<00:10, 699.08 examples/s]
Running tokenizer on dataset (num_proc=320):  70%|██████▉   | 17171/24618 [00:45<00:14, 523.33 examples/s]
Running tokenizer on dataset (num_proc=320):  71%|███████   | 17402/24618 [00:45<00:10, 702.11 examples/s]
Running tokenizer on dataset (num_proc=320):  71%|███████▏  | 17556/24618 [00:46<00:17, 406.40 examples/s]
Running tokenizer on dataset (num_proc=320):  72%|███████▏  | 17633/24618 [00:46<00:16, 413.06 examples/s]
Running tokenizer on dataset (num_proc=320):  72%|███████▏  | 17710/24618 [00:47<00:27, 249.14 examples/s]
Running tokenizer on dataset (num_proc=320):  72%|███████▏  | 17787/24618 [00:47<00:27, 250.10 examples/s]
Running tokenizer on dataset (num_proc=320):  73%|███████▎  | 17864/24618 [00:47<00:26, 253.98 examples/s]
Running tokenizer on dataset (num_proc=320):  73%|███████▎  | 18018/24618 [00:48<00:18, 366.57 examples/s]
Running tokenizer on dataset (num_proc=320):  74%|███████▎  | 18095/24618 [00:48<00:20, 311.79 examples/s]
Running tokenizer on dataset (num_proc=320):  74%|███████▍  | 18172/24618 [00:48<00:19, 330.08 examples/s]
Running tokenizer on dataset (num_proc=320):  74%|███████▍  | 18326/24618 [00:48<00:14, 436.49 examples/s]
Running tokenizer on dataset (num_proc=320):  75%|███████▌  | 18557/24618 [00:48<00:09, 657.50 examples/s]
Running tokenizer on dataset (num_proc=320):  76%|███████▌  | 18711/24618 [00:49<00:15, 379.01 examples/s]
Running tokenizer on dataset (num_proc=320):  76%|███████▋  | 18788/24618 [00:49<00:14, 393.42 examples/s]
Running tokenizer on dataset (num_proc=320):  77%|███████▋  | 18865/24618 [00:50<00:14, 406.81 examples/s]
Running tokenizer on dataset (num_proc=320):  77%|███████▋  | 19019/24618 [00:50<00:12, 448.38 examples/s]
Running tokenizer on dataset (num_proc=320):  78%|███████▊  | 19096/24618 [00:52<00:35, 157.33 examples/s]
Running tokenizer on dataset (num_proc=320):  78%|███████▊  | 19173/24618 [00:52<00:31, 171.70 examples/s]
Running tokenizer on dataset (num_proc=320):  79%|███████▊  | 19327/24618 [00:52<00:20, 254.98 examples/s]
Running tokenizer on dataset (num_proc=320):  79%|███████▉  | 19404/24618 [00:52<00:18, 286.28 examples/s]
Running tokenizer on dataset (num_proc=320):  79%|███████▉  | 19481/24618 [00:52<00:15, 321.09 examples/s]
Running tokenizer on dataset (num_proc=320):  79%|███████▉  | 19558/24618 [00:52<00:14, 357.57 examples/s]
Running tokenizer on dataset (num_proc=320):  80%|████████  | 19712/24618 [00:53<00:12, 395.26 examples/s]
Running tokenizer on dataset (num_proc=320):  80%|████████  | 19789/24618 [00:53<00:12, 399.45 examples/s]
Running tokenizer on dataset (num_proc=320):  82%|████████▏ | 20097/24618 [00:53<00:08, 554.19 examples/s]
Running tokenizer on dataset (num_proc=320):  82%|████████▏ | 20174/24618 [00:54<00:13, 321.73 examples/s]
Running tokenizer on dataset (num_proc=320):  83%|████████▎ | 20328/24618 [00:54<00:10, 425.41 examples/s]
Running tokenizer on dataset (num_proc=320):  84%|████████▎ | 20559/24618 [00:54<00:06, 604.72 examples/s]
Running tokenizer on dataset (num_proc=320):  84%|████████▍ | 20713/24618 [00:55<00:08, 436.97 examples/s]
Running tokenizer on dataset (num_proc=320):  84%|████████▍ | 20790/24618 [00:55<00:08, 426.03 examples/s]
Running tokenizer on dataset (num_proc=320):  85%|████████▌ | 20944/24618 [00:56<00:08, 449.04 examples/s]
Running tokenizer on dataset (num_proc=320):  86%|████████▌ | 21175/24618 [00:56<00:08, 399.17 examples/s]
Running tokenizer on dataset (num_proc=320):  87%|████████▋ | 21329/24618 [00:56<00:06, 501.07 examples/s]
Running tokenizer on dataset (num_proc=320):  87%|████████▋ | 21483/24618 [00:57<00:07, 441.46 examples/s]
Running tokenizer on dataset (num_proc=320):  88%|████████▊ | 21637/24618 [00:57<00:06, 478.76 examples/s]
Running tokenizer on dataset (num_proc=320):  88%|████████▊ | 21714/24618 [00:57<00:07, 405.50 examples/s]
Running tokenizer on dataset (num_proc=320):  89%|████████▉ | 21868/24618 [00:57<00:05, 507.27 examples/s]
Running tokenizer on dataset (num_proc=320):  89%|████████▉ | 21945/24618 [00:58<00:06, 431.27 examples/s]
Running tokenizer on dataset (num_proc=320):  89%|████████▉ | 22022/24618 [00:58<00:07, 359.95 examples/s]
Running tokenizer on dataset (num_proc=320):  90%|████████▉ | 22099/24618 [00:58<00:06, 371.80 examples/s]
Running tokenizer on dataset (num_proc=320):  90%|█████████ | 22176/24618 [00:59<00:06, 370.70 examples/s]
Running tokenizer on dataset (num_proc=320):  91%|█████████ | 22407/24618 [00:59<00:03, 597.06 examples/s]
Running tokenizer on dataset (num_proc=320):  92%|█████████▏| 22561/24618 [00:59<00:03, 568.00 examples/s]
Running tokenizer on dataset (num_proc=320):  92%|█████████▏| 22715/24618 [00:59<00:03, 536.64 examples/s]
Running tokenizer on dataset (num_proc=320):  93%|█████████▎| 22792/24618 [01:00<00:03, 457.51 examples/s]
Running tokenizer on dataset (num_proc=320):  93%|█████████▎| 22868/24618 [01:00<00:05, 348.73 examples/s]
Running tokenizer on dataset (num_proc=320):  93%|█████████▎| 22945/24618 [01:00<00:04, 361.55 examples/s]
Running tokenizer on dataset (num_proc=320):  94%|█████████▍| 23098/24618 [01:01<00:04, 308.79 examples/s]
Running tokenizer on dataset (num_proc=320):  95%|█████████▍| 23326/24618 [01:01<00:02, 448.26 examples/s]
Running tokenizer on dataset (num_proc=320):  96%|█████████▌| 23554/24618 [01:01<00:02, 503.78 examples/s]
Running tokenizer on dataset (num_proc=320):  96%|█████████▌| 23630/24618 [01:02<00:02, 467.40 examples/s]
Running tokenizer on dataset (num_proc=320):  96%|█████████▋| 23706/24618 [01:02<00:02, 432.12 examples/s]
Running tokenizer on dataset (num_proc=320):  97%|█████████▋| 23858/24618 [01:02<00:01, 458.44 examples/s]
Running tokenizer on dataset (num_proc=320):  97%|█████████▋| 23934/24618 [01:02<00:01, 396.87 examples/s]
Running tokenizer on dataset (num_proc=320):  98%|█████████▊| 24010/24618 [01:03<00:01, 375.34 examples/s]
Running tokenizer on dataset (num_proc=320):  98%|█████████▊| 24238/24618 [01:03<00:00, 611.21 examples/s]
Running tokenizer on dataset (num_proc=320):  99%|█████████▉| 24466/24618 [01:03<00:00, 581.91 examples/s]
Running tokenizer on dataset (num_proc=320): 100%|█████████▉| 24542/24618 [01:03<00:00, 518.46 examples/s]
Running tokenizer on dataset (num_proc=320): 100%|██████████| 24618/24618 [01:04<00:00, 445.32 examples/s]
Running tokenizer on dataset (num_proc=320): 100%|██████████| 24618/24618 [01:04<00:00, 382.83 examples/s]
training example:
input_ids:
[151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 151652, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151653, 198, 2610, 525, 264, 16435, 3383, 6203, 11, 358, 686, 3410, 498, 448, 264, 1550, 11591, 7600, 11, 264, 36190, 448, 1181, 12159, 39700, 4916, 382, 11976, 11591, 7600, 25, 18995, 279, 14332, 906, 323, 470, 311, 279, 2114, 4171, 382, 85369, 4916, 25, 5212, 5954, 3792, 6382, 469, 8144, 31197, 17, 469, 8144, 31197, 17, 39125, 14, 27171, 58189, 788, 11993, 20, 19, 15, 13, 15, 11, 220, 16, 17, 18, 17, 13, 20, 11583, 330, 2016, 6207, 788, 11993, 20, 19, 15, 13, 15, 11, 220, 17, 16, 22, 24, 13, 20, 11583, 330, 22808, 26966, 788, 11993, 16, 17, 19, 13, 15, 11, 220, 16, 24, 16, 13, 15, 11583, 330, 37, 1715, 44, 6666, 3298, 788, 11993, 18, 22, 16, 13, 15, 11, 220, 16, 24, 16, 13, 15, 11583, 330, 57, 12, 788, 11993, 21, 15, 24, 13, 20, 11, 220, 16, 24, 16, 13, 15, 11583, 330, 57, 80479, 11993, 23, 19, 15, 13, 20, 11, 220, 16, 24, 16, 13, 15, 11583, 330, 2507, 2923, 9021, 374, 1007, 788, 11993, 18, 23, 17, 13, 15, 11, 220, 16, 24, 17, 22, 13, 15, 11583, 330, 3543, 5128, 1007, 788, 11993, 20, 19, 15, 13, 15, 11, 220, 16, 24, 17, 22, 13, 15, 11583, 330, 3707, 6249, 788, 11993, 21, 24, 23, 13, 15, 11, 220, 16, 24, 17, 22, 13, 15, 9940, 630, 5501, 6923, 279, 3347, 11591, 3381, 323, 1917, 369, 279, 1790, 3019, 13, 151645, 198, 151644, 77091, 198, 24187, 11591, 3381, 25, 8445, 279, 1182, 3137, 311, 4869, 279, 14332, 906, 323, 470, 311, 279, 2114, 4171, 624, 1311, 25, 5212, 1311, 1819, 3252, 70839, 3895, 9207, 151645]
inputs:
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
<|vision_start|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|image_pad|><|vision_end|>
You are a GUI task expert, I will provide you with a high-level instruction, a screenshot with its corresponding accessibility tree.

High-level instruction: Exit the Camera app and return to the home screen.

Accessibility tree: {"android.view.View com.android.camera2 com.android.camera2:id/preview_overlay": "(540.0, 1232.5)", "Shutter": "(540.0, 2179.5)", "MODE LIST": "(124.0, 191.0)", "FILMSTRIP": "(371.0, 191.0)", "Z-": "(609.5, 191.0)", "Z+": "(840.5, 191.0)", "Countdown timer is off": "(382.0, 1927.0)", "Grid lines off": "(540.0, 1927.0)", "Back camera": "(698.0, 1927.0)"}

Please generate the low-level thought and action for the next step.<|im_end|>
<|im_start|>assistant
Low-level thought: Press the back button to exit the Camera app and return to the home screen.
action: {"action_type":"navigate_back"}<|im_end|>
label_ids:
[-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 24187, 11591, 3381, 25, 8445, 279, 1182, 3137, 311, 4869, 279, 14332, 906, 323, 470, 311, 279, 2114, 4171, 624, 1311, 25, 5212, 1311, 1819, 3252, 70839, 3895, 9207, 151645]
labels:
Low-level thought: Press the back button to exit the Camera app and return to the home screen.
action: {"action_type":"navigate_back"}<|im_end|>
[INFO|configuration_utils.py:670] 2024-12-05 13:40:16,953 >> loading configuration file /nas/shared/NLP_A100/wuzhenyu/LLMs/Qwen2-VL-7B-Instruct/config.json
[WARNING|modeling_rope_utils.py:379] 2024-12-05 13:40:16,953 >> Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}
[INFO|configuration_utils.py:739] 2024-12-05 13:40:16,954 >> Model config Qwen2VLConfig {
  "_name_or_path": "/nas/shared/NLP_A100/wuzhenyu/LLMs/Qwen2-VL-7B-Instruct",
  "architectures": [
    "Qwen2VLForConditionalGeneration"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "hidden_act": "silu",
  "hidden_size": 3584,
  "image_token_id": 151655,
  "initializer_range": 0.02,
  "intermediate_size": 18944,
  "max_position_embeddings": 32768,
  "max_window_layers": 28,
  "model_type": "qwen2_vl",
  "num_attention_heads": 28,
  "num_hidden_layers": 28,
  "num_key_value_heads": 4,
  "rms_norm_eps": 1e-06,
  "rope_scaling": {
    "mrope_section": [
      16,
      24,
      24
    ],
    "rope_type": "default",
    "type": "default"
  },
  "rope_theta": 1000000.0,
  "sliding_window": 32768,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.45.0",
  "use_cache": true,
  "use_sliding_window": false,
  "video_token_id": 151656,
  "vision_config": {
    "in_chans": 3,
    "model_type": "qwen2_vl",
    "spatial_patch_size": 14
  },
  "vision_end_token_id": 151653,
  "vision_start_token_id": 151652,
  "vision_token_id": 151654,
  "vocab_size": 152064
}

[INFO|modeling_utils.py:3723] 2024-12-05 13:40:17,192 >> loading weights file /nas/shared/NLP_A100/wuzhenyu/LLMs/Qwen2-VL-7B-Instruct/model.safetensors.index.json
[INFO|modeling_utils.py:1622] 2024-12-05 13:40:17,294 >> Instantiating Qwen2VLForConditionalGeneration model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:1099] 2024-12-05 13:40:17,295 >> Generate config GenerationConfig {
  "bos_token_id": 151643,
  "eos_token_id": 151645
}

[WARNING|logging.py:328] 2024-12-05 13:40:17,317 >> `Qwen2VLRotaryEmbedding` can now be fully parameterized by passing the model config through the `config` argument. All other arguments will be removed in v4.46

Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}
`Qwen2VLRotaryEmbedding` can now be fully parameterized by passing the model config through the `config` argument. All other arguments will be removed in v4.46
`Qwen2VLRotaryEmbedding` can now be fully parameterized by passing the model config through the `config` argument. All other arguments will be removed in v4.46
`Qwen2VLRotaryEmbedding` can now be fully parameterized by passing the model config through the `config` argument. All other arguments will be removed in v4.46
`Qwen2VLRotaryEmbedding` can now be fully parameterized by passing the model config through the `config` argument. All other arguments will be removed in v4.46
`Qwen2VLRotaryEmbedding` can now be fully parameterized by passing the model config through the `config` argument. All other arguments will be removed in v4.46
`Qwen2VLRotaryEmbedding` can now be fully parameterized by passing the model config through the `config` argument. All other arguments will be removed in v4.46
`Qwen2VLRotaryEmbedding` can now be fully parameterized by passing the model config through the `config` argument. All other arguments will be removed in v4.46

Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:25<01:42, 25.61s/it]
Loading checkpoint shards:  20%|██        | 1/5 [00:25<01:41, 25.41s/it]
Loading checkpoint shards:  20%|██        | 1/5 [00:25<01:41, 25.41s/it]
Loading checkpoint shards:  20%|██        | 1/5 [00:25<01:41, 25.40s/it]
Loading checkpoint shards:  20%|██        | 1/5 [00:25<01:41, 25.41s/it]
Loading checkpoint shards:  20%|██        | 1/5 [00:25<01:41, 25.41s/it]
Loading checkpoint shards:  20%|██        | 1/5 [00:25<01:41, 25.42s/it]
Loading checkpoint shards:  20%|██        | 1/5 [00:25<01:41, 25.43s/it]
Loading checkpoint shards:  40%|████      | 2/5 [00:54<01:22, 27.65s/it]
Loading checkpoint shards:  40%|████      | 2/5 [00:55<01:24, 28.02s/it]
Loading checkpoint shards:  40%|████      | 2/5 [00:55<01:24, 28.02s/it]
Loading checkpoint shards:  40%|████      | 2/5 [00:55<01:24, 28.02s/it]
Loading checkpoint shards:  40%|████      | 2/5 [00:55<01:24, 28.03s/it]
Loading checkpoint shards:  40%|████      | 2/5 [00:55<01:24, 28.03s/it]
Loading checkpoint shards:  40%|████      | 2/5 [00:55<01:25, 28.40s/it]
Loading checkpoint shards:  40%|████      | 2/5 [00:55<01:24, 28.03s/it]
Loading checkpoint shards:  60%|██████    | 3/5 [01:20<00:53, 26.81s/it]
Loading checkpoint shards:  60%|██████    | 3/5 [01:20<00:53, 26.76s/it]
Loading checkpoint shards:  60%|██████    | 3/5 [01:20<00:53, 26.76s/it]
Loading checkpoint shards:  60%|██████    | 3/5 [01:20<00:53, 26.76s/it]
Loading checkpoint shards:  60%|██████    | 3/5 [01:20<00:53, 26.76s/it]
Loading checkpoint shards:  60%|██████    | 3/5 [01:20<00:53, 26.85s/it]
Loading checkpoint shards:  60%|██████    | 3/5 [01:20<00:53, 26.77s/it]
Loading checkpoint shards:  60%|██████    | 3/5 [01:20<00:53, 26.77s/it]
Loading checkpoint shards:  80%|████████  | 4/5 [01:46<00:26, 26.28s/it]
Loading checkpoint shards:  80%|████████  | 4/5 [01:46<00:26, 26.45s/it]
Loading checkpoint shards:  80%|████████  | 4/5 [01:46<00:26, 26.45s/it]
Loading checkpoint shards:  80%|████████  | 4/5 [01:46<00:26, 26.50s/it]
Loading checkpoint shards:  80%|████████  | 4/5 [01:46<00:26, 26.45s/it]
Loading checkpoint shards:  80%|████████  | 4/5 [01:46<00:26, 26.45s/it]
Loading checkpoint shards:  80%|████████  | 4/5 [01:46<00:26, 26.45s/it]
Loading checkpoint shards:  80%|████████  | 4/5 [01:47<00:26, 26.61s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [01:53<00:00, 19.31s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [01:53<00:00, 22.63s/it]
12/05/2024 13:42:11 - INFO - llamafactory.model.model_utils.checkpointing - Gradient checkpointing enabled.
12/05/2024 13:42:11 - INFO - llamafactory.model.model_utils.attention - Using torch SDPA for faster training and inference.
12/05/2024 13:42:11 - INFO - llamafactory.model.adapter - Upcasting trainable params to float32.
12/05/2024 13:42:11 - INFO - llamafactory.model.adapter - Fine-tuning method: Full

Loading checkpoint shards: 100%|██████████| 5/5 [01:53<00:00, 19.38s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [01:53<00:00, 22.65s/it]

Loading checkpoint shards: 100%|██████████| 5/5 [01:53<00:00, 19.39s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [01:53<00:00, 22.65s/it]

Loading checkpoint shards: 100%|██████████| 5/5 [01:53<00:00, 19.35s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [01:53<00:00, 22.65s/it]

Loading checkpoint shards: 100%|██████████| 5/5 [01:53<00:00, 19.35s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [01:53<00:00, 22.65s/it]

Loading checkpoint shards: 100%|██████████| 5/5 [01:53<00:00, 19.35s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [01:53<00:00, 22.65s/it]

Loading checkpoint shards: 100%|██████████| 5/5 [01:53<00:00, 19.35s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [01:53<00:00, 22.65s/it]

Loading checkpoint shards: 100%|██████████| 5/5 [01:53<00:00, 19.45s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [01:53<00:00, 22.79s/it]
[INFO|modeling_utils.py:4568] 2024-12-05 13:42:11,359 >> All model checkpoint weights were used when initializing Qwen2VLForConditionalGeneration.

[INFO|modeling_utils.py:4576] 2024-12-05 13:42:11,359 >> All the weights of Qwen2VLForConditionalGeneration were initialized from the model checkpoint at /nas/shared/NLP_A100/wuzhenyu/LLMs/Qwen2-VL-7B-Instruct.
If your task is similar to the task the model of the checkpoint was trained on, you can already use Qwen2VLForConditionalGeneration for predictions without further training.
12/05/2024 13:42:11 - INFO - llamafactory.model.model_utils.checkpointing - Gradient checkpointing enabled.
12/05/2024 13:42:11 - INFO - llamafactory.model.model_utils.attention - Using torch SDPA for faster training and inference.
12/05/2024 13:42:11 - INFO - llamafactory.model.adapter - Upcasting trainable params to float32.
12/05/2024 13:42:11 - INFO - llamafactory.model.adapter - Fine-tuning method: Full
[INFO|configuration_utils.py:1052] 2024-12-05 13:42:11,380 >> loading configuration file /nas/shared/NLP_A100/wuzhenyu/LLMs/Qwen2-VL-7B-Instruct/generation_config.json
[INFO|configuration_utils.py:1099] 2024-12-05 13:42:11,380 >> Generate config GenerationConfig {
  "bos_token_id": 151643,
  "do_sample": true,
  "eos_token_id": [
    151645,
    151643
  ],
  "pad_token_id": 151643,
  "temperature": 0.01,
  "top_k": 1,
  "top_p": 0.001
}

12/05/2024 13:42:11 - INFO - llamafactory.model.model_utils.checkpointing - Gradient checkpointing enabled.
12/05/2024 13:42:11 - INFO - llamafactory.model.model_utils.attention - Using torch SDPA for faster training and inference.
12/05/2024 13:42:11 - INFO - llamafactory.model.adapter - Upcasting trainable params to float32.
12/05/2024 13:42:11 - INFO - llamafactory.model.adapter - Fine-tuning method: Full
12/05/2024 13:42:11 - INFO - llamafactory.model.model_utils.checkpointing - Gradient checkpointing enabled.
12/05/2024 13:42:11 - INFO - llamafactory.model.model_utils.attention - Using torch SDPA for faster training and inference.
12/05/2024 13:42:11 - INFO - llamafactory.model.adapter - Upcasting trainable params to float32.
12/05/2024 13:42:11 - INFO - llamafactory.model.adapter - Fine-tuning method: Full
12/05/2024 13:42:11 - INFO - llamafactory.model.model_utils.checkpointing - Gradient checkpointing enabled.
12/05/2024 13:42:11 - INFO - llamafactory.model.model_utils.attention - Using torch SDPA for faster training and inference.
12/05/2024 13:42:11 - INFO - llamafactory.model.model_utils.checkpointing - Gradient checkpointing enabled.
12/05/2024 13:42:11 - INFO - llamafactory.model.adapter - Upcasting trainable params to float32.
12/05/2024 13:42:11 - INFO - llamafactory.model.adapter - Fine-tuning method: Full
12/05/2024 13:42:11 - INFO - llamafactory.model.model_utils.attention - Using torch SDPA for faster training and inference.
12/05/2024 13:42:11 - INFO - llamafactory.model.model_utils.checkpointing - Gradient checkpointing enabled.
12/05/2024 13:42:11 - INFO - llamafactory.model.adapter - Upcasting trainable params to float32.
12/05/2024 13:42:11 - INFO - llamafactory.model.adapter - Fine-tuning method: Full
12/05/2024 13:42:11 - INFO - llamafactory.model.model_utils.attention - Using torch SDPA for faster training and inference.
12/05/2024 13:42:11 - INFO - llamafactory.model.adapter - Upcasting trainable params to float32.
12/05/2024 13:42:11 - INFO - llamafactory.model.adapter - Fine-tuning method: Full
12/05/2024 13:42:11 - INFO - llamafactory.model.model_utils.checkpointing - Gradient checkpointing enabled.
12/05/2024 13:42:11 - INFO - llamafactory.model.model_utils.attention - Using torch SDPA for faster training and inference.
12/05/2024 13:42:11 - INFO - llamafactory.model.adapter - Upcasting trainable params to float32.
12/05/2024 13:42:11 - INFO - llamafactory.model.adapter - Fine-tuning method: Full
12/05/2024 13:42:11 - INFO - llamafactory.model.loader - trainable params: 7,615,616,512 || all params: 8,291,375,616 || trainable%: 91.8499
12/05/2024 13:42:11 - INFO - llamafactory.model.loader - trainable params: 7,615,616,512 || all params: 8,291,375,616 || trainable%: 91.8499
12/05/2024 13:42:11 - INFO - llamafactory.model.loader - trainable params: 7,615,616,512 || all params: 8,291,375,616 || trainable%: 91.8499
12/05/2024 13:42:11 - INFO - llamafactory.model.loader - trainable params: 7,615,616,512 || all params: 8,291,375,616 || trainable%: 91.8499
12/05/2024 13:42:11 - INFO - llamafactory.model.loader - trainable params: 7,615,616,512 || all params: 8,291,375,616 || trainable%: 91.8499
12/05/2024 13:42:11 - INFO - llamafactory.model.loader - trainable params: 7,615,616,512 || all params: 8,291,375,616 || trainable%: 91.8499
12/05/2024 13:42:11 - INFO - llamafactory.model.loader - trainable params: 7,615,616,512 || all params: 8,291,375,616 || trainable%: 91.8499
12/05/2024 13:42:11 - INFO - llamafactory.model.loader - trainable params: 7,615,616,512 || all params: 8,291,375,616 || trainable%: 91.8499
Detected kernel version 4.19.91, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
[INFO|trainer.py:667] 2024-12-05 13:42:11,800 >> Using auto half precision backend
[2024-12-05 13:42:15,269] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed info: version=0.15.4, git-hash=unknown, git-branch=unknown
[2024-12-05 13:42:15,269] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Using non-device net plugin version 0
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Using non-device net plugin version 0
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Using non-device net plugin version 0
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Using non-device net plugin version 0
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Using network IB
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Using network IB
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Using non-device net plugin version 0
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Using non-device net plugin version 0
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Using network IB
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Using network IB
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Using network IB
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Using network IB
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Using non-device net plugin version 0
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Using network IB
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Using non-device net plugin version 0
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Using network IB
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO bootstrapSplit: comm 0xa6a0b1d0 parent 0xa4c31ca0 rank 1 nranks 8 color -934961569 key 1 prev 0 next 2 - DONE
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO bootstrapSplit: comm 0xa6f91730 parent 0xa547c690 rank 7 nranks 8 color -934961569 key 7 prev 6 next 0 - DONE
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO bootstrapSplit: comm 0xa79ebd30 parent 0xa4b2bc20 rank 5 nranks 8 color -934961569 key 5 prev 4 next 6 - DONE
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO bootstrapSplit: comm 0xa6c5e6a0 parent 0xaf287310 rank 4 nranks 8 color -934961569 key 4 prev 3 next 5 - DONE
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO bootstrapSplit: comm 0xa7df0940 parent 0xa55b37a0 rank 6 nranks 8 color -934961569 key 6 prev 5 next 7 - DONE
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO bootstrapSplit: comm 0xc33c3ce0 parent 0xc0177130 rank 0 nranks 8 color -934961569 key 0 prev 7 next 1 - DONE
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO comm 0xa6f91730 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId 80 commId 0x1900e06671419955 - Init START
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO comm 0xa79ebd30 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId 60 commId 0x1900e06671419955 - Init START
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO comm 0xa7df0940 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId 70 commId 0x1900e06671419955 - Init START
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO comm 0xa6a0b1d0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 20 commId 0x1900e06671419955 - Init START
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO comm 0xa6c5e6a0 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId 50 commId 0x1900e06671419955 - Init START
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO bootstrapSplit: comm 0xa5e6c900 parent 0xa75046a0 rank 3 nranks 8 color -934961569 key 3 prev 2 next 4 - DONE
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO comm 0xc33c3ce0 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 10 commId 0x1900e06671419955 - Init START
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO comm 0xa5e6c900 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId 40 commId 0x1900e06671419955 - Init START
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO bootstrapSplit: comm 0xa57b0e70 parent 0xa6df0650 rank 2 nranks 8 color -934961569 key 2 prev 1 next 3 - DONE
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO comm 0xa57b0e70 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId 30 commId 0x1900e06671419955 - Init START
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ffffffff,ffffffff
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO NVLS multicast support is not available on dev 1
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO NVLS multicast support is not available on dev 7
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO NVLS multicast support is not available on dev 6
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO NVLS multicast support is not available on dev 4
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Setting affinity for GPU 3 to ffff,ffffffff,ffffffff
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO NVLS multicast support is not available on dev 3
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO NVLS multicast support is not available on dev 5
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Setting affinity for GPU 0 to ffff,ffffffff,ffffffff
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO NVLS multicast support is not available on dev 0
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Setting affinity for GPU 2 to ffff,ffffffff,ffffffff
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO NVLS multicast support is not available on dev 2
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO comm 0xa6a0b1d0 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO comm 0xc33c3ce0 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO comm 0xa6f91730 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO comm 0xa7df0940 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO comm 0xa6c5e6a0 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 00/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO comm 0xa79ebd30 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 01/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 02/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO P2P Chunksize set to 524288
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 03/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO comm 0xa5e6c900 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 04/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO P2P Chunksize set to 524288
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO comm 0xa57b0e70 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 05/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 06/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO P2P Chunksize set to 524288
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO P2P Chunksize set to 524288
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 07/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO P2P Chunksize set to 524288
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 08/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 09/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO P2P Chunksize set to 524288
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO P2P Chunksize set to 524288
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 10/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 11/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 12/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 13/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 14/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 15/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 16/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 17/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 18/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 19/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 20/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 21/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 22/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 23/24 :    0   1   2   3   4   5   6   7
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO P2P Chunksize set to 524288
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 00/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 01/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 02/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 03/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 04/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 04/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 04/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 04/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 05/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 05/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 05/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 05/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 05/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 05/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 06/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 06/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 06/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 06/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 06/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 06/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 07/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 07/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 07/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 07/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 07/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 08/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 08/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 08/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 08/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 09/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 09/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 09/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 09/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 09/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 10/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 10/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 10/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 10/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 10/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 10/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 11/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 11/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 11/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 11/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 11/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 12/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 12/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 12/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 12/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 13/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 13/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 13/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 13/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 13/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 13/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 14/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 14/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 14/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 14/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 14/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 14/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 15/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 15/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 15/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 15/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 15/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 16/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 16/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 16/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 16/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 16/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 16/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 17/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 17/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 17/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 17/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 17/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 17/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 18/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 18/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 18/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 18/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 18/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 18/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 19/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 19/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 19/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 19/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 19/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 19/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 20/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 20/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 20/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 20/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 20/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 20/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 21/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 21/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 21/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 21/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 21/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 21/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 22/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 22/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 22/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 22/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 22/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 22/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 23/0 : 1[1] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 23/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 23/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 23/0 : 3[3] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 23/0 : 4[4] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 23/0 : 5[5] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Connected all rings
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Connected all rings
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Connected all rings
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Connected all rings
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Connected all rings
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 05/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 11/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 13/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 16/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 17/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 18/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Connected all rings
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 00/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Connected all rings
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Connected all rings
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 19/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 01/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 20/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 02/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 21/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 03/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 22/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 04/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Channel 23/0 : 2[2] -> 1[1] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 05/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 06/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 07/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 08/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 09/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 10/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 00/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 11/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 01/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 12/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 02/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 13/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 03/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 14/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 04/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 15/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 05/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 16/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 06/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 17/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 07/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 18/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 08/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 19/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 09/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 20/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 10/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 21/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 11/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 22/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 12/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Channel 23/0 : 7[7] -> 6[6] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 13/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 00/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 14/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 01/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 15/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 01/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 02/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 16/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 03/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 17/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 03/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 04/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 18/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 04/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 05/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 19/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 05/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 06/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 20/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 06/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 07/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 21/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 07/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 08/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 22/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 08/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 09/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Channel 23/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 09/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 10/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 10/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 11/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 11/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 12/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 12/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 13/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 13/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 14/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 14/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 15/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 15/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 16/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 16/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 17/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 17/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 18/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 18/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 19/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 19/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 20/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 20/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 21/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 21/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 22/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 22/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Channel 23/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Channel 23/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO Connected all trees
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO Connected all trees
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO Connected all trees
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO Connected all trees
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO Connected all trees
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO Connected all trees
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO Connected all trees
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO Connected all trees
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
dlc1o8747bm15inu-master-0:73:3059 [2] NCCL INFO comm 0xa57b0e70 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId 30 commId 0x1900e06671419955 - Init COMPLETE
dlc1o8747bm15inu-master-0:78:3060 [7] NCCL INFO comm 0xa6f91730 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId 80 commId 0x1900e06671419955 - Init COMPLETE
dlc1o8747bm15inu-master-0:77:3061 [6] NCCL INFO comm 0xa7df0940 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId 70 commId 0x1900e06671419955 - Init COMPLETE
dlc1o8747bm15inu-master-0:76:3058 [5] NCCL INFO comm 0xa79ebd30 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId 60 commId 0x1900e06671419955 - Init COMPLETE
dlc1o8747bm15inu-master-0:75:3056 [4] NCCL INFO comm 0xa6c5e6a0 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId 50 commId 0x1900e06671419955 - Init COMPLETE
dlc1o8747bm15inu-master-0:74:3057 [3] NCCL INFO comm 0xa5e6c900 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId 40 commId 0x1900e06671419955 - Init COMPLETE
dlc1o8747bm15inu-master-0:71:3054 [0] NCCL INFO comm 0xc33c3ce0 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 10 commId 0x1900e06671419955 - Init COMPLETE
dlc1o8747bm15inu-master-0:72:3055 [1] NCCL INFO comm 0xa6a0b1d0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 20 commId 0x1900e06671419955 - Init COMPLETE
[2024-12-05 13:42:17,406] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
Using /root/.cache/torch_extensions/py311_cu121 as PyTorch extensions root...Using /root/.cache/torch_extensions/py311_cu121 as PyTorch extensions root...Using /root/.cache/torch_extensions/py311_cu121 as PyTorch extensions root...Using /root/.cache/torch_extensions/py311_cu121 as PyTorch extensions root...



Using /root/.cache/torch_extensions/py311_cu121 as PyTorch extensions root...
Creating extension directory /root/.cache/torch_extensions/py311_cu121/fused_adam...Creating extension directory /root/.cache/torch_extensions/py311_cu121/fused_adam...Creating extension directory /root/.cache/torch_extensions/py311_cu121/fused_adam...


Creating extension directory /root/.cache/torch_extensions/py311_cu121/fused_adam...
Creating extension directory /root/.cache/torch_extensions/py311_cu121/fused_adam...
Using /root/.cache/torch_extensions/py311_cu121 as PyTorch extensions root...Using /root/.cache/torch_extensions/py311_cu121 as PyTorch extensions root...

Creating extension directory /root/.cache/torch_extensions/py311_cu121/fused_adam...Creating extension directory /root/.cache/torch_extensions/py311_cu121/fused_adam...

Using /root/.cache/torch_extensions/py311_cu121 as PyTorch extensions root...
Creating extension directory /root/.cache/torch_extensions/py311_cu121/fused_adam...
Detected CUDA files, patching ldflags
Emitting ninja build file /root/.cache/torch_extensions/py311_cu121/fused_adam/build.ninja...
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/torch/utils/cpp_extension.py:1965: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
Building extension module fused_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] c++ -MMD -MF fused_adam_frontend.o.d -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/deepspeed/ops/csrc/includes -I/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/deepspeed/ops/csrc/adam -isystem /cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/torch/include -isystem /cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/torch/include/TH -isystem /cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DBF16_AVAILABLE -c /cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/deepspeed/ops/csrc/adam/fused_adam_frontend.cpp -o fused_adam_frontend.o 
[2/3] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output multi_tensor_adam.cuda.o.d -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/deepspeed/ops/csrc/includes -I/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/deepspeed/ops/csrc/adam -isystem /cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/torch/include -isystem /cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/torch/include/TH -isystem /cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_80,code=sm_80 --compiler-options '-fPIC' -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -lineinfo --use_fast_math -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_80,code=compute_80 -DBF16_AVAILABLE -U__CUDA_NO_BFLOAT16_OPERATORS__ -U__CUDA_NO_BFLOAT162_OPERATORS__ -U__CUDA_NO_BFLOAT16_CONVERSIONS__ -std=c++17 -c /cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/deepspeed/ops/csrc/adam/multi_tensor_adam.cu -o multi_tensor_adam.cuda.o 
[3/3] c++ fused_adam_frontend.o multi_tensor_adam.cuda.o -shared -L/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/usr/local/cuda/lib64 -lcudart -o fused_adam.so
Loading extension module fused_adam...
Time to load fused_adam op: 34.222880601882935 seconds
[2024-12-05 13:42:51,649] [INFO] [logging.py:128:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer
[2024-12-05 13:42:51,649] [INFO] [logging.py:128:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
Loading extension module fused_adam...Loading extension module fused_adam...

Loading extension module fused_adam...Loading extension module fused_adam...

Time to load fused_adam op: 34.243221044540405 secondsTime to load fused_adam op: 34.24309253692627 seconds

Loading extension module fused_adam...
Time to load fused_adam op: 34.24324321746826 seconds
Time to load fused_adam op: 34.243529319763184 seconds
Loading extension module fused_adam...
Time to load fused_adam op: 34.243879318237305 seconds
Time to load fused_adam op: 34.244356632232666 seconds
[2024-12-05 13:42:51,666] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Basic Optimizer = FusedAdam
[2024-12-05 13:42:51,666] [INFO] [utils.py:59:is_zero_supported_optimizer] Checking ZeRO support for optimizer=FusedAdam type=<class 'deepspeed.ops.adam.fused_adam.FusedAdam'>
[2024-12-05 13:42:51,666] [INFO] [logging.py:128:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 1 optimizer
[2024-12-05 13:42:51,666] [INFO] [stage_1_and_2.py:149:__init__] Reduce bucket size 1000000000
[2024-12-05 13:42:51,666] [INFO] [stage_1_and_2.py:150:__init__] Allgather bucket size 1000000000
[2024-12-05 13:42:51,666] [INFO] [stage_1_and_2.py:151:__init__] CPU Offload: False
[2024-12-05 13:42:51,666] [INFO] [stage_1_and_2.py:152:__init__] Round robin gradient partitioning: False
Loading extension module fused_adam...
Time to load fused_adam op: 34.29689931869507 seconds
[2024-12-05 13:43:03,153] [INFO] [utils.py:781:see_memory_usage] Before initializing optimizer states
[2024-12-05 13:43:03,154] [INFO] [utils.py:782:see_memory_usage] MA 18.99 GB         Max_MA 20.77 GB         CA 20.9 GB         Max_CA 21 GB 
[2024-12-05 13:43:03,154] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 133.14 GB, percent = 13.3%
[2024-12-05 13:43:03,464] [INFO] [utils.py:781:see_memory_usage] After initializing optimizer states
[2024-12-05 13:43:03,465] [INFO] [utils.py:782:see_memory_usage] MA 18.99 GB         Max_MA 22.54 GB         CA 24.44 GB         Max_CA 24 GB 
[2024-12-05 13:43:03,465] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 133.14 GB, percent = 13.3%
[2024-12-05 13:43:03,465] [INFO] [stage_1_and_2.py:544:__init__] optimizer state initialized
[2024-12-05 13:43:03,760] [INFO] [utils.py:781:see_memory_usage] After initializing ZeRO optimizer
[2024-12-05 13:43:03,761] [INFO] [utils.py:782:see_memory_usage] MA 18.99 GB         Max_MA 18.99 GB         CA 24.44 GB         Max_CA 24 GB 
[2024-12-05 13:43:03,761] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 118.96 GB, percent = 11.9%
[2024-12-05 13:43:03,763] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Final Optimizer = DeepSpeedZeroOptimizer
[2024-12-05 13:43:03,763] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed using client callable to create LR scheduler
[2024-12-05 13:43:03,763] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed LR Scheduler = <torch.optim.lr_scheduler.LambdaLR object at 0x7f5aa8380050>
[2024-12-05 13:43:03,763] [INFO] [logging.py:128:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0], mom=[[0.9, 0.999]]
[2024-12-05 13:43:03,765] [INFO] [config.py:999:print] DeepSpeedEngine configuration:
[2024-12-05 13:43:03,765] [INFO] [config.py:1003:print]   activation_checkpointing_config  {
    "partition_activations": false, 
    "contiguous_memory_optimization": false, 
    "cpu_checkpointing": false, 
    "number_checkpoints": null, 
    "synchronize_checkpoint_boundary": false, 
    "profile": false
}
[2024-12-05 13:43:03,765] [INFO] [config.py:1003:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True, 'use_gds': False}
[2024-12-05 13:43:03,765] [INFO] [config.py:1003:print]   amp_enabled .................. False
[2024-12-05 13:43:03,765] [INFO] [config.py:1003:print]   amp_params ................... False
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   autotuning_config ............ {
    "enabled": false, 
    "start_step": null, 
    "end_step": null, 
    "metric_path": null, 
    "arg_mappings": null, 
    "metric": "throughput", 
    "model_info": null, 
    "results_dir": "autotuning_results", 
    "exps_dir": "autotuning_exps", 
    "overwrite": true, 
    "fast": true, 
    "start_profile_step": 3, 
    "end_profile_step": 5, 
    "tuner_type": "gridsearch", 
    "tuner_early_stopping": 5, 
    "tuner_num_trials": 50, 
    "model_info_path": null, 
    "mp_size": 1, 
    "max_train_batch_size": null, 
    "min_train_batch_size": 1, 
    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
    "min_train_micro_batch_size_per_gpu": 1, 
    "num_tuning_micro_batch_sizes": 3
}
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   bfloat16_enabled ............. True
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   bfloat16_immediate_grad_update  False
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   checkpoint_parallel_write_pipeline  False
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   checkpoint_tag_validation_enabled  True
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   checkpoint_tag_validation_fail  False
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7f5abc8f2d90>
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   communication_data_type ...... None
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   curriculum_enabled_legacy .... False
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   curriculum_params_legacy ..... False
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   data_efficiency_enabled ...... False
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   dataloader_drop_last ......... False
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   disable_allgather ............ False
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   dump_state ................... False
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   dynamic_loss_scale_args ...... None
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   eigenvalue_enabled ........... False
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   eigenvalue_gas_boundary_resolution  1
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   eigenvalue_layer_num ......... 0
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   eigenvalue_max_iter .......... 100
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   eigenvalue_stability ......... 1e-06
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   eigenvalue_tol ............... 0.01
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   eigenvalue_verbose ........... False
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   elasticity_enabled ........... False
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   flops_profiler_config ........ {
    "enabled": false, 
    "recompute_fwd_factor": 0.0, 
    "profile_step": 1, 
    "module_depth": -1, 
    "top_modules": 1, 
    "detailed": true, 
    "output_file": null
}
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   fp16_auto_cast ............... None
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   fp16_enabled ................. False
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   fp16_master_weights_and_gradients  False
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   global_rank .................. 0
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   grad_accum_dtype ............. None
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   gradient_accumulation_steps .. 16
[2024-12-05 13:43:03,766] [INFO] [config.py:1003:print]   gradient_clipping ............ 1.0
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   gradient_predivide_factor .... 1.0
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   graph_harvesting ............. False
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   initial_dynamic_scale ........ 1
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   load_universal_checkpoint .... False
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   loss_scale ................... 1.0
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   memory_breakdown ............. False
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   mics_hierarchial_params_gather  False
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   mics_shard_size .............. -1
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName')
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   nebula_config ................ {
    "enabled": false, 
    "persistent_storage_path": null, 
    "persistent_time_interval": 100, 
    "num_of_version_in_retention": 2, 
    "enable_nebula_load": true, 
    "load_path": null
}
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   optimizer_legacy_fusion ...... False
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   optimizer_name ............... adamw
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   optimizer_params ............. {'lr': 1e-06, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.001}
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   pld_enabled .................. False
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   pld_params ................... False
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   prescale_gradients ........... False
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   scheduler_name ............... None
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   scheduler_params ............. None
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   seq_parallel_communication_data_type  torch.float32
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   sparse_attention ............. None
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   sparse_gradients_enabled ..... False
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   steps_per_print .............. inf
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   timers_config ................ enabled=True synchronized=True
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   train_batch_size ............. 128
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   train_micro_batch_size_per_gpu  1
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   use_data_before_expert_parallel_  False
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   use_node_local_storage ....... False
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   wall_clock_breakdown ......... True
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   weight_quantization_config ... None
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   world_size ................... 8
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   zero_allow_untested_optimizer  False
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   zero_config .................. stage=1 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=1000000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=1000000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=False use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   zero_enabled ................. True
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   zero_force_ds_cpu_optimizer .. True
[2024-12-05 13:43:03,767] [INFO] [config.py:1003:print]   zero_optimization_stage ...... 1
[2024-12-05 13:43:03,768] [INFO] [config.py:989:print_user_config]   json = {
    "zero_optimization": {
        "stage": 1, 
        "allgather_partitions": true, 
        "allgather_bucket_size": 1.000000e+09, 
        "overlap_comm": true, 
        "reduce_scatter": true, 
        "reduce_bucket_size": 1.000000e+09, 
        "contiguous_gradients": true
    }, 
    "fp16": {
        "enabled": false, 
        "auto_cast": true, 
        "loss_scale": 0, 
        "initial_scale_power": 32, 
        "loss_scale_window": 1000, 
        "hysteresis": 2, 
        "min_loss_scale": 1
    }, 
    "bf16": {
        "enabled": true
    }, 
    "optimizer": {
        "type": "AdamW", 
        "params": {
            "lr": 1e-06, 
            "betas": [0.9, 0.999], 
            "eps": 1e-08, 
            "weight_decay": 0.001
        }
    }, 
    "gradient_accumulation_steps": 16, 
    "gradient_clipping": 1.0, 
    "steps_per_print": inf, 
    "train_batch_size": 128, 
    "train_micro_batch_size_per_gpu": 1, 
    "wall_clock_breakdown": true
}
[INFO|trainer.py:2243] 2024-12-05 13:43:03,768 >> ***** Running training *****
[INFO|trainer.py:2244] 2024-12-05 13:43:03,768 >>   Num examples = 24,618
[INFO|trainer.py:2245] 2024-12-05 13:43:03,768 >>   Num Epochs = 2
[INFO|trainer.py:2246] 2024-12-05 13:43:03,768 >>   Instantaneous batch size per device = 1
[INFO|trainer.py:2249] 2024-12-05 13:43:03,768 >>   Total train batch size (w. parallel, distributed & accumulation) = 128
[INFO|trainer.py:2250] 2024-12-05 13:43:03,768 >>   Gradient Accumulation steps = 16
[INFO|trainer.py:2251] 2024-12-05 13:43:03,768 >>   Total optimization steps = 384
[INFO|trainer.py:2252] 2024-12-05 13:43:03,770 >>   Number of trainable parameters = 7,615,616,512

  0%|          | 0/384 [00:00<?, ?it/s]/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
/cpfs01/user/wuzhenyu/anaconda3/envs/llama-factory/lib/python3.11/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
[2024-12-05 13:43:07,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3061.53 | bwd_microstep: 539.19 | bwd_inner_microstep: 539.04 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.05
[2024-12-05 13:43:08,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 325.20 | bwd_microstep: 620.91 | bwd_inner_microstep: 620.79 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:43:09,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.47 | bwd_microstep: 430.96 | bwd_inner_microstep: 430.82 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.04
[2024-12-05 13:43:10,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.06 | bwd_microstep: 436.35 | bwd_inner_microstep: 436.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.04
[2024-12-05 13:43:11,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 367.35 | bwd_microstep: 745.95 | bwd_inner_microstep: 745.78 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.06
[2024-12-05 13:43:12,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.53 | bwd_microstep: 436.32 | bwd_inner_microstep: 436.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.05
[2024-12-05 13:43:13,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.18 | bwd_microstep: 590.49 | bwd_inner_microstep: 590.36 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.05
[2024-12-05 13:43:14,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.76 | bwd_microstep: 427.88 | bwd_inner_microstep: 427.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.04
[2024-12-05 13:43:15,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.80 | bwd_microstep: 444.97 | bwd_inner_microstep: 444.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.05
[2024-12-05 13:43:16,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.16 | bwd_microstep: 428.56 | bwd_inner_microstep: 428.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.04
[2024-12-05 13:43:16,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.06 | bwd_microstep: 410.86 | bwd_inner_microstep: 410.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.03
[2024-12-05 13:43:17,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.20 | bwd_microstep: 485.26 | bwd_inner_microstep: 485.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.04
[2024-12-05 13:43:18,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.96 | bwd_microstep: 582.01 | bwd_inner_microstep: 581.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.04
[2024-12-05 13:43:19,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.62 | bwd_microstep: 444.16 | bwd_inner_microstep: 444.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.04
[2024-12-05 13:43:20,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.12 | bwd_microstep: 492.88 | bwd_inner_microstep: 492.71 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.04
[2024-12-05 13:43:22,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 105.35 | optimizer_gradients: 14.59 | optimizer_step: 44.76
[2024-12-05 13:43:22,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 344.47 | bwd_microstep: 1065.75 | bwd_inner_microstep: 1053.57 | bwd_allreduce_microstep: 12.09 | step_microstep: 192.54
[2024-12-05 13:43:22,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 7421.55 | bwd: 8582.51 | bwd_inner: 8568.54 | bwd_allreduce: 12.83 | step: 193.20

  0%|          | 1/384 [00:18<1:56:31, 18.25s/it]
                                                 
{'loss': 1.4958, 'grad_norm': 46.50929641723633, 'learning_rate': 8.333333333333333e-08, 'epoch': 0.01}

  0%|          | 1/384 [00:18<1:56:31, 18.25s/it][2024-12-05 13:43:22,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.98 | bwd_microstep: 448.81 | bwd_inner_microstep: 448.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.04
[2024-12-05 13:43:23,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.02 | bwd_microstep: 485.92 | bwd_inner_microstep: 485.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.04
[2024-12-05 13:43:24,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.13 | bwd_microstep: 473.88 | bwd_inner_microstep: 473.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.04
[2024-12-05 13:43:25,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.05 | bwd_microstep: 527.72 | bwd_inner_microstep: 527.55 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.05
[2024-12-05 13:43:26,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.15 | bwd_microstep: 603.69 | bwd_inner_microstep: 603.55 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.05
[2024-12-05 13:43:27,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.06 | bwd_microstep: 565.31 | bwd_inner_microstep: 565.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.04
[2024-12-05 13:43:28,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 238.15 | bwd_microstep: 353.40 | bwd_inner_microstep: 353.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.04
[2024-12-05 13:43:29,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.95 | bwd_microstep: 487.16 | bwd_inner_microstep: 487.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.04
[2024-12-05 13:43:30,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.63 | bwd_microstep: 568.59 | bwd_inner_microstep: 568.45 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.04
[2024-12-05 13:43:30,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.37 | bwd_microstep: 431.99 | bwd_inner_microstep: 431.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.04
[2024-12-05 13:43:31,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.83 | bwd_microstep: 444.61 | bwd_inner_microstep: 444.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.04
[2024-12-05 13:43:32,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.96 | bwd_microstep: 433.00 | bwd_inner_microstep: 432.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.04
[2024-12-05 13:43:33,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.08 | bwd_microstep: 682.67 | bwd_inner_microstep: 682.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.05
[2024-12-05 13:43:34,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.62 | bwd_microstep: 422.23 | bwd_inner_microstep: 422.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.04
[2024-12-05 13:43:35,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 381.81 | bwd_microstep: 839.04 | bwd_inner_microstep: 838.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.04
[2024-12-05 13:43:38,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.07 | optimizer_gradients: 14.11 | optimizer_step: 23.19
[2024-12-05 13:43:38,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.28 | bwd_microstep: 2108.46 | bwd_inner_microstep: 2096.17 | bwd_allreduce_microstep: 12.19 | step_microstep: 115.49
[2024-12-05 13:43:38,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4615.13 | bwd: 9876.45 | bwd_inner: 9862.48 | bwd_allreduce: 12.85 | step: 116.10

  1%|          | 2/384 [00:34<1:48:08, 16.99s/it]
                                                 
{'loss': 1.2222, 'grad_norm': 36.618709564208984, 'learning_rate': 1.6666666666666665e-07, 'epoch': 0.01}

  1%|          | 2/384 [00:34<1:48:08, 16.99s/it][2024-12-05 13:43:38,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.82 | bwd_microstep: 398.38 | bwd_inner_microstep: 398.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:43:39,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.80 | bwd_microstep: 508.23 | bwd_inner_microstep: 508.10 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:43:40,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.26 | bwd_microstep: 353.57 | bwd_inner_microstep: 353.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 13:43:41,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.06 | bwd_microstep: 468.30 | bwd_inner_microstep: 468.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:43:42,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.45 | bwd_microstep: 472.66 | bwd_inner_microstep: 472.53 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:43:43,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.78 | bwd_microstep: 563.31 | bwd_inner_microstep: 563.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:43:43,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.66 | bwd_microstep: 479.64 | bwd_inner_microstep: 479.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:43:44,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.72 | bwd_microstep: 376.42 | bwd_inner_microstep: 376.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:43:45,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.64 | bwd_microstep: 414.22 | bwd_inner_microstep: 414.07 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:43:46,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.91 | bwd_microstep: 411.53 | bwd_inner_microstep: 411.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:43:47,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.95 | bwd_microstep: 602.29 | bwd_inner_microstep: 602.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:43:48,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.95 | bwd_microstep: 599.19 | bwd_inner_microstep: 599.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:43:49,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 351.63 | bwd_microstep: 635.88 | bwd_inner_microstep: 635.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:43:50,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.34 | bwd_microstep: 411.56 | bwd_inner_microstep: 411.42 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 13:43:51,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.47 | bwd_microstep: 606.71 | bwd_inner_microstep: 606.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:43:52,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.59 | optimizer_gradients: 14.12 | optimizer_step: 23.34
[2024-12-05 13:43:52,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 342.56 | bwd_microstep: 1107.26 | bwd_inner_microstep: 1095.23 | bwd_allreduce_microstep: 11.93 | step_microstep: 113.20
[2024-12-05 13:43:52,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4571.07 | bwd: 8409.14 | bwd_inner: 8395.41 | bwd_allreduce: 12.63 | step: 114.48

  1%|          | 3/384 [00:48<1:40:57, 15.90s/it]
                                                 
{'loss': 1.2627, 'grad_norm': 42.09670639038086, 'learning_rate': 2.5e-07, 'epoch': 0.02}

  1%|          | 3/384 [00:48<1:40:57, 15.90s/it][2024-12-05 13:43:53,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.24 | bwd_microstep: 400.56 | bwd_inner_microstep: 400.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:43:54,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.22 | bwd_microstep: 454.80 | bwd_inner_microstep: 454.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:43:55,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.54 | bwd_microstep: 416.40 | bwd_inner_microstep: 416.27 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:43:55,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.41 | bwd_microstep: 439.65 | bwd_inner_microstep: 439.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:43:56,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.82 | bwd_microstep: 444.79 | bwd_inner_microstep: 444.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:43:57,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.27 | bwd_microstep: 455.75 | bwd_inner_microstep: 455.63 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:43:58,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.31 | bwd_microstep: 364.86 | bwd_inner_microstep: 364.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:43:58,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.61 | bwd_microstep: 364.33 | bwd_inner_microstep: 364.19 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:43:59,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.63 | bwd_microstep: 434.63 | bwd_inner_microstep: 434.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:44:00,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.84 | bwd_microstep: 454.77 | bwd_inner_microstep: 454.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:44:01,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.29 | bwd_microstep: 520.38 | bwd_inner_microstep: 520.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:44:02,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.43 | bwd_microstep: 646.18 | bwd_inner_microstep: 646.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:44:03,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.91 | bwd_microstep: 438.89 | bwd_inner_microstep: 438.75 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 13:44:03,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.61 | bwd_microstep: 505.44 | bwd_inner_microstep: 505.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:44:05,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 355.32 | bwd_microstep: 707.74 | bwd_inner_microstep: 707.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:44:07,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.30 | optimizer_gradients: 14.24 | optimizer_step: 23.26
[2024-12-05 13:44:07,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.31 | bwd_microstep: 2003.03 | bwd_inner_microstep: 1990.36 | bwd_allreduce_microstep: 12.55 | step_microstep: 114.09
[2024-12-05 13:44:07,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4395.84 | bwd: 9052.21 | bwd_inner: 9037.87 | bwd_allreduce: 13.21 | step: 115.30

  1%|          | 4/384 [01:03<1:38:02, 15.48s/it]
                                                 
{'loss': 1.2579, 'grad_norm': 38.13116455078125, 'learning_rate': 3.333333333333333e-07, 'epoch': 0.02}

  1%|          | 4/384 [01:03<1:38:02, 15.48s/it][2024-12-05 13:44:08,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.64 | bwd_microstep: 538.06 | bwd_inner_microstep: 537.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:44:09,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 346.83 | bwd_microstep: 711.00 | bwd_inner_microstep: 710.88 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:44:10,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.40 | bwd_microstep: 534.46 | bwd_inner_microstep: 534.32 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:44:11,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.12 | bwd_microstep: 495.53 | bwd_inner_microstep: 495.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:44:12,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.56 | bwd_microstep: 489.64 | bwd_inner_microstep: 489.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 13:44:13,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.51 | bwd_microstep: 549.50 | bwd_inner_microstep: 549.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:44:14,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.46 | bwd_microstep: 447.28 | bwd_inner_microstep: 447.13 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 13:44:14,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.55 | bwd_microstep: 441.33 | bwd_inner_microstep: 441.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:44:15,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.80 | bwd_microstep: 414.99 | bwd_inner_microstep: 414.85 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:44:16,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.67 | bwd_microstep: 447.39 | bwd_inner_microstep: 447.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:44:17,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 325.69 | bwd_microstep: 627.71 | bwd_inner_microstep: 627.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:44:18,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.47 | bwd_microstep: 569.28 | bwd_inner_microstep: 569.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:44:19,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.47 | bwd_microstep: 406.69 | bwd_inner_microstep: 406.55 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 13:44:20,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.57 | bwd_microstep: 438.02 | bwd_inner_microstep: 437.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:44:20,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.06 | bwd_microstep: 510.44 | bwd_inner_microstep: 510.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:44:22,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.46 | optimizer_gradients: 14.06 | optimizer_step: 23.04
[2024-12-05 13:44:22,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.19 | bwd_microstep: 1265.76 | bwd_inner_microstep: 1253.59 | bwd_allreduce_microstep: 12.08 | step_microstep: 113.09
[2024-12-05 13:44:22,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4590.08 | bwd: 8887.06 | bwd_inner: 8873.19 | bwd_allreduce: 12.77 | step: 114.37

  1%|▏         | 5/384 [01:18<1:36:51, 15.33s/it]
                                                 
{'loss': 1.2421, 'grad_norm': 40.15380859375, 'learning_rate': 4.1666666666666667e-07, 'epoch': 0.03}

  1%|▏         | 5/384 [01:18<1:36:51, 15.33s/it][2024-12-05 13:44:23,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.80 | bwd_microstep: 419.78 | bwd_inner_microstep: 419.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:44:24,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.82 | bwd_microstep: 561.01 | bwd_inner_microstep: 560.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:44:25,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.87 | bwd_microstep: 458.37 | bwd_inner_microstep: 458.24 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 13:44:25,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.23 | bwd_microstep: 438.69 | bwd_inner_microstep: 438.56 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.11
[2024-12-05 13:44:26,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.81 | bwd_microstep: 598.15 | bwd_inner_microstep: 598.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:44:27,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.45 | bwd_microstep: 360.23 | bwd_inner_microstep: 360.03 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11
[2024-12-05 13:44:28,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.14 | bwd_microstep: 411.51 | bwd_inner_microstep: 411.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:44:29,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.62 | bwd_microstep: 467.83 | bwd_inner_microstep: 467.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:44:30,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.54 | bwd_microstep: 423.22 | bwd_inner_microstep: 423.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:44:30,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.47 | bwd_microstep: 436.47 | bwd_inner_microstep: 436.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:44:31,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.40 | bwd_microstep: 437.01 | bwd_inner_microstep: 436.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:44:32,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.37 | bwd_microstep: 595.77 | bwd_inner_microstep: 595.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:44:33,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.80 | bwd_microstep: 440.51 | bwd_inner_microstep: 440.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 13:44:34,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.66 | bwd_microstep: 447.93 | bwd_inner_microstep: 447.76 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 13:44:35,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.81 | bwd_microstep: 449.03 | bwd_inner_microstep: 448.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.12
[2024-12-05 13:44:37,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.91 | optimizer_gradients: 14.07 | optimizer_step: 23.12
[2024-12-05 13:44:37,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.44 | bwd_microstep: 1717.57 | bwd_inner_microstep: 1705.31 | bwd_allreduce_microstep: 12.18 | step_microstep: 112.71
[2024-12-05 13:44:37,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4371.32 | bwd: 8663.07 | bwd_inner: 8648.99 | bwd_allreduce: 12.88 | step: 114.10

  2%|▏         | 6/384 [01:33<1:34:59, 15.08s/it]
                                                 
{'loss': 1.264, 'grad_norm': 43.696563720703125, 'learning_rate': 5e-07, 'epoch': 0.03}

  2%|▏         | 6/384 [01:33<1:34:59, 15.08s/it][2024-12-05 13:44:37,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.64 | bwd_microstep: 406.79 | bwd_inner_microstep: 406.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:44:38,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.05 | bwd_microstep: 430.98 | bwd_inner_microstep: 430.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:44:39,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.19 | bwd_microstep: 415.12 | bwd_inner_microstep: 414.98 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:44:40,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.30 | bwd_microstep: 533.70 | bwd_inner_microstep: 533.57 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:44:41,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.94 | bwd_microstep: 360.90 | bwd_inner_microstep: 360.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:44:41,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.09 | bwd_microstep: 381.40 | bwd_inner_microstep: 381.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:44:42,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.76 | bwd_microstep: 532.37 | bwd_inner_microstep: 532.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:44:43,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.58 | bwd_microstep: 539.93 | bwd_inner_microstep: 539.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:44:44,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.85 | bwd_microstep: 453.00 | bwd_inner_microstep: 452.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:44:45,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 240.16 | bwd_microstep: 352.27 | bwd_inner_microstep: 352.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:44:45,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.81 | bwd_microstep: 463.39 | bwd_inner_microstep: 463.25 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:44:46,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.62 | bwd_microstep: 487.01 | bwd_inner_microstep: 486.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:44:47,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.23 | bwd_microstep: 565.46 | bwd_inner_microstep: 565.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:44:48,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.98 | bwd_microstep: 407.58 | bwd_inner_microstep: 407.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:44:49,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.35 | bwd_microstep: 491.71 | bwd_inner_microstep: 491.57 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:44:51,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.50 | optimizer_gradients: 14.06 | optimizer_step: 23.40
[2024-12-05 13:44:51,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.50 | bwd_microstep: 1696.58 | bwd_inner_microstep: 1684.32 | bwd_allreduce_microstep: 12.16 | step_microstep: 113.59
[2024-12-05 13:44:51,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4358.09 | bwd: 8518.17 | bwd_inner: 8504.21 | bwd_allreduce: 12.83 | step: 114.80

  2%|▏         | 7/384 [01:47<1:32:57, 14.79s/it]
                                                 
{'loss': 1.3881, 'grad_norm': 44.71561813354492, 'learning_rate': 5.833333333333334e-07, 'epoch': 0.04}

  2%|▏         | 7/384 [01:47<1:32:57, 14.79s/it][2024-12-05 13:44:52,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.87 | bwd_microstep: 386.66 | bwd_inner_microstep: 386.53 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:44:52,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 240.09 | bwd_microstep: 365.14 | bwd_inner_microstep: 365.01 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:44:53,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.88 | bwd_microstep: 413.74 | bwd_inner_microstep: 413.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:44:54,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.48 | bwd_microstep: 405.92 | bwd_inner_microstep: 405.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:44:55,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.44 | bwd_microstep: 487.80 | bwd_inner_microstep: 487.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:44:55,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.59 | bwd_microstep: 494.60 | bwd_inner_microstep: 494.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:44:56,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.64 | bwd_microstep: 443.38 | bwd_inner_microstep: 443.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:44:57,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.61 | bwd_microstep: 419.53 | bwd_inner_microstep: 419.40 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:44:58,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.37 | bwd_microstep: 495.31 | bwd_inner_microstep: 495.19 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:44:59,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.63 | bwd_microstep: 539.72 | bwd_inner_microstep: 539.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:45:00,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.66 | bwd_microstep: 486.59 | bwd_inner_microstep: 486.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:45:00,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.22 | bwd_microstep: 495.96 | bwd_inner_microstep: 495.83 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:45:01,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.75 | bwd_microstep: 502.75 | bwd_inner_microstep: 502.62 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:45:02,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.02 | bwd_microstep: 421.88 | bwd_inner_microstep: 421.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:45:03,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.46 | bwd_microstep: 458.39 | bwd_inner_microstep: 458.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:45:06,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.17 | optimizer_gradients: 14.16 | optimizer_step: 23.49
[2024-12-05 13:45:06,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.36 | bwd_microstep: 2179.02 | bwd_inner_microstep: 2166.77 | bwd_allreduce_microstep: 12.15 | step_microstep: 112.65
[2024-12-05 13:45:06,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4315.15 | bwd: 8996.39 | bwd_inner: 8982.46 | bwd_allreduce: 12.83 | step: 113.89

  2%|▏         | 8/384 [02:02<1:32:20, 14.74s/it]
                                                 
{'loss': 1.1705, 'grad_norm': 41.07535171508789, 'learning_rate': 6.666666666666666e-07, 'epoch': 0.04}

  2%|▏         | 8/384 [02:02<1:32:20, 14.74s/it][2024-12-05 13:45:06,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.53 | bwd_microstep: 411.73 | bwd_inner_microstep: 411.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:45:07,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.68 | bwd_microstep: 437.36 | bwd_inner_microstep: 437.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:45:08,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.14 | bwd_microstep: 535.12 | bwd_inner_microstep: 534.99 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:45:09,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.73 | bwd_microstep: 443.96 | bwd_inner_microstep: 443.83 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:45:10,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.07 | bwd_microstep: 440.57 | bwd_inner_microstep: 440.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:45:10,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.67 | bwd_microstep: 404.68 | bwd_inner_microstep: 404.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:45:11,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.64 | bwd_microstep: 495.80 | bwd_inner_microstep: 495.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:45:12,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.87 | bwd_microstep: 455.22 | bwd_inner_microstep: 455.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:45:13,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.63 | bwd_microstep: 456.15 | bwd_inner_microstep: 456.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:45:14,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.97 | bwd_microstep: 460.08 | bwd_inner_microstep: 459.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:45:14,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.89 | bwd_microstep: 471.33 | bwd_inner_microstep: 471.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:45:15,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.33 | bwd_microstep: 449.05 | bwd_inner_microstep: 448.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:45:16,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.02 | bwd_microstep: 454.10 | bwd_inner_microstep: 453.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:45:17,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.51 | bwd_microstep: 440.38 | bwd_inner_microstep: 440.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:45:18,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.15 | bwd_microstep: 421.02 | bwd_inner_microstep: 420.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:45:21,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.61 | optimizer_gradients: 14.07 | optimizer_step: 23.13
[2024-12-05 13:45:21,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.86 | bwd_microstep: 2507.50 | bwd_inner_microstep: 2495.24 | bwd_allreduce_microstep: 12.16 | step_microstep: 113.98
[2024-12-05 13:45:21,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4370.75 | bwd: 9284.04 | bwd_inner: 9270.10 | bwd_allreduce: 12.81 | step: 115.15

  2%|▏         | 9/384 [02:17<1:32:49, 14.85s/it]
                                                 
{'loss': 1.1108, 'grad_norm': 37.13037109375, 'learning_rate': 7.5e-07, 'epoch': 0.05}

  2%|▏         | 9/384 [02:17<1:32:49, 14.85s/it][2024-12-05 13:45:21,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.11 | bwd_microstep: 426.24 | bwd_inner_microstep: 426.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:45:22,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.65 | bwd_microstep: 417.53 | bwd_inner_microstep: 417.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:45:23,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.42 | bwd_microstep: 414.69 | bwd_inner_microstep: 414.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:45:24,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.53 | bwd_microstep: 436.18 | bwd_inner_microstep: 436.05 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:45:25,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.49 | bwd_microstep: 422.54 | bwd_inner_microstep: 422.40 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:45:25,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.04 | bwd_microstep: 441.61 | bwd_inner_microstep: 441.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:45:26,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.50 | bwd_microstep: 444.40 | bwd_inner_microstep: 444.25 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:45:27,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.27 | bwd_microstep: 413.17 | bwd_inner_microstep: 413.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:45:28,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.94 | bwd_microstep: 466.27 | bwd_inner_microstep: 466.10 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 13:45:28,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.35 | bwd_microstep: 408.33 | bwd_inner_microstep: 408.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:45:29,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.30 | bwd_microstep: 626.30 | bwd_inner_microstep: 626.17 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 13:45:30,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.18 | bwd_microstep: 422.66 | bwd_inner_microstep: 422.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:45:31,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.25 | bwd_microstep: 473.17 | bwd_inner_microstep: 473.04 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:45:32,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.25 | bwd_microstep: 450.72 | bwd_inner_microstep: 450.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 13:45:33,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.59 | bwd_microstep: 475.85 | bwd_inner_microstep: 475.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:45:36,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.01 | optimizer_gradients: 14.13 | optimizer_step: 23.91
[2024-12-05 13:45:36,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.24 | bwd_microstep: 2328.79 | bwd_inner_microstep: 2316.67 | bwd_allreduce_microstep: 12.03 | step_microstep: 113.05
[2024-12-05 13:45:36,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4301.16 | bwd: 9068.43 | bwd_inner: 9054.53 | bwd_allreduce: 12.73 | step: 114.37

  3%|▎         | 10/384 [02:32<1:32:34, 14.85s/it]
                                                  
{'loss': 1.1024, 'grad_norm': 39.47488021850586, 'learning_rate': 8.333333333333333e-07, 'epoch': 0.05}

  3%|▎         | 10/384 [02:32<1:32:34, 14.85s/it][2024-12-05 13:45:36,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 238.67 | bwd_microstep: 334.08 | bwd_inner_microstep: 333.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:45:37,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.90 | bwd_microstep: 531.45 | bwd_inner_microstep: 531.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:45:38,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.86 | bwd_microstep: 525.57 | bwd_inner_microstep: 525.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:45:39,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.77 | bwd_microstep: 488.78 | bwd_inner_microstep: 488.61 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:45:40,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.30 | bwd_microstep: 444.47 | bwd_inner_microstep: 444.32 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:45:40,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.77 | bwd_microstep: 440.20 | bwd_inner_microstep: 440.06 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:45:41,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.83 | bwd_microstep: 523.32 | bwd_inner_microstep: 523.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:45:42,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.28 | bwd_microstep: 404.18 | bwd_inner_microstep: 404.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:45:43,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 366.61 | bwd_microstep: 778.37 | bwd_inner_microstep: 778.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:45:44,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.49 | bwd_microstep: 431.19 | bwd_inner_microstep: 431.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:45:45,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.41 | bwd_microstep: 716.34 | bwd_inner_microstep: 716.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:45:46,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.53 | bwd_microstep: 463.51 | bwd_inner_microstep: 463.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:45:47,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.62 | bwd_microstep: 435.75 | bwd_inner_microstep: 435.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:45:48,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.10 | bwd_microstep: 442.30 | bwd_inner_microstep: 442.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:45:48,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.87 | bwd_microstep: 419.31 | bwd_inner_microstep: 419.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:45:51,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.80 | optimizer_gradients: 14.08 | optimizer_step: 23.16
[2024-12-05 13:45:51,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.14 | bwd_microstep: 2018.03 | bwd_inner_microstep: 2005.95 | bwd_allreduce_microstep: 11.99 | step_microstep: 113.22
[2024-12-05 13:45:51,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4477.22 | bwd: 9396.84 | bwd_inner: 9383.03 | bwd_allreduce: 12.67 | step: 114.39

  3%|▎         | 11/384 [02:47<1:33:23, 15.02s/it]
                                                  
{'loss': 1.0848, 'grad_norm': 35.2398681640625, 'learning_rate': 9.166666666666665e-07, 'epoch': 0.06}

  3%|▎         | 11/384 [02:47<1:33:23, 15.02s/it][2024-12-05 13:45:52,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.48 | bwd_microstep: 459.37 | bwd_inner_microstep: 459.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:45:53,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.46 | bwd_microstep: 628.30 | bwd_inner_microstep: 628.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:45:54,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.57 | bwd_microstep: 568.36 | bwd_inner_microstep: 568.22 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:45:54,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.11 | bwd_microstep: 397.81 | bwd_inner_microstep: 397.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:45:55,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.59 | bwd_microstep: 416.04 | bwd_inner_microstep: 415.91 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:45:56,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.19 | bwd_microstep: 433.86 | bwd_inner_microstep: 433.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:45:57,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.80 | bwd_microstep: 362.67 | bwd_inner_microstep: 362.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:45:58,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 382.72 | bwd_microstep: 847.21 | bwd_inner_microstep: 847.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:45:59,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.22 | bwd_microstep: 404.82 | bwd_inner_microstep: 404.68 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 13:46:00,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.85 | bwd_microstep: 439.92 | bwd_inner_microstep: 439.78 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:46:00,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.13 | bwd_microstep: 450.71 | bwd_inner_microstep: 450.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:46:01,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.20 | bwd_microstep: 439.77 | bwd_inner_microstep: 439.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:46:02,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.74 | bwd_microstep: 572.84 | bwd_inner_microstep: 572.71 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:46:03,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.56 | bwd_microstep: 393.97 | bwd_inner_microstep: 393.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:46:04,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.59 | bwd_microstep: 438.36 | bwd_inner_microstep: 438.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:46:06,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.87 | optimizer_gradients: 14.09 | optimizer_step: 23.11
[2024-12-05 13:46:06,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 343.00 | bwd_microstep: 1384.82 | bwd_inner_microstep: 1372.58 | bwd_allreduce_microstep: 12.14 | step_microstep: 112.86
[2024-12-05 13:46:06,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4536.30 | bwd: 8638.80 | bwd_inner: 8624.84 | bwd_allreduce: 12.81 | step: 114.07

  3%|▎         | 12/384 [03:02<1:32:27, 14.91s/it]
                                                  
{'loss': 0.7678, 'grad_norm': 26.39617347717285, 'learning_rate': 1e-06, 'epoch': 0.06}

  3%|▎         | 12/384 [03:02<1:32:27, 14.91s/it][2024-12-05 13:46:07,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.54 | bwd_microstep: 545.38 | bwd_inner_microstep: 545.23 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09
[2024-12-05 13:46:08,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.34 | bwd_microstep: 563.51 | bwd_inner_microstep: 563.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:46:08,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.50 | bwd_microstep: 432.92 | bwd_inner_microstep: 432.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:46:09,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.85 | bwd_microstep: 680.87 | bwd_inner_microstep: 680.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:46:10,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.15 | bwd_microstep: 490.65 | bwd_inner_microstep: 490.49 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.11
[2024-12-05 13:46:11,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.35 | bwd_microstep: 453.31 | bwd_inner_microstep: 453.17 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 13:46:12,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 348.74 | bwd_microstep: 706.18 | bwd_inner_microstep: 706.06 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:46:13,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.20 | bwd_microstep: 410.02 | bwd_inner_microstep: 409.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:46:14,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.61 | bwd_microstep: 427.35 | bwd_inner_microstep: 427.21 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:46:15,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.84 | bwd_microstep: 426.89 | bwd_inner_microstep: 426.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:46:16,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.96 | bwd_microstep: 557.38 | bwd_inner_microstep: 557.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:46:16,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.90 | bwd_microstep: 426.58 | bwd_inner_microstep: 426.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:46:17,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.07 | bwd_microstep: 614.77 | bwd_inner_microstep: 614.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:46:18,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 360.08 | bwd_microstep: 741.92 | bwd_inner_microstep: 741.79 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:46:19,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.62 | bwd_microstep: 405.93 | bwd_inner_microstep: 405.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:46:21,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.20 | optimizer_gradients: 14.13 | optimizer_step: 23.13
[2024-12-05 13:46:21,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.93 | bwd_microstep: 1456.74 | bwd_inner_microstep: 1444.63 | bwd_allreduce_microstep: 12.02 | step_microstep: 112.52
[2024-12-05 13:46:21,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4750.79 | bwd: 9340.39 | bwd_inner: 9326.53 | bwd_allreduce: 12.68 | step: 113.84

  3%|▎         | 13/384 [03:17<1:33:31, 15.12s/it]
                                                  
{'loss': 0.7753, 'grad_norm': 23.569713592529297, 'learning_rate': 9.999821700020548e-07, 'epoch': 0.07}

  3%|▎         | 13/384 [03:17<1:33:31, 15.12s/it][2024-12-05 13:46:22,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.31 | bwd_microstep: 381.48 | bwd_inner_microstep: 381.29 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 13:46:23,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 331.13 | bwd_microstep: 624.92 | bwd_inner_microstep: 624.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:46:24,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.53 | bwd_microstep: 412.06 | bwd_inner_microstep: 411.93 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 13:46:25,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.09 | bwd_microstep: 490.57 | bwd_inner_microstep: 490.41 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:46:25,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.98 | bwd_microstep: 406.78 | bwd_inner_microstep: 406.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:46:26,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.38 | bwd_microstep: 418.17 | bwd_inner_microstep: 418.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:46:27,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.98 | bwd_microstep: 427.91 | bwd_inner_microstep: 427.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:46:28,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.88 | bwd_microstep: 686.84 | bwd_inner_microstep: 686.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:46:29,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.91 | bwd_microstep: 443.64 | bwd_inner_microstep: 443.52 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:46:30,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.57 | bwd_microstep: 454.90 | bwd_inner_microstep: 454.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:46:30,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.19 | bwd_microstep: 452.56 | bwd_inner_microstep: 452.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:46:31,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.30 | bwd_microstep: 592.87 | bwd_inner_microstep: 592.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:46:32,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.23 | bwd_microstep: 564.69 | bwd_inner_microstep: 564.56 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:46:33,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.62 | bwd_microstep: 430.80 | bwd_inner_microstep: 430.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:46:34,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.59 | bwd_microstep: 600.70 | bwd_inner_microstep: 600.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:46:36,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.54 | optimizer_gradients: 14.07 | optimizer_step: 23.25
[2024-12-05 13:46:36,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.92 | bwd_microstep: 1075.98 | bwd_inner_microstep: 1063.72 | bwd_allreduce_microstep: 12.18 | step_microstep: 112.34
[2024-12-05 13:46:36,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4506.67 | bwd: 8464.86 | bwd_inner: 8450.86 | bwd_allreduce: 12.85 | step: 113.56

  4%|▎         | 14/384 [03:32<1:32:14, 14.96s/it]
                                                  
{'loss': 0.6663, 'grad_norm': 20.85965919494629, 'learning_rate': 9.99928681279855e-07, 'epoch': 0.07}

  4%|▎         | 14/384 [03:32<1:32:14, 14.96s/it][2024-12-05 13:46:37,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.64 | bwd_microstep: 603.93 | bwd_inner_microstep: 603.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:46:38,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.73 | bwd_microstep: 451.71 | bwd_inner_microstep: 451.57 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 13:46:38,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.88 | bwd_microstep: 458.93 | bwd_inner_microstep: 458.77 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:46:39,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.44 | bwd_microstep: 571.98 | bwd_inner_microstep: 571.82 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 13:46:40,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.01 | bwd_microstep: 468.06 | bwd_inner_microstep: 467.90 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 13:46:41,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.82 | bwd_microstep: 445.10 | bwd_inner_microstep: 444.93 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 13:46:42,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.66 | bwd_microstep: 537.58 | bwd_inner_microstep: 537.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:46:43,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.52 | bwd_microstep: 631.06 | bwd_inner_microstep: 630.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:46:44,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.42 | bwd_microstep: 358.52 | bwd_inner_microstep: 358.39 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:46:45,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.39 | bwd_microstep: 597.49 | bwd_inner_microstep: 597.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 13:46:46,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.26 | bwd_microstep: 564.20 | bwd_inner_microstep: 564.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 13:46:47,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.55 | bwd_microstep: 443.84 | bwd_inner_microstep: 443.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:46:48,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 362.43 | bwd_microstep: 748.96 | bwd_inner_microstep: 748.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:46:49,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.08 | bwd_microstep: 430.85 | bwd_inner_microstep: 430.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:46:49,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.13 | bwd_microstep: 452.21 | bwd_inner_microstep: 452.08 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:46:51,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.87 | optimizer_gradients: 14.27 | optimizer_step: 23.29
[2024-12-05 13:46:51,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.37 | bwd_microstep: 972.72 | bwd_inner_microstep: 960.05 | bwd_allreduce_microstep: 12.54 | step_microstep: 113.76
[2024-12-05 13:46:51,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4614.40 | bwd: 8737.12 | bwd_inner: 8722.64 | bwd_allreduce: 13.26 | step: 115.08

  4%|▍         | 15/384 [03:47<1:32:21, 15.02s/it]
                                                  
{'loss': 0.5468, 'grad_norm': 19.964326858520508, 'learning_rate': 9.998395376482152e-07, 'epoch': 0.08}

  4%|▍         | 15/384 [03:47<1:32:21, 15.02s/it][2024-12-05 13:46:52,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.93 | bwd_microstep: 432.50 | bwd_inner_microstep: 432.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:46:52,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.48 | bwd_microstep: 408.67 | bwd_inner_microstep: 408.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:46:53,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.96 | bwd_microstep: 447.52 | bwd_inner_microstep: 447.40 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:46:54,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.99 | bwd_microstep: 467.86 | bwd_inner_microstep: 467.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:46:55,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.86 | bwd_microstep: 410.14 | bwd_inner_microstep: 409.99 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 13:46:56,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.49 | bwd_microstep: 407.94 | bwd_inner_microstep: 407.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:46:57,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.57 | bwd_microstep: 586.21 | bwd_inner_microstep: 586.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:46:57,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.79 | bwd_microstep: 414.49 | bwd_inner_microstep: 414.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:46:58,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.98 | bwd_microstep: 586.56 | bwd_inner_microstep: 586.42 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 13:46:59,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.33 | bwd_microstep: 572.93 | bwd_inner_microstep: 572.79 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 13:47:00,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.57 | bwd_microstep: 566.07 | bwd_inner_microstep: 565.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:01,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.15 | bwd_microstep: 453.91 | bwd_inner_microstep: 453.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:47:02,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 352.32 | bwd_microstep: 713.83 | bwd_inner_microstep: 713.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:47:03,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.07 | bwd_microstep: 566.41 | bwd_inner_microstep: 566.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:04,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.69 | bwd_microstep: 441.14 | bwd_inner_microstep: 441.01 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:47:07,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.98 | optimizer_gradients: 14.22 | optimizer_step: 23.20
[2024-12-05 13:47:07,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.97 | bwd_microstep: 1975.42 | bwd_inner_microstep: 1963.25 | bwd_allreduce_microstep: 12.09 | step_microstep: 113.24
[2024-12-05 13:47:07,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4518.19 | bwd: 9451.61 | bwd_inner: 9437.73 | bwd_allreduce: 12.76 | step: 114.53

  4%|▍         | 16/384 [04:03<1:33:26, 15.23s/it]
                                                  
{'loss': 0.4754, 'grad_norm': 10.887548446655273, 'learning_rate': 9.997147454648588e-07, 'epoch': 0.08}

  4%|▍         | 16/384 [04:03<1:33:26, 15.23s/it][2024-12-05 13:47:07,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.47 | bwd_microstep: 417.74 | bwd_inner_microstep: 417.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:47:09,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 342.40 | bwd_microstep: 693.88 | bwd_inner_microstep: 693.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:47:09,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.60 | bwd_microstep: 441.67 | bwd_inner_microstep: 441.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:47:10,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.10 | bwd_microstep: 445.28 | bwd_inner_microstep: 445.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:47:11,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.17 | bwd_microstep: 679.58 | bwd_inner_microstep: 679.44 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:47:12,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.37 | bwd_microstep: 525.19 | bwd_inner_microstep: 525.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:47:13,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.25 | bwd_microstep: 411.38 | bwd_inner_microstep: 411.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:47:14,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.85 | bwd_microstep: 351.70 | bwd_inner_microstep: 351.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:47:14,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.93 | bwd_microstep: 433.10 | bwd_inner_microstep: 432.96 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:47:15,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.68 | bwd_microstep: 430.30 | bwd_inner_microstep: 430.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:47:16,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 350.72 | bwd_microstep: 716.14 | bwd_inner_microstep: 715.95 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 13:47:17,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.04 | bwd_microstep: 353.62 | bwd_inner_microstep: 353.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:18,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.95 | bwd_microstep: 455.72 | bwd_inner_microstep: 455.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:19,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.71 | bwd_microstep: 491.01 | bwd_inner_microstep: 490.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:47:20,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.24 | bwd_microstep: 451.30 | bwd_inner_microstep: 451.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:21,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.41 | optimizer_gradients: 14.09 | optimizer_step: 23.15
[2024-12-05 13:47:21,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.37 | bwd_microstep: 929.80 | bwd_inner_microstep: 917.59 | bwd_allreduce_microstep: 12.11 | step_microstep: 114.22
[2024-12-05 13:47:21,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4559.88 | bwd: 8227.40 | bwd_inner: 8213.48 | bwd_allreduce: 12.78 | step: 115.40

  4%|▍         | 17/384 [04:17<1:31:22, 14.94s/it]
                                                  
{'loss': 0.4547, 'grad_norm': 10.726633071899414, 'learning_rate': 9.995543136299635e-07, 'epoch': 0.09}

  4%|▍         | 17/384 [04:17<1:31:22, 14.94s/it][2024-12-05 13:47:22,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.82 | bwd_microstep: 432.44 | bwd_inner_microstep: 432.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:23,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.84 | bwd_microstep: 463.68 | bwd_inner_microstep: 463.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:23,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.91 | bwd_microstep: 432.17 | bwd_inner_microstep: 432.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:24,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.66 | bwd_microstep: 365.69 | bwd_inner_microstep: 365.53 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:47:25,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.45 | bwd_microstep: 437.94 | bwd_inner_microstep: 437.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:47:26,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.99 | bwd_microstep: 627.19 | bwd_inner_microstep: 627.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:27,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.39 | bwd_microstep: 437.61 | bwd_inner_microstep: 437.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:47:28,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.67 | bwd_microstep: 599.74 | bwd_inner_microstep: 599.61 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:47:28,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.62 | bwd_microstep: 422.76 | bwd_inner_microstep: 422.57 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.12
[2024-12-05 13:47:29,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.99 | bwd_microstep: 433.79 | bwd_inner_microstep: 433.64 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 13:47:30,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.07 | bwd_microstep: 444.58 | bwd_inner_microstep: 444.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:31,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.22 | bwd_microstep: 456.00 | bwd_inner_microstep: 455.82 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 13:47:32,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.75 | bwd_microstep: 563.22 | bwd_inner_microstep: 563.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 13:47:33,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.96 | bwd_microstep: 402.53 | bwd_inner_microstep: 402.38 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 13:47:33,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.49 | bwd_microstep: 482.68 | bwd_inner_microstep: 482.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:36,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.98 | optimizer_gradients: 14.10 | optimizer_step: 23.42
[2024-12-05 13:47:36,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.24 | bwd_microstep: 2008.96 | bwd_inner_microstep: 1996.71 | bwd_allreduce_microstep: 12.15 | step_microstep: 112.50
[2024-12-05 13:47:36,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4356.15 | bwd: 9011.00 | bwd_inner: 8996.91 | bwd_allreduce: 12.88 | step: 113.87

  5%|▍         | 18/384 [04:32<1:31:13, 14.96s/it]
                                                  
{'loss': 0.4634, 'grad_norm': 9.673012733459473, 'learning_rate': 9.993582535855263e-07, 'epoch': 0.09}

  5%|▍         | 18/384 [04:32<1:31:13, 14.96s/it][2024-12-05 13:47:37,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.22 | bwd_microstep: 424.02 | bwd_inner_microstep: 423.86 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:47:37,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.05 | bwd_microstep: 346.75 | bwd_inner_microstep: 346.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:47:38,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.45 | bwd_microstep: 477.00 | bwd_inner_microstep: 476.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:39,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.62 | bwd_microstep: 441.29 | bwd_inner_microstep: 441.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:47:40,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.99 | bwd_microstep: 361.56 | bwd_inner_microstep: 361.43 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:47:40,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.54 | bwd_microstep: 443.28 | bwd_inner_microstep: 443.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:41,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 357.48 | bwd_microstep: 454.95 | bwd_inner_microstep: 454.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:47:42,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.00 | bwd_microstep: 416.75 | bwd_inner_microstep: 416.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:43,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.28 | bwd_microstep: 407.71 | bwd_inner_microstep: 407.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:44,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.57 | bwd_microstep: 408.61 | bwd_inner_microstep: 408.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:44,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.79 | bwd_microstep: 467.12 | bwd_inner_microstep: 466.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:47:45,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.52 | bwd_microstep: 435.62 | bwd_inner_microstep: 435.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:46,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.55 | bwd_microstep: 431.18 | bwd_inner_microstep: 431.04 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:47:47,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.70 | bwd_microstep: 436.37 | bwd_inner_microstep: 436.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:48,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.57 | bwd_microstep: 405.41 | bwd_inner_microstep: 405.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:47:51,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.55 | optimizer_gradients: 14.11 | optimizer_step: 23.27
[2024-12-05 13:47:51,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.76 | bwd_microstep: 2635.64 | bwd_inner_microstep: 2623.39 | bwd_allreduce_microstep: 12.14 | step_microstep: 114.57
[2024-12-05 13:47:51,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4394.20 | bwd: 8993.20 | bwd_inner: 8979.25 | bwd_allreduce: 12.78 | step: 115.79

  5%|▍         | 19/384 [04:47<1:30:42, 14.91s/it]
                                                  
{'loss': 0.3635, 'grad_norm': 7.258286476135254, 'learning_rate': 9.991265793145479e-07, 'epoch': 0.1}

  5%|▍         | 19/384 [04:47<1:30:42, 14.91s/it][2024-12-05 13:47:52,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.35 | bwd_microstep: 538.83 | bwd_inner_microstep: 538.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:47:52,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.97 | bwd_microstep: 453.36 | bwd_inner_microstep: 453.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:47:53,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.61 | bwd_microstep: 523.65 | bwd_inner_microstep: 523.51 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 13:47:54,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.54 | bwd_microstep: 596.68 | bwd_inner_microstep: 596.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:47:55,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.40 | bwd_microstep: 478.30 | bwd_inner_microstep: 478.17 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:47:56,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.72 | bwd_microstep: 448.95 | bwd_inner_microstep: 448.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:47:57,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.80 | bwd_microstep: 463.51 | bwd_inner_microstep: 463.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:58,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.50 | bwd_microstep: 399.80 | bwd_inner_microstep: 399.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:47:59,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.17 | bwd_microstep: 592.76 | bwd_inner_microstep: 592.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:47:59,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.13 | bwd_microstep: 456.06 | bwd_inner_microstep: 455.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:00,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 241.96 | bwd_microstep: 362.13 | bwd_inner_microstep: 362.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:01,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.21 | bwd_microstep: 458.08 | bwd_inner_microstep: 457.91 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 13:48:02,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.31 | bwd_microstep: 541.07 | bwd_inner_microstep: 540.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:48:03,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.36 | bwd_microstep: 463.77 | bwd_inner_microstep: 463.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:48:04,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.16 | bwd_microstep: 561.02 | bwd_inner_microstep: 560.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:06,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.45 | optimizer_gradients: 14.11 | optimizer_step: 23.22
[2024-12-05 13:48:06,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.71 | bwd_microstep: 1803.37 | bwd_inner_microstep: 1791.22 | bwd_allreduce_microstep: 12.07 | step_microstep: 112.72
[2024-12-05 13:48:06,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4515.95 | bwd: 9141.33 | bwd_inner: 9127.42 | bwd_allreduce: 12.73 | step: 113.88

  5%|▌         | 20/384 [05:02<1:30:56, 14.99s/it]
                                                  
{'loss': 0.356, 'grad_norm': 5.402667999267578, 'learning_rate': 9.988593073400354e-07, 'epoch': 0.1}

  5%|▌         | 20/384 [05:02<1:30:56, 14.99s/it][2024-12-05 13:48:07,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.74 | bwd_microstep: 407.69 | bwd_inner_microstep: 407.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:07,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.79 | bwd_microstep: 399.27 | bwd_inner_microstep: 399.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:08,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.01 | bwd_microstep: 436.42 | bwd_inner_microstep: 436.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:09,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.00 | bwd_microstep: 497.90 | bwd_inner_microstep: 497.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:10,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.23 | bwd_microstep: 442.56 | bwd_inner_microstep: 442.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:11,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.45 | bwd_microstep: 467.08 | bwd_inner_microstep: 466.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:48:11,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.80 | bwd_microstep: 446.10 | bwd_inner_microstep: 445.97 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:48:13,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.69 | bwd_microstep: 620.07 | bwd_inner_microstep: 619.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:48:13,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.42 | bwd_microstep: 434.17 | bwd_inner_microstep: 434.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:14,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.64 | bwd_microstep: 358.23 | bwd_inner_microstep: 358.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:15,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.75 | bwd_microstep: 444.54 | bwd_inner_microstep: 444.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:16,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.48 | bwd_microstep: 463.35 | bwd_inner_microstep: 463.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:17,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.47 | bwd_microstep: 603.26 | bwd_inner_microstep: 603.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:48:18,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.74 | bwd_microstep: 595.11 | bwd_inner_microstep: 594.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:18,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.18 | bwd_microstep: 335.97 | bwd_inner_microstep: 335.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:48:21,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.89 | optimizer_gradients: 14.28 | optimizer_step: 23.57
[2024-12-05 13:48:21,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.08 | bwd_microstep: 1751.69 | bwd_inner_microstep: 1739.20 | bwd_allreduce_microstep: 12.39 | step_microstep: 115.60
[2024-12-05 13:48:21,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4399.53 | bwd: 8703.42 | bwd_inner: 8689.21 | bwd_allreduce: 13.01 | step: 116.72

  5%|▌         | 21/384 [05:17<1:30:25, 14.95s/it]
                                                  
{'loss': 0.3692, 'grad_norm': 4.859364032745361, 'learning_rate': 9.985564567238236e-07, 'epoch': 0.11}

  5%|▌         | 21/384 [05:17<1:30:25, 14.95s/it][2024-12-05 13:48:22,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.95 | bwd_microstep: 424.84 | bwd_inner_microstep: 424.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:22,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 238.91 | bwd_microstep: 359.19 | bwd_inner_microstep: 359.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:48:23,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.81 | bwd_microstep: 414.86 | bwd_inner_microstep: 414.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:24,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.54 | bwd_microstep: 431.38 | bwd_inner_microstep: 431.25 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:48:25,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 361.49 | bwd_microstep: 747.08 | bwd_inner_microstep: 746.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:26,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.04 | bwd_microstep: 420.26 | bwd_inner_microstep: 420.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:26,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.31 | bwd_microstep: 455.41 | bwd_inner_microstep: 455.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:27,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.60 | bwd_microstep: 453.22 | bwd_inner_microstep: 453.09 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:48:28,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.95 | bwd_microstep: 489.80 | bwd_inner_microstep: 489.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:48:29,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 352.14 | bwd_microstep: 724.63 | bwd_inner_microstep: 724.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:30,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.37 | bwd_microstep: 499.49 | bwd_inner_microstep: 499.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:31,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.34 | bwd_microstep: 451.38 | bwd_inner_microstep: 451.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:32,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.83 | bwd_microstep: 418.03 | bwd_inner_microstep: 417.88 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 13:48:32,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 239.08 | bwd_microstep: 363.71 | bwd_inner_microstep: 363.57 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:48:33,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.23 | bwd_microstep: 417.34 | bwd_inner_microstep: 417.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:36,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.67 | optimizer_gradients: 14.09 | optimizer_step: 23.11
[2024-12-05 13:48:36,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.45 | bwd_microstep: 1935.65 | bwd_inner_microstep: 1923.37 | bwd_allreduce_microstep: 12.18 | step_microstep: 114.31
[2024-12-05 13:48:36,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4420.10 | bwd: 9006.28 | bwd_inner: 8992.30 | bwd_allreduce: 12.86 | step: 115.39

  6%|▌         | 22/384 [05:32<1:30:03, 14.93s/it]
                                                  
{'loss': 0.2976, 'grad_norm': 3.8276686668395996, 'learning_rate': 9.982180490652164e-07, 'epoch': 0.11}

  6%|▌         | 22/384 [05:32<1:30:03, 14.93s/it][2024-12-05 13:48:36,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.89 | bwd_microstep: 403.70 | bwd_inner_microstep: 403.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:48:37,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.34 | bwd_microstep: 587.58 | bwd_inner_microstep: 587.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:38,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.11 | bwd_microstep: 445.04 | bwd_inner_microstep: 444.92 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:48:39,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.40 | bwd_microstep: 399.40 | bwd_inner_microstep: 399.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:40,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 359.73 | bwd_microstep: 748.28 | bwd_inner_microstep: 748.14 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:48:41,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.85 | bwd_microstep: 553.65 | bwd_inner_microstep: 553.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:42,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.94 | bwd_microstep: 467.53 | bwd_inner_microstep: 467.34 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 13:48:43,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.39 | bwd_microstep: 408.76 | bwd_inner_microstep: 408.56 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10
[2024-12-05 13:48:43,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.74 | bwd_microstep: 337.14 | bwd_inner_microstep: 337.00 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:48:44,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.94 | bwd_microstep: 689.92 | bwd_inner_microstep: 689.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:45,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.22 | bwd_microstep: 492.56 | bwd_inner_microstep: 492.42 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:48:46,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.39 | bwd_microstep: 408.68 | bwd_inner_microstep: 408.54 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:48:47,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.32 | bwd_microstep: 617.89 | bwd_inner_microstep: 617.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:48:48,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.26 | bwd_microstep: 408.20 | bwd_inner_microstep: 408.07 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:48:49,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.16 | bwd_microstep: 485.70 | bwd_inner_microstep: 485.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:48:50,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.42 | optimizer_gradients: 14.10 | optimizer_step: 23.14
[2024-12-05 13:48:50,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.24 | bwd_microstep: 1001.88 | bwd_inner_microstep: 989.79 | bwd_allreduce_microstep: 12.01 | step_microstep: 112.32
[2024-12-05 13:48:50,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4511.98 | bwd: 8455.91 | bwd_inner: 8441.97 | bwd_allreduce: 12.73 | step: 113.55

  6%|▌         | 23/384 [05:46<1:29:06, 14.81s/it]
                                                  
{'loss': 0.2635, 'grad_norm': 3.0185964107513428, 'learning_rate': 9.97844108499445e-07, 'epoch': 0.12}

  6%|▌         | 23/384 [05:46<1:29:06, 14.81s/it][2024-12-05 13:48:52,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 398.14 | bwd_microstep: 844.32 | bwd_inner_microstep: 844.18 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 13:48:53,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.69 | bwd_microstep: 556.89 | bwd_inner_microstep: 556.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:53,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.57 | bwd_microstep: 437.42 | bwd_inner_microstep: 437.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:54,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.94 | bwd_microstep: 395.54 | bwd_inner_microstep: 395.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:55,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.55 | bwd_microstep: 512.89 | bwd_inner_microstep: 512.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:56,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.26 | bwd_microstep: 537.27 | bwd_inner_microstep: 537.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:57,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.65 | bwd_microstep: 417.06 | bwd_inner_microstep: 416.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:58,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.66 | bwd_microstep: 594.48 | bwd_inner_microstep: 594.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:58,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.28 | bwd_microstep: 341.51 | bwd_inner_microstep: 341.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:48:59,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.35 | bwd_microstep: 439.21 | bwd_inner_microstep: 439.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:00,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.93 | bwd_microstep: 499.95 | bwd_inner_microstep: 499.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:01,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.52 | bwd_microstep: 402.56 | bwd_inner_microstep: 402.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:02,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.57 | bwd_microstep: 408.74 | bwd_inner_microstep: 408.58 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:49:03,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.78 | bwd_microstep: 561.58 | bwd_inner_microstep: 561.44 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:49:04,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.72 | bwd_microstep: 573.19 | bwd_inner_microstep: 573.05 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:49:05,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.21 | optimizer_gradients: 14.07 | optimizer_step: 23.09
[2024-12-05 13:49:05,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.15 | bwd_microstep: 1176.41 | bwd_inner_microstep: 1164.36 | bwd_allreduce_microstep: 11.95 | step_microstep: 113.15
[2024-12-05 13:49:05,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4611.83 | bwd: 8699.00 | bwd_inner: 8685.26 | bwd_allreduce: 12.60 | step: 114.24

  6%|▋         | 24/384 [06:02<1:29:27, 14.91s/it]
                                                  
{'loss': 0.3086, 'grad_norm': 2.6632726192474365, 'learning_rate': 9.974346616959475e-07, 'epoch': 0.12}

  6%|▋         | 24/384 [06:02<1:29:27, 14.91s/it][2024-12-05 13:49:06,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.56 | bwd_microstep: 462.80 | bwd_inner_microstep: 462.67 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:49:07,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.57 | bwd_microstep: 448.44 | bwd_inner_microstep: 448.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:49:08,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.84 | bwd_microstep: 526.96 | bwd_inner_microstep: 526.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:49:09,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.93 | bwd_microstep: 391.59 | bwd_inner_microstep: 391.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:49:10,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.46 | bwd_microstep: 569.71 | bwd_inner_microstep: 569.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:49:11,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.88 | bwd_microstep: 400.30 | bwd_inner_microstep: 400.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:11,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.03 | bwd_microstep: 442.08 | bwd_inner_microstep: 441.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:12,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.84 | bwd_microstep: 432.80 | bwd_inner_microstep: 432.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:13,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.40 | bwd_microstep: 437.02 | bwd_inner_microstep: 436.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:49:14,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 374.80 | bwd_microstep: 793.89 | bwd_inner_microstep: 793.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:49:15,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.85 | bwd_microstep: 489.45 | bwd_inner_microstep: 489.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:16,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.75 | bwd_microstep: 414.50 | bwd_inner_microstep: 414.36 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:49:17,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.48 | bwd_microstep: 448.67 | bwd_inner_microstep: 448.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:18,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.53 | bwd_microstep: 434.66 | bwd_inner_microstep: 434.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:18,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.28 | bwd_microstep: 432.68 | bwd_inner_microstep: 432.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:20,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.41 | optimizer_gradients: 14.16 | optimizer_step: 23.45
[2024-12-05 13:49:20,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.24 | bwd_microstep: 1139.53 | bwd_inner_microstep: 1127.30 | bwd_allreduce_microstep: 12.12 | step_microstep: 113.11
[2024-12-05 13:49:20,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4454.49 | bwd: 8265.09 | bwd_inner: 8251.23 | bwd_allreduce: 12.76 | step: 114.26

  7%|▋         | 25/384 [06:16<1:28:43, 14.83s/it]
                                                  
{'loss': 0.2746, 'grad_norm': 2.5440852642059326, 'learning_rate': 9.969897378564667e-07, 'epoch': 0.13}

  7%|▋         | 25/384 [06:16<1:28:43, 14.83s/it][2024-12-05 13:49:21,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.20 | bwd_microstep: 504.75 | bwd_inner_microstep: 504.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:49:22,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.38 | bwd_microstep: 494.95 | bwd_inner_microstep: 494.82 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:49:23,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.16 | bwd_microstep: 496.46 | bwd_inner_microstep: 496.32 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:49:23,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.27 | bwd_microstep: 465.04 | bwd_inner_microstep: 464.85 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09
[2024-12-05 13:49:24,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.96 | bwd_microstep: 434.41 | bwd_inner_microstep: 434.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:25,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.86 | bwd_microstep: 603.70 | bwd_inner_microstep: 603.56 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:49:26,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.02 | bwd_microstep: 501.74 | bwd_inner_microstep: 501.62 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.11
[2024-12-05 13:49:27,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.02 | bwd_microstep: 489.36 | bwd_inner_microstep: 489.22 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 13:49:28,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.87 | bwd_microstep: 408.52 | bwd_inner_microstep: 408.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:49:28,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.29 | bwd_microstep: 457.95 | bwd_inner_microstep: 457.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:49:29,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.82 | bwd_microstep: 414.69 | bwd_inner_microstep: 414.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:49:30,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.38 | bwd_microstep: 411.81 | bwd_inner_microstep: 411.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:49:31,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.42 | bwd_microstep: 466.04 | bwd_inner_microstep: 465.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:49:31,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.96 | bwd_microstep: 445.69 | bwd_inner_microstep: 445.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:49:32,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.11 | bwd_microstep: 438.99 | bwd_inner_microstep: 438.85 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:49:34,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.99 | optimizer_gradients: 14.09 | optimizer_step: 23.84
[2024-12-05 13:49:34,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.56 | bwd_microstep: 1613.00 | bwd_inner_microstep: 1600.92 | bwd_allreduce_microstep: 11.98 | step_microstep: 113.17
[2024-12-05 13:49:34,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4385.33 | bwd: 8647.08 | bwd_inner: 8633.25 | bwd_allreduce: 12.68 | step: 114.44

  7%|▋         | 26/384 [06:31<1:27:39, 14.69s/it]
                                                  
{'loss': 0.2889, 'grad_norm': 2.642413377761841, 'learning_rate': 9.965093687129667e-07, 'epoch': 0.14}

  7%|▋         | 26/384 [06:31<1:27:39, 14.69s/it][2024-12-05 13:49:35,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.83 | bwd_microstep: 395.22 | bwd_inner_microstep: 395.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:36,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.51 | bwd_microstep: 558.71 | bwd_inner_microstep: 558.58 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:49:37,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.09 | bwd_microstep: 454.14 | bwd_inner_microstep: 453.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:38,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.15 | bwd_microstep: 678.54 | bwd_inner_microstep: 678.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:49:39,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.67 | bwd_microstep: 410.56 | bwd_inner_microstep: 410.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:40,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.62 | bwd_microstep: 543.43 | bwd_inner_microstep: 543.31 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:49:40,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.06 | bwd_microstep: 464.94 | bwd_inner_microstep: 464.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:41,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.54 | bwd_microstep: 509.67 | bwd_inner_microstep: 509.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:42,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.73 | bwd_microstep: 416.85 | bwd_inner_microstep: 416.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:49:43,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.61 | bwd_microstep: 561.97 | bwd_inner_microstep: 561.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:49:44,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.13 | bwd_microstep: 486.07 | bwd_inner_microstep: 485.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:45,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.43 | bwd_microstep: 434.45 | bwd_inner_microstep: 434.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:45,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.25 | bwd_microstep: 440.83 | bwd_inner_microstep: 440.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:46,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.52 | bwd_microstep: 471.51 | bwd_inner_microstep: 471.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:49:47,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.98 | bwd_microstep: 485.24 | bwd_inner_microstep: 485.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:49:49,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.29 | optimizer_gradients: 14.28 | optimizer_step: 23.29
[2024-12-05 13:49:49,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.04 | bwd_microstep: 1178.99 | bwd_inner_microstep: 1166.39 | bwd_allreduce_microstep: 12.50 | step_microstep: 113.49
[2024-12-05 13:49:49,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4491.25 | bwd: 8491.12 | bwd_inner: 8476.85 | bwd_allreduce: 13.16 | step: 114.59

  7%|▋         | 27/384 [06:45<1:27:05, 14.64s/it]
                                                  
{'loss': 0.2778, 'grad_norm': 2.513338565826416, 'learning_rate': 9.959935885253715e-07, 'epoch': 0.14}

  7%|▋         | 27/384 [06:45<1:27:05, 14.64s/it][2024-12-05 13:49:50,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.28 | bwd_microstep: 411.30 | bwd_inner_microstep: 411.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:49:50,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.70 | bwd_microstep: 436.92 | bwd_inner_microstep: 436.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:49:51,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.44 | bwd_microstep: 425.14 | bwd_inner_microstep: 425.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:49:52,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.39 | bwd_microstep: 626.69 | bwd_inner_microstep: 626.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:49:53,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.31 | bwd_microstep: 461.81 | bwd_inner_microstep: 461.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:49:54,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.81 | bwd_microstep: 410.83 | bwd_inner_microstep: 410.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:49:55,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 358.73 | bwd_microstep: 759.80 | bwd_inner_microstep: 759.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:49:56,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.02 | bwd_microstep: 460.41 | bwd_inner_microstep: 460.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:49:57,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.14 | bwd_microstep: 562.25 | bwd_inner_microstep: 562.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.13
[2024-12-05 13:49:57,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.17 | bwd_microstep: 435.87 | bwd_inner_microstep: 435.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:49:58,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.81 | bwd_microstep: 465.76 | bwd_inner_microstep: 465.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:49:59,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.80 | bwd_microstep: 427.97 | bwd_inner_microstep: 427.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:50:00,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.88 | bwd_microstep: 510.27 | bwd_inner_microstep: 510.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:50:01,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.88 | bwd_microstep: 474.77 | bwd_inner_microstep: 474.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:50:01,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.52 | bwd_microstep: 436.29 | bwd_inner_microstep: 436.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:50:03,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.60 | optimizer_gradients: 14.08 | optimizer_step: 23.25
[2024-12-05 13:50:03,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.24 | bwd_microstep: 1204.79 | bwd_inner_microstep: 1192.74 | bwd_allreduce_microstep: 11.96 | step_microstep: 112.89
[2024-12-05 13:50:03,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4415.19 | bwd: 8510.89 | bwd_inner: 8497.10 | bwd_allreduce: 12.61 | step: 114.22

  7%|▋         | 28/384 [06:59<1:26:17, 14.54s/it]
                                                  
{'loss': 0.2311, 'grad_norm': 2.2585127353668213, 'learning_rate': 9.954424340791195e-07, 'epoch': 0.15}

  7%|▋         | 28/384 [06:59<1:26:17, 14.54s/it][2024-12-05 13:50:04,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.04 | bwd_microstep: 514.56 | bwd_inner_microstep: 514.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:05,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.83 | bwd_microstep: 452.76 | bwd_inner_microstep: 452.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:06,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.76 | bwd_microstep: 469.35 | bwd_inner_microstep: 469.22 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:50:07,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.24 | bwd_microstep: 571.49 | bwd_inner_microstep: 571.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:50:07,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.56 | bwd_microstep: 426.40 | bwd_inner_microstep: 426.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:50:08,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.07 | bwd_microstep: 406.32 | bwd_inner_microstep: 406.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:50:09,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.90 | bwd_microstep: 534.06 | bwd_inner_microstep: 533.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:50:10,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 363.43 | bwd_microstep: 688.41 | bwd_inner_microstep: 688.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:11,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.17 | bwd_microstep: 551.80 | bwd_inner_microstep: 551.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:12,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.03 | bwd_microstep: 418.99 | bwd_inner_microstep: 418.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:13,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.20 | bwd_microstep: 541.18 | bwd_inner_microstep: 541.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:14,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.88 | bwd_microstep: 408.22 | bwd_inner_microstep: 408.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:14,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.63 | bwd_microstep: 427.57 | bwd_inner_microstep: 427.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:15,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.92 | bwd_microstep: 395.92 | bwd_inner_microstep: 395.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:50:16,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.61 | bwd_microstep: 535.67 | bwd_inner_microstep: 535.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:18,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.18 | optimizer_gradients: 14.03 | optimizer_step: 23.27
[2024-12-05 13:50:18,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.97 | bwd_microstep: 1508.54 | bwd_inner_microstep: 1496.60 | bwd_allreduce_microstep: 11.85 | step_microstep: 111.84
[2024-12-05 13:50:18,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4556.30 | bwd: 8851.21 | bwd_inner: 8837.64 | bwd_allreduce: 12.50 | step: 112.96

  8%|▊         | 29/384 [07:14<1:26:36, 14.64s/it]
                                                  
{'loss': 0.2403, 'grad_norm': 2.021958351135254, 'learning_rate': 9.948559446825411e-07, 'epoch': 0.15}

  8%|▊         | 29/384 [07:14<1:26:36, 14.64s/it][2024-12-05 13:50:19,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.80 | bwd_microstep: 563.67 | bwd_inner_microstep: 563.50 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 13:50:20,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.46 | bwd_microstep: 421.80 | bwd_inner_microstep: 421.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:21,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.23 | bwd_microstep: 695.84 | bwd_inner_microstep: 695.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:22,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.65 | bwd_microstep: 457.22 | bwd_inner_microstep: 457.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:50:23,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.98 | bwd_microstep: 709.45 | bwd_inner_microstep: 709.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:24,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.55 | bwd_microstep: 488.73 | bwd_inner_microstep: 488.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:25,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.32 | bwd_microstep: 622.07 | bwd_inner_microstep: 621.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:50:25,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.27 | bwd_microstep: 461.75 | bwd_inner_microstep: 461.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:26,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.30 | bwd_microstep: 458.45 | bwd_inner_microstep: 458.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:50:27,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.47 | bwd_microstep: 357.79 | bwd_inner_microstep: 357.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:50:28,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.92 | bwd_microstep: 445.10 | bwd_inner_microstep: 444.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:50:29,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 384.06 | bwd_microstep: 841.01 | bwd_inner_microstep: 840.87 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:50:30,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.68 | bwd_microstep: 456.67 | bwd_inner_microstep: 456.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:50:31,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 366.60 | bwd_microstep: 744.32 | bwd_inner_microstep: 744.14 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 13:50:32,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.70 | bwd_microstep: 442.61 | bwd_inner_microstep: 442.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:50:33,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.91 | optimizer_gradients: 14.05 | optimizer_step: 23.37
[2024-12-05 13:50:33,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.28 | bwd_microstep: 1136.51 | bwd_inner_microstep: 1124.30 | bwd_allreduce_microstep: 12.11 | step_microstep: 113.66
[2024-12-05 13:50:33,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4703.34 | bwd: 9302.97 | bwd_inner: 9289.08 | bwd_allreduce: 12.79 | step: 114.92

  8%|▊         | 30/384 [07:30<1:27:45, 14.87s/it]
                                                  
{'loss': 0.2984, 'grad_norm': 2.938659429550171, 'learning_rate': 9.942341621640557e-07, 'epoch': 0.16}

  8%|▊         | 30/384 [07:30<1:27:45, 14.87s/it][2024-12-05 13:50:34,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.08 | bwd_microstep: 376.01 | bwd_inner_microstep: 375.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:50:35,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.93 | bwd_microstep: 341.07 | bwd_inner_microstep: 340.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:50:36,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.18 | bwd_microstep: 404.76 | bwd_inner_microstep: 404.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:50:37,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.85 | bwd_microstep: 593.97 | bwd_inner_microstep: 593.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:50:37,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.19 | bwd_microstep: 338.30 | bwd_inner_microstep: 338.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:50:38,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.62 | bwd_microstep: 442.11 | bwd_inner_microstep: 441.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:50:39,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.59 | bwd_microstep: 405.00 | bwd_inner_microstep: 404.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:50:40,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.97 | bwd_microstep: 434.75 | bwd_inner_microstep: 434.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:50:41,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.13 | bwd_microstep: 591.20 | bwd_inner_microstep: 591.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:50:41,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.27 | bwd_microstep: 405.65 | bwd_inner_microstep: 405.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:50:42,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.92 | bwd_microstep: 437.01 | bwd_inner_microstep: 436.86 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 13:50:43,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.25 | bwd_microstep: 431.79 | bwd_inner_microstep: 431.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:44,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.40 | bwd_microstep: 454.31 | bwd_inner_microstep: 454.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:50:44,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.68 | bwd_microstep: 359.23 | bwd_inner_microstep: 359.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:45,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.48 | bwd_microstep: 591.96 | bwd_inner_microstep: 591.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:49,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.49 | optimizer_gradients: 14.07 | optimizer_step: 23.09
[2024-12-05 13:50:49,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.68 | bwd_microstep: 2665.65 | bwd_inner_microstep: 2653.44 | bwd_allreduce_microstep: 12.11 | step_microstep: 112.15
[2024-12-05 13:50:49,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4243.27 | bwd: 9272.75 | bwd_inner: 9258.90 | bwd_allreduce: 12.77 | step: 113.44

  8%|▊         | 31/384 [07:45<1:27:59, 14.96s/it]
                                                  
{'loss': 0.2607, 'grad_norm': 1.9811211824417114, 'learning_rate': 9.93577130869187e-07, 'epoch': 0.16}

  8%|▊         | 31/384 [07:45<1:27:59, 14.96s/it][2024-12-05 13:50:49,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.43 | bwd_microstep: 418.33 | bwd_inner_microstep: 418.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:50:50,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.39 | bwd_microstep: 406.91 | bwd_inner_microstep: 406.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:50:51,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.40 | bwd_microstep: 570.90 | bwd_inner_microstep: 570.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:50:52,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.01 | bwd_microstep: 485.92 | bwd_inner_microstep: 485.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:50:53,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.29 | bwd_microstep: 439.27 | bwd_inner_microstep: 439.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:50:53,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.59 | bwd_microstep: 462.08 | bwd_inner_microstep: 461.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:54,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.30 | bwd_microstep: 408.58 | bwd_inner_microstep: 408.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:55,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.78 | bwd_microstep: 569.98 | bwd_inner_microstep: 569.82 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 13:50:56,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.59 | bwd_microstep: 453.10 | bwd_inner_microstep: 452.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:50:57,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.43 | bwd_microstep: 628.79 | bwd_inner_microstep: 628.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:50:58,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.89 | bwd_microstep: 543.32 | bwd_inner_microstep: 543.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:50:59,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.32 | bwd_microstep: 527.30 | bwd_inner_microstep: 527.16 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:51:00,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 351.28 | bwd_microstep: 726.73 | bwd_inner_microstep: 726.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:51:01,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 238.89 | bwd_microstep: 407.04 | bwd_inner_microstep: 406.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:51:02,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.14 | bwd_microstep: 571.73 | bwd_inner_microstep: 571.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 13:51:03,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.27 | optimizer_gradients: 14.07 | optimizer_step: 23.28
[2024-12-05 13:51:03,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.10 | bwd_microstep: 1005.31 | bwd_inner_microstep: 993.04 | bwd_allreduce_microstep: 12.17 | step_microstep: 112.14
[2024-12-05 13:51:03,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4550.90 | bwd: 8625.28 | bwd_inner: 8611.36 | bwd_allreduce: 12.84 | step: 113.46

  8%|▊         | 32/384 [07:59<1:27:10, 14.86s/it]
                                                  
{'loss': 0.2236, 'grad_norm': 1.8804433345794678, 'learning_rate': 9.928848976574018e-07, 'epoch': 0.17}

  8%|▊         | 32/384 [07:59<1:27:10, 14.86s/it][2024-12-05 13:51:04,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.08 | bwd_microstep: 400.88 | bwd_inner_microstep: 400.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:51:05,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.13 | bwd_microstep: 450.15 | bwd_inner_microstep: 450.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:51:06,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.76 | bwd_microstep: 453.10 | bwd_inner_microstep: 452.95 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:51:06,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.30 | bwd_microstep: 438.00 | bwd_inner_microstep: 437.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:51:07,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.73 | bwd_microstep: 492.88 | bwd_inner_microstep: 492.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:51:08,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.63 | bwd_microstep: 457.19 | bwd_inner_microstep: 457.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:51:09,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.53 | bwd_microstep: 433.06 | bwd_inner_microstep: 432.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:51:10,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 395.91 | bwd_microstep: 877.99 | bwd_inner_microstep: 877.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:51:11,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.56 | bwd_microstep: 596.15 | bwd_inner_microstep: 596.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:51:12,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.14 | bwd_microstep: 411.85 | bwd_inner_microstep: 411.71 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:51:13,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.90 | bwd_microstep: 502.79 | bwd_inner_microstep: 502.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:51:14,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 342.80 | bwd_microstep: 695.76 | bwd_inner_microstep: 695.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:51:15,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.25 | bwd_microstep: 495.21 | bwd_inner_microstep: 495.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:51:15,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.65 | bwd_microstep: 429.80 | bwd_inner_microstep: 429.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:51:16,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.33 | bwd_microstep: 457.65 | bwd_inner_microstep: 457.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:51:18,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.18 | optimizer_gradients: 14.07 | optimizer_step: 23.45
[2024-12-05 13:51:18,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.75 | bwd_microstep: 1208.73 | bwd_inner_microstep: 1196.42 | bwd_allreduce_microstep: 12.21 | step_microstep: 112.17
[2024-12-05 13:51:18,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4536.51 | bwd: 8801.16 | bwd_inner: 8787.13 | bwd_allreduce: 12.86 | step: 113.41

  9%|▊         | 33/384 [08:14<1:26:32, 14.79s/it]
                                                  
{'loss': 0.247, 'grad_norm': 2.2095425128936768, 'learning_rate': 9.921575118987671e-07, 'epoch': 0.17}

  9%|▊         | 33/384 [08:14<1:26:32, 14.79s/it][2024-12-05 13:51:19,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.85 | bwd_microstep: 423.02 | bwd_inner_microstep: 422.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:51:19,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.24 | bwd_microstep: 449.60 | bwd_inner_microstep: 449.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:51:20,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.40 | bwd_microstep: 603.08 | bwd_inner_microstep: 602.94 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:51:21,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.09 | bwd_microstep: 436.95 | bwd_inner_microstep: 436.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:51:22,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.65 | bwd_microstep: 356.97 | bwd_inner_microstep: 356.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:51:23,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.58 | bwd_microstep: 428.38 | bwd_inner_microstep: 428.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:51:23,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.19 | bwd_microstep: 435.28 | bwd_inner_microstep: 435.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:51:24,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.16 | bwd_microstep: 537.58 | bwd_inner_microstep: 537.44 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:51:25,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.57 | bwd_microstep: 535.99 | bwd_inner_microstep: 535.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:51:26,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.57 | bwd_microstep: 442.85 | bwd_inner_microstep: 442.70 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:51:27,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 341.25 | bwd_microstep: 697.79 | bwd_inner_microstep: 697.65 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:51:28,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.90 | bwd_microstep: 419.10 | bwd_inner_microstep: 418.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:51:29,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.40 | bwd_microstep: 407.63 | bwd_inner_microstep: 407.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:51:30,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.97 | bwd_microstep: 594.72 | bwd_inner_microstep: 594.57 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 13:51:30,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.26 | bwd_microstep: 428.27 | bwd_inner_microstep: 428.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:51:33,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.46 | optimizer_gradients: 14.09 | optimizer_step: 23.17
[2024-12-05 13:51:33,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.67 | bwd_microstep: 2530.50 | bwd_inner_microstep: 2518.52 | bwd_allreduce_microstep: 11.90 | step_microstep: 112.23
[2024-12-05 13:51:33,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4394.80 | bwd: 9727.73 | bwd_inner: 9713.99 | bwd_allreduce: 12.59 | step: 113.53

  9%|▉         | 34/384 [08:30<1:27:25, 14.99s/it]
                                                  
{'loss': 0.2519, 'grad_norm': 2.0361135005950928, 'learning_rate': 9.91395025470429e-07, 'epoch': 0.18}

  9%|▉         | 34/384 [08:30<1:27:25, 14.99s/it][2024-12-05 13:51:34,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.35 | bwd_microstep: 380.73 | bwd_inner_microstep: 380.59 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:51:35,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.79 | bwd_microstep: 374.61 | bwd_inner_microstep: 374.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:51:36,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.40 | bwd_microstep: 543.34 | bwd_inner_microstep: 543.21 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:51:37,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.40 | bwd_microstep: 566.06 | bwd_inner_microstep: 565.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:51:37,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.87 | bwd_microstep: 426.79 | bwd_inner_microstep: 426.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:51:38,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.42 | bwd_microstep: 432.49 | bwd_inner_microstep: 432.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:51:39,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.27 | bwd_microstep: 495.30 | bwd_inner_microstep: 495.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:51:40,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.45 | bwd_microstep: 590.60 | bwd_inner_microstep: 590.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:51:41,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.58 | bwd_microstep: 440.53 | bwd_inner_microstep: 440.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:51:41,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.00 | bwd_microstep: 355.71 | bwd_inner_microstep: 355.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:51:42,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.21 | bwd_microstep: 411.46 | bwd_inner_microstep: 411.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:51:43,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.94 | bwd_microstep: 485.79 | bwd_inner_microstep: 485.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:51:44,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.28 | bwd_microstep: 552.15 | bwd_inner_microstep: 552.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:51:45,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.37 | bwd_microstep: 621.67 | bwd_inner_microstep: 621.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:51:46,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.97 | bwd_microstep: 410.26 | bwd_inner_microstep: 410.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:51:48,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.48 | optimizer_gradients: 14.11 | optimizer_step: 23.27
[2024-12-05 13:51:48,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.66 | bwd_microstep: 1846.13 | bwd_inner_microstep: 1834.15 | bwd_allreduce_microstep: 11.90 | step_microstep: 112.41
[2024-12-05 13:51:48,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4424.03 | bwd: 8933.59 | bwd_inner: 8920.00 | bwd_allreduce: 12.53 | step: 113.52

  9%|▉         | 35/384 [08:44<1:26:30, 14.87s/it]
                                                  
{'loss': 0.2387, 'grad_norm': 1.9882704019546509, 'learning_rate': 9.905974927529133e-07, 'epoch': 0.18}

  9%|▉         | 35/384 [08:44<1:26:30, 14.87s/it][2024-12-05 13:51:49,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.23 | bwd_microstep: 400.65 | bwd_inner_microstep: 400.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:51:50,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.93 | bwd_microstep: 432.69 | bwd_inner_microstep: 432.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:51:50,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.55 | bwd_microstep: 434.00 | bwd_inner_microstep: 433.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:51:51,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 348.10 | bwd_microstep: 712.42 | bwd_inner_microstep: 712.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:51:52,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.75 | bwd_microstep: 406.55 | bwd_inner_microstep: 406.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:51:53,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.80 | bwd_microstep: 399.74 | bwd_inner_microstep: 399.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:51:54,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 346.32 | bwd_microstep: 706.27 | bwd_inner_microstep: 706.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:51:55,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.29 | bwd_microstep: 433.87 | bwd_inner_microstep: 433.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:51:56,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.57 | bwd_microstep: 451.68 | bwd_inner_microstep: 451.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:51:57,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.26 | bwd_microstep: 460.86 | bwd_inner_microstep: 460.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:51:57,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.96 | bwd_microstep: 447.97 | bwd_inner_microstep: 447.81 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.06
[2024-12-05 13:51:58,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.04 | bwd_microstep: 401.03 | bwd_inner_microstep: 400.91 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 13:51:59,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.67 | bwd_microstep: 485.87 | bwd_inner_microstep: 485.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:52:00,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.06 | bwd_microstep: 452.76 | bwd_inner_microstep: 452.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:52:01,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.90 | bwd_microstep: 435.18 | bwd_inner_microstep: 435.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:52:03,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.47 | optimizer_gradients: 14.11 | optimizer_step: 23.33
[2024-12-05 13:52:03,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.17 | bwd_microstep: 1806.19 | bwd_inner_microstep: 1793.95 | bwd_allreduce_microstep: 12.14 | step_microstep: 112.39
[2024-12-05 13:52:03,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4393.66 | bwd: 8867.72 | bwd_inner: 8853.84 | bwd_allreduce: 12.75 | step: 113.35

  9%|▉         | 36/384 [08:59<1:26:16, 14.87s/it]
                                                  
{'loss': 0.2506, 'grad_norm': 2.1970348358154297, 'learning_rate': 9.897649706262473e-07, 'epoch': 0.19}

  9%|▉         | 36/384 [08:59<1:26:16, 14.87s/it][2024-12-05 13:52:04,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.11 | bwd_microstep: 647.08 | bwd_inner_microstep: 646.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:52:05,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.34 | bwd_microstep: 534.73 | bwd_inner_microstep: 534.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:52:06,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.04 | bwd_microstep: 441.39 | bwd_inner_microstep: 441.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:52:06,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.85 | bwd_microstep: 451.07 | bwd_inner_microstep: 450.89 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 13:52:07,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.41 | bwd_microstep: 456.54 | bwd_inner_microstep: 456.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:52:08,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 405.65 | bwd_microstep: 904.56 | bwd_inner_microstep: 904.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:52:10,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 356.91 | bwd_microstep: 730.64 | bwd_inner_microstep: 730.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:52:10,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.66 | bwd_microstep: 440.19 | bwd_inner_microstep: 440.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:52:11,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.33 | bwd_microstep: 434.45 | bwd_inner_microstep: 434.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:52:12,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.68 | bwd_microstep: 412.17 | bwd_inner_microstep: 412.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:52:13,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.16 | bwd_microstep: 426.33 | bwd_inner_microstep: 426.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:52:13,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.64 | bwd_microstep: 404.19 | bwd_inner_microstep: 404.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:52:14,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.76 | bwd_microstep: 426.46 | bwd_inner_microstep: 426.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:52:15,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.66 | bwd_microstep: 450.63 | bwd_inner_microstep: 450.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:52:16,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.31 | bwd_microstep: 401.14 | bwd_inner_microstep: 401.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:52:18,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.57 | optimizer_gradients: 14.00 | optimizer_step: 24.00
[2024-12-05 13:52:18,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.64 | bwd_microstep: 1481.22 | bwd_inner_microstep: 1469.28 | bwd_allreduce_microstep: 11.85 | step_microstep: 111.89
[2024-12-05 13:52:18,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4553.21 | bwd: 9042.81 | bwd_inner: 9029.25 | bwd_allreduce: 12.49 | step: 112.99

 10%|▉         | 37/384 [09:14<1:26:07, 14.89s/it]
                                                  
{'loss': 0.2491, 'grad_norm': 1.9535129070281982, 'learning_rate': 9.888975184659016e-07, 'epoch': 0.19}

 10%|▉         | 37/384 [09:14<1:26:07, 14.89s/it][2024-12-05 13:52:18,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.28 | bwd_microstep: 423.60 | bwd_inner_microstep: 423.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:52:19,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 328.69 | bwd_microstep: 619.03 | bwd_inner_microstep: 618.86 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 13:52:20,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.93 | bwd_microstep: 447.67 | bwd_inner_microstep: 447.50 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.13
[2024-12-05 13:52:21,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.39 | bwd_microstep: 410.57 | bwd_inner_microstep: 410.39 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.12
[2024-12-05 13:52:22,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.76 | bwd_microstep: 531.67 | bwd_inner_microstep: 531.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:52:23,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.23 | bwd_microstep: 407.10 | bwd_inner_microstep: 406.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:52:23,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.99 | bwd_microstep: 431.15 | bwd_inner_microstep: 431.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:52:24,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.95 | bwd_microstep: 448.87 | bwd_inner_microstep: 448.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 13:52:25,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.39 | bwd_microstep: 581.52 | bwd_inner_microstep: 581.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 13:52:26,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 350.56 | bwd_microstep: 706.24 | bwd_inner_microstep: 706.09 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 13:52:27,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.08 | bwd_microstep: 578.68 | bwd_inner_microstep: 578.54 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.11
[2024-12-05 13:52:28,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.29 | bwd_microstep: 553.73 | bwd_inner_microstep: 553.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:52:29,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.22 | bwd_microstep: 483.39 | bwd_inner_microstep: 483.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 13:52:30,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.35 | bwd_microstep: 406.79 | bwd_inner_microstep: 406.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:52:31,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.78 | bwd_microstep: 586.33 | bwd_inner_microstep: 586.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:52:33,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.16 | optimizer_gradients: 14.07 | optimizer_step: 23.13
[2024-12-05 13:52:33,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.42 | bwd_microstep: 1691.57 | bwd_inner_microstep: 1679.30 | bwd_allreduce_microstep: 12.14 | step_microstep: 113.89
[2024-12-05 13:52:33,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4608.36 | bwd: 9307.95 | bwd_inner: 9293.87 | bwd_allreduce: 12.88 | step: 115.41

 10%|▉         | 38/384 [09:29<1:26:23, 14.98s/it]
                                                  
{'loss': 0.2002, 'grad_norm': 1.7368297576904297, 'learning_rate': 9.879951981385577e-07, 'epoch': 0.2}

 10%|▉         | 38/384 [09:29<1:26:23, 14.98s/it][2024-12-05 13:52:34,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.51 | bwd_microstep: 374.45 | bwd_inner_microstep: 374.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:52:34,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.43 | bwd_microstep: 333.47 | bwd_inner_microstep: 333.34 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:52:35,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.78 | bwd_microstep: 448.43 | bwd_inner_microstep: 448.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:52:36,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.81 | bwd_microstep: 488.36 | bwd_inner_microstep: 488.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:52:37,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.75 | bwd_microstep: 431.89 | bwd_inner_microstep: 431.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:52:38,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.17 | bwd_microstep: 595.65 | bwd_inner_microstep: 595.51 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:52:39,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.40 | bwd_microstep: 568.68 | bwd_inner_microstep: 568.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:52:39,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.69 | bwd_microstep: 455.36 | bwd_inner_microstep: 455.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:52:40,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.39 | bwd_microstep: 434.64 | bwd_inner_microstep: 434.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:52:41,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.58 | bwd_microstep: 463.56 | bwd_inner_microstep: 463.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:52:42,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.86 | bwd_microstep: 461.22 | bwd_inner_microstep: 461.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:52:43,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.05 | bwd_microstep: 435.44 | bwd_inner_microstep: 435.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:52:44,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.18 | bwd_microstep: 567.66 | bwd_inner_microstep: 567.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:52:45,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 343.38 | bwd_microstep: 694.37 | bwd_inner_microstep: 694.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:52:46,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.45 | bwd_microstep: 590.17 | bwd_inner_microstep: 590.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:52:47,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.71 | optimizer_gradients: 14.04 | optimizer_step: 23.10
[2024-12-05 13:52:47,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.63 | bwd_microstep: 1113.86 | bwd_inner_microstep: 1101.92 | bwd_allreduce_microstep: 11.85 | step_microstep: 112.27
[2024-12-05 13:52:47,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4498.14 | bwd: 8457.23 | bwd_inner: 8443.62 | bwd_allreduce: 12.48 | step: 113.41

 10%|█         | 39/384 [09:43<1:25:00, 14.79s/it]
                                                  
{'loss': 0.2107, 'grad_norm': 1.8394443988800049, 'learning_rate': 9.870580739976935e-07, 'epoch': 0.2}

 10%|█         | 39/384 [09:43<1:25:00, 14.79s/it][2024-12-05 13:52:48,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.29 | bwd_microstep: 453.26 | bwd_inner_microstep: 453.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:52:49,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.02 | bwd_microstep: 598.57 | bwd_inner_microstep: 598.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:52:50,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.76 | bwd_microstep: 404.20 | bwd_inner_microstep: 404.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:52:51,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.20 | bwd_microstep: 442.50 | bwd_inner_microstep: 442.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:52:51,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.85 | bwd_microstep: 484.52 | bwd_inner_microstep: 484.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:52:52,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.26 | bwd_microstep: 428.35 | bwd_inner_microstep: 428.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:52:53,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.76 | bwd_microstep: 482.79 | bwd_inner_microstep: 482.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:52:54,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.28 | bwd_microstep: 440.01 | bwd_inner_microstep: 439.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:52:55,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.01 | bwd_microstep: 438.40 | bwd_inner_microstep: 438.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:52:56,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 369.47 | bwd_microstep: 775.45 | bwd_inner_microstep: 775.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:52:57,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.63 | bwd_microstep: 493.76 | bwd_inner_microstep: 493.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:52:58,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.16 | bwd_microstep: 591.14 | bwd_inner_microstep: 591.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:52:58,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.57 | bwd_microstep: 484.58 | bwd_inner_microstep: 484.44 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 13:52:59,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 325.90 | bwd_microstep: 634.60 | bwd_inner_microstep: 634.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:00,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.25 | bwd_microstep: 484.08 | bwd_inner_microstep: 483.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:53:02,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.36 | optimizer_gradients: 14.04 | optimizer_step: 23.07
[2024-12-05 13:53:02,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.38 | bwd_microstep: 1497.37 | bwd_inner_microstep: 1485.18 | bwd_allreduce_microstep: 12.09 | step_microstep: 111.91
[2024-12-05 13:53:02,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4623.84 | bwd: 9133.58 | bwd_inner: 9119.71 | bwd_allreduce: 12.73 | step: 113.11

 10%|█         | 40/384 [09:58<1:25:12, 14.86s/it]
                                                  
{'loss': 0.2373, 'grad_norm': 2.0104002952575684, 'learning_rate': 9.860862128789952e-07, 'epoch': 0.21}

 10%|█         | 40/384 [09:58<1:25:12, 14.86s/it][2024-12-05 13:53:03,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.49 | bwd_microstep: 324.63 | bwd_inner_microstep: 324.51 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 13:53:04,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.77 | bwd_microstep: 435.55 | bwd_inner_microstep: 435.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:53:05,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.59 | bwd_microstep: 466.16 | bwd_inner_microstep: 466.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:53:05,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.00 | bwd_microstep: 452.26 | bwd_inner_microstep: 452.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:53:06,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.14 | bwd_microstep: 417.88 | bwd_inner_microstep: 417.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:53:07,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.54 | bwd_microstep: 457.70 | bwd_inner_microstep: 457.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:53:08,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.58 | bwd_microstep: 440.24 | bwd_inner_microstep: 440.10 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:53:09,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.67 | bwd_microstep: 566.19 | bwd_inner_microstep: 566.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:09,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.73 | bwd_microstep: 457.63 | bwd_inner_microstep: 457.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 13:53:10,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.66 | bwd_microstep: 416.58 | bwd_inner_microstep: 416.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:11,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.06 | bwd_microstep: 544.43 | bwd_inner_microstep: 544.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:12,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.71 | bwd_microstep: 492.45 | bwd_inner_microstep: 492.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:13,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.01 | bwd_microstep: 611.73 | bwd_inner_microstep: 611.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:14,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 238.83 | bwd_microstep: 359.15 | bwd_inner_microstep: 359.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:14,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.47 | bwd_microstep: 437.51 | bwd_inner_microstep: 437.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:53:17,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.10 | optimizer_gradients: 14.07 | optimizer_step: 23.44
[2024-12-05 13:53:17,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.96 | bwd_microstep: 2058.88 | bwd_inner_microstep: 2046.85 | bwd_allreduce_microstep: 11.93 | step_microstep: 112.10
[2024-12-05 13:53:17,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4308.27 | bwd: 8938.95 | bwd_inner: 8925.26 | bwd_allreduce: 12.58 | step: 113.28

 11%|█         | 41/384 [10:13<1:24:43, 14.82s/it]
                                                  
{'loss': 0.1881, 'grad_norm': 1.7731590270996094, 'learning_rate': 9.850796840955899e-07, 'epoch': 0.21}

 11%|█         | 41/384 [10:13<1:24:43, 14.82s/it][2024-12-05 13:53:18,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.00 | bwd_microstep: 370.96 | bwd_inner_microstep: 370.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:19,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.43 | bwd_microstep: 590.10 | bwd_inner_microstep: 589.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:19,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.43 | bwd_microstep: 448.81 | bwd_inner_microstep: 448.67 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 13:53:21,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 370.17 | bwd_microstep: 832.70 | bwd_inner_microstep: 832.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:53:22,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.21 | bwd_microstep: 537.36 | bwd_inner_microstep: 537.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:22,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.62 | bwd_microstep: 406.32 | bwd_inner_microstep: 406.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:23,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.46 | bwd_microstep: 403.84 | bwd_inner_microstep: 403.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:53:24,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.82 | bwd_microstep: 358.33 | bwd_inner_microstep: 358.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:25,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.24 | bwd_microstep: 402.94 | bwd_inner_microstep: 402.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:25,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.36 | bwd_microstep: 435.04 | bwd_inner_microstep: 434.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:53:26,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.09 | bwd_microstep: 488.95 | bwd_inner_microstep: 488.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:53:27,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.83 | bwd_microstep: 410.02 | bwd_inner_microstep: 409.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:53:28,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.01 | bwd_microstep: 416.38 | bwd_inner_microstep: 416.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:53:28,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.41 | bwd_microstep: 452.03 | bwd_inner_microstep: 451.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:53:29,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.05 | bwd_microstep: 460.55 | bwd_inner_microstep: 460.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:53:31,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.23 | optimizer_gradients: 14.02 | optimizer_step: 23.29
[2024-12-05 13:53:31,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.61 | bwd_microstep: 1631.53 | bwd_inner_microstep: 1619.59 | bwd_allreduce_microstep: 11.84 | step_microstep: 112.50
[2024-12-05 13:53:31,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4371.81 | bwd: 8645.85 | bwd_inner: 8632.31 | bwd_allreduce: 12.47 | step: 113.49

 11%|█         | 42/384 [10:28<1:23:44, 14.69s/it]
                                                  
{'loss': 0.2238, 'grad_norm': 1.937873363494873, 'learning_rate': 9.840385594331022e-07, 'epoch': 0.22}

 11%|█         | 42/384 [10:28<1:23:44, 14.69s/it][2024-12-05 13:53:32,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.02 | bwd_microstep: 586.27 | bwd_inner_microstep: 586.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:53:33,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.81 | bwd_microstep: 449.33 | bwd_inner_microstep: 449.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:34,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.63 | bwd_microstep: 509.13 | bwd_inner_microstep: 509.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:53:35,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.25 | bwd_microstep: 428.23 | bwd_inner_microstep: 428.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:36,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.99 | bwd_microstep: 618.76 | bwd_inner_microstep: 618.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:53:37,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.22 | bwd_microstep: 452.87 | bwd_inner_microstep: 452.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:53:37,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.57 | bwd_microstep: 487.74 | bwd_inner_microstep: 487.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:38,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.77 | bwd_microstep: 589.54 | bwd_inner_microstep: 589.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:39,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.65 | bwd_microstep: 453.13 | bwd_inner_microstep: 453.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:40,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.72 | bwd_microstep: 427.11 | bwd_inner_microstep: 426.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:41,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.61 | bwd_microstep: 339.78 | bwd_inner_microstep: 339.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:42,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.27 | bwd_microstep: 532.93 | bwd_inner_microstep: 532.79 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 13:53:42,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.42 | bwd_microstep: 407.89 | bwd_inner_microstep: 407.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:53:43,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.79 | bwd_microstep: 442.73 | bwd_inner_microstep: 442.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:44,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.26 | bwd_microstep: 431.06 | bwd_inner_microstep: 430.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:46,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.18 | optimizer_gradients: 14.12 | optimizer_step: 23.23
[2024-12-05 13:53:46,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 350.68 | bwd_microstep: 1196.68 | bwd_inner_microstep: 1184.50 | bwd_allreduce_microstep: 12.08 | step_microstep: 112.37
[2024-12-05 13:53:46,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4524.73 | bwd: 8353.14 | bwd_inner: 8339.27 | bwd_allreduce: 12.73 | step: 113.41

 11%|█         | 43/384 [10:42<1:22:36, 14.53s/it]
                                                  
{'loss': 0.2237, 'grad_norm': 1.9435638189315796, 'learning_rate': 9.82962913144534e-07, 'epoch': 0.22}

 11%|█         | 43/384 [10:42<1:22:36, 14.53s/it][2024-12-05 13:53:47,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 325.76 | bwd_microstep: 594.84 | bwd_inner_microstep: 594.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:47,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.49 | bwd_microstep: 463.71 | bwd_inner_microstep: 463.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:48,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 342.36 | bwd_microstep: 678.88 | bwd_inner_microstep: 678.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:49,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.18 | bwd_microstep: 357.51 | bwd_inner_microstep: 357.39 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 13:53:50,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.32 | bwd_microstep: 532.58 | bwd_inner_microstep: 532.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:51,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.88 | bwd_microstep: 500.90 | bwd_inner_microstep: 500.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:52,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.18 | bwd_microstep: 454.03 | bwd_inner_microstep: 453.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:53,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.17 | bwd_microstep: 547.84 | bwd_inner_microstep: 547.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:53,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.93 | bwd_microstep: 452.93 | bwd_inner_microstep: 452.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:54,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.47 | bwd_microstep: 449.98 | bwd_inner_microstep: 449.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:55,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.92 | bwd_microstep: 456.28 | bwd_inner_microstep: 456.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:53:56,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.77 | bwd_microstep: 537.02 | bwd_inner_microstep: 536.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:57,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.61 | bwd_microstep: 457.47 | bwd_inner_microstep: 457.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:58,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.71 | bwd_microstep: 487.98 | bwd_inner_microstep: 487.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:53:59,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.23 | bwd_microstep: 533.39 | bwd_inner_microstep: 533.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:00,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.59 | optimizer_gradients: 14.10 | optimizer_step: 23.06
[2024-12-05 13:54:00,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.18 | bwd_microstep: 1029.15 | bwd_inner_microstep: 1016.93 | bwd_allreduce_microstep: 12.14 | step_microstep: 112.65
[2024-12-05 13:54:00,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4586.22 | bwd: 8534.49 | bwd_inner: 8520.62 | bwd_allreduce: 12.79 | step: 113.70

 11%|█▏        | 44/384 [10:56<1:22:23, 14.54s/it]
                                                  
{'loss': 0.1951, 'grad_norm': 1.786537766456604, 'learning_rate': 9.818528219449704e-07, 'epoch': 0.23}

 11%|█▏        | 44/384 [10:56<1:22:23, 14.54s/it][2024-12-05 13:54:01,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.02 | bwd_microstep: 402.40 | bwd_inner_microstep: 402.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:02,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.94 | bwd_microstep: 338.06 | bwd_inner_microstep: 337.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:54:03,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 325.07 | bwd_microstep: 622.61 | bwd_inner_microstep: 622.42 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.11
[2024-12-05 13:54:03,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.33 | bwd_microstep: 523.75 | bwd_inner_microstep: 523.58 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 13:54:04,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.97 | bwd_microstep: 543.12 | bwd_inner_microstep: 542.94 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 13:54:05,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.73 | bwd_microstep: 639.03 | bwd_inner_microstep: 638.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:06,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.01 | bwd_microstep: 595.55 | bwd_inner_microstep: 595.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:54:07,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.97 | bwd_microstep: 485.60 | bwd_inner_microstep: 485.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:08,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.38 | bwd_microstep: 451.15 | bwd_inner_microstep: 451.03 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:54:09,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.45 | bwd_microstep: 329.87 | bwd_inner_microstep: 329.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:54:09,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.73 | bwd_microstep: 453.04 | bwd_inner_microstep: 452.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:54:10,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.08 | bwd_microstep: 502.93 | bwd_inner_microstep: 502.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:54:11,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.85 | bwd_microstep: 432.09 | bwd_inner_microstep: 431.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:54:12,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 393.87 | bwd_microstep: 875.30 | bwd_inner_microstep: 875.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:54:13,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.50 | bwd_microstep: 560.82 | bwd_inner_microstep: 560.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:15,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.24 | optimizer_gradients: 14.05 | optimizer_step: 23.29
[2024-12-05 13:54:15,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.99 | bwd_microstep: 823.53 | bwd_inner_microstep: 811.19 | bwd_allreduce_microstep: 12.26 | step_microstep: 112.04
[2024-12-05 13:54:15,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4704.97 | bwd: 8578.86 | bwd_inner: 8564.70 | bwd_allreduce: 12.95 | step: 113.30

 12%|█▏        | 45/384 [11:11<1:22:15, 14.56s/it]
                                                  
{'loss': 0.2257, 'grad_norm': 1.7915631532669067, 'learning_rate': 9.807083650061062e-07, 'epoch': 0.23}

 12%|█▏        | 45/384 [11:11<1:22:15, 14.56s/it][2024-12-05 13:54:16,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.39 | bwd_microstep: 558.06 | bwd_inner_microstep: 557.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:54:17,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 370.41 | bwd_microstep: 748.11 | bwd_inner_microstep: 747.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:54:18,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.47 | bwd_microstep: 451.90 | bwd_inner_microstep: 451.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:19,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.05 | bwd_microstep: 549.07 | bwd_inner_microstep: 548.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:20,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.82 | bwd_microstep: 544.40 | bwd_inner_microstep: 544.27 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:54:21,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 350.42 | bwd_microstep: 713.61 | bwd_inner_microstep: 713.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:54:21,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.09 | bwd_microstep: 460.15 | bwd_inner_microstep: 460.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:22,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.11 | bwd_microstep: 532.20 | bwd_inner_microstep: 532.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:23,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.28 | bwd_microstep: 451.88 | bwd_inner_microstep: 451.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:24,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.29 | bwd_microstep: 464.34 | bwd_inner_microstep: 464.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:25,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.20 | bwd_microstep: 542.83 | bwd_inner_microstep: 542.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:26,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.69 | bwd_microstep: 437.17 | bwd_inner_microstep: 437.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:54:26,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.59 | bwd_microstep: 412.26 | bwd_inner_microstep: 412.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:54:27,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.74 | bwd_microstep: 619.66 | bwd_inner_microstep: 619.50 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 13:54:28,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.30 | bwd_microstep: 589.84 | bwd_inner_microstep: 589.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:54:30,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.32 | optimizer_gradients: 14.05 | optimizer_step: 23.09
[2024-12-05 13:54:30,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.33 | bwd_microstep: 902.20 | bwd_inner_microstep: 889.98 | bwd_allreduce_microstep: 12.14 | step_microstep: 113.41
[2024-12-05 13:54:30,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4677.23 | bwd: 8977.69 | bwd_inner: 8963.76 | bwd_allreduce: 12.81 | step: 114.68

 12%|█▏        | 46/384 [11:26<1:22:49, 14.70s/it]
                                                  
{'loss': 0.236, 'grad_norm': 1.798353910446167, 'learning_rate': 9.79529623950601e-07, 'epoch': 0.24}

 12%|█▏        | 46/384 [11:26<1:22:49, 14.70s/it][2024-12-05 13:54:31,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.86 | bwd_microstep: 411.98 | bwd_inner_microstep: 411.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:54:32,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.27 | bwd_microstep: 436.51 | bwd_inner_microstep: 436.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:54:33,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.82 | bwd_microstep: 493.90 | bwd_inner_microstep: 493.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:54:34,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.03 | bwd_microstep: 409.40 | bwd_inner_microstep: 409.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:54:34,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.10 | bwd_microstep: 417.41 | bwd_inner_microstep: 417.26 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:54:35,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.71 | bwd_microstep: 567.18 | bwd_inner_microstep: 567.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:54:36,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.31 | bwd_microstep: 433.89 | bwd_inner_microstep: 433.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:54:37,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.40 | bwd_microstep: 544.93 | bwd_inner_microstep: 544.79 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:54:38,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.58 | bwd_microstep: 429.94 | bwd_inner_microstep: 429.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:39,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.76 | bwd_microstep: 443.70 | bwd_inner_microstep: 443.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:40,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.14 | bwd_microstep: 619.32 | bwd_inner_microstep: 619.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:54:41,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.95 | bwd_microstep: 429.42 | bwd_inner_microstep: 429.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:54:42,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.48 | bwd_microstep: 585.16 | bwd_inner_microstep: 585.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:54:42,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.92 | bwd_microstep: 437.75 | bwd_inner_microstep: 437.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:54:43,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.07 | bwd_microstep: 437.45 | bwd_inner_microstep: 437.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:54:45,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.46 | optimizer_gradients: 14.05 | optimizer_step: 23.18
[2024-12-05 13:54:45,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.54 | bwd_microstep: 1684.61 | bwd_inner_microstep: 1672.40 | bwd_allreduce_microstep: 12.12 | step_microstep: 112.09
[2024-12-05 13:54:45,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4414.01 | bwd: 8782.53 | bwd_inner: 8768.63 | bwd_allreduce: 12.79 | step: 113.36

 12%|█▏        | 47/384 [11:41<1:23:56, 14.95s/it]
                                                  
{'loss': 0.2354, 'grad_norm': 1.872049331665039, 'learning_rate': 9.783166828462572e-07, 'epoch': 0.24}

 12%|█▏        | 47/384 [11:41<1:23:56, 14.95s/it][2024-12-05 13:54:46,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.05 | bwd_microstep: 592.80 | bwd_inner_microstep: 592.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:54:47,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.93 | bwd_microstep: 530.59 | bwd_inner_microstep: 530.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:48,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.33 | bwd_microstep: 588.32 | bwd_inner_microstep: 588.17 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 13:54:49,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.45 | bwd_microstep: 437.17 | bwd_inner_microstep: 437.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:50,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 345.92 | bwd_microstep: 719.02 | bwd_inner_microstep: 718.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:54:51,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.77 | bwd_microstep: 414.24 | bwd_inner_microstep: 414.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:54:52,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 355.05 | bwd_microstep: 710.00 | bwd_inner_microstep: 709.83 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:54:53,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.76 | bwd_microstep: 478.50 | bwd_inner_microstep: 478.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:53,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.58 | bwd_microstep: 415.66 | bwd_inner_microstep: 415.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:54,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.71 | bwd_microstep: 453.16 | bwd_inner_microstep: 453.01 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 13:54:55,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.00 | bwd_microstep: 433.41 | bwd_inner_microstep: 433.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:56,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.13 | bwd_microstep: 543.85 | bwd_inner_microstep: 543.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:54:57,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.03 | bwd_microstep: 439.46 | bwd_inner_microstep: 439.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:54:58,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.74 | bwd_microstep: 485.35 | bwd_inner_microstep: 485.22 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:54:59,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.71 | bwd_microstep: 717.71 | bwd_inner_microstep: 717.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:55:00,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.15 | optimizer_gradients: 14.17 | optimizer_step: 23.18
[2024-12-05 13:55:00,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 390.42 | bwd_microstep: 1065.25 | bwd_inner_microstep: 1053.12 | bwd_allreduce_microstep: 12.03 | step_microstep: 112.32
[2024-12-05 13:55:00,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4785.66 | bwd: 9024.47 | bwd_inner: 9010.69 | bwd_allreduce: 12.69 | step: 113.61

 12%|█▎        | 48/384 [11:57<1:23:50, 14.97s/it]
                                                  
{'loss': 0.2229, 'grad_norm': 1.879210114479065, 'learning_rate': 9.770696282000244e-07, 'epoch': 0.25}

 12%|█▎        | 48/384 [11:57<1:23:50, 14.97s/it][2024-12-05 13:55:01,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.74 | bwd_microstep: 404.29 | bwd_inner_microstep: 404.11 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:55:02,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.04 | bwd_microstep: 400.95 | bwd_inner_microstep: 400.80 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.06
[2024-12-05 13:55:03,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.82 | bwd_microstep: 483.72 | bwd_inner_microstep: 483.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:04,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 346.63 | bwd_microstep: 681.62 | bwd_inner_microstep: 681.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:04,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.72 | bwd_microstep: 365.57 | bwd_inner_microstep: 365.45 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 13:55:05,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.68 | bwd_microstep: 546.94 | bwd_inner_microstep: 546.80 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.12
[2024-12-05 13:55:06,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.92 | bwd_microstep: 438.35 | bwd_inner_microstep: 438.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:07,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.07 | bwd_microstep: 339.58 | bwd_inner_microstep: 339.46 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 13:55:08,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.48 | bwd_microstep: 430.00 | bwd_inner_microstep: 429.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:09,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.20 | bwd_microstep: 590.18 | bwd_inner_microstep: 590.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:55:09,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.74 | bwd_microstep: 463.20 | bwd_inner_microstep: 463.06 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:55:10,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.89 | bwd_microstep: 403.18 | bwd_inner_microstep: 403.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:11,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.56 | bwd_microstep: 455.00 | bwd_inner_microstep: 454.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:12,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.56 | bwd_microstep: 626.70 | bwd_inner_microstep: 626.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:13,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.81 | bwd_microstep: 403.50 | bwd_inner_microstep: 403.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:15,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.09 | optimizer_gradients: 14.22 | optimizer_step: 23.16
[2024-12-05 13:55:15,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 327.15 | bwd_microstep: 1452.57 | bwd_inner_microstep: 1440.51 | bwd_allreduce_microstep: 11.97 | step_microstep: 112.84
[2024-12-05 13:55:15,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4526.08 | bwd: 8485.34 | bwd_inner: 8471.54 | bwd_allreduce: 12.66 | step: 113.97

 13%|█▎        | 49/384 [12:11<1:22:41, 14.81s/it]
                                                  
{'loss': 0.2461, 'grad_norm': 1.9663130044937134, 'learning_rate': 9.757885489518296e-07, 'epoch': 0.25}

 13%|█▎        | 49/384 [12:11<1:22:41, 14.81s/it][2024-12-05 13:55:16,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 346.22 | bwd_microstep: 662.66 | bwd_inner_microstep: 662.53 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:55:17,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.75 | bwd_microstep: 419.29 | bwd_inner_microstep: 419.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:17,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.99 | bwd_microstep: 432.55 | bwd_inner_microstep: 432.42 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:55:18,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.61 | bwd_microstep: 466.03 | bwd_inner_microstep: 465.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:55:19,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.86 | bwd_microstep: 441.48 | bwd_inner_microstep: 441.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:20,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.53 | bwd_microstep: 415.64 | bwd_inner_microstep: 415.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:20,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.82 | bwd_microstep: 465.83 | bwd_inner_microstep: 465.69 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:55:21,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.63 | bwd_microstep: 479.90 | bwd_inner_microstep: 479.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:23,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 372.69 | bwd_microstep: 787.73 | bwd_inner_microstep: 787.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:23,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.68 | bwd_microstep: 498.52 | bwd_inner_microstep: 498.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:24,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.28 | bwd_microstep: 361.79 | bwd_inner_microstep: 361.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:25,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.16 | bwd_microstep: 434.27 | bwd_inner_microstep: 434.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:26,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.05 | bwd_microstep: 492.38 | bwd_inner_microstep: 492.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:27,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.96 | bwd_microstep: 462.49 | bwd_inner_microstep: 462.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:55:27,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.25 | bwd_microstep: 448.88 | bwd_inner_microstep: 448.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:55:30,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.88 | optimizer_gradients: 14.12 | optimizer_step: 23.19
[2024-12-05 13:55:30,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.60 | bwd_microstep: 1767.43 | bwd_inner_microstep: 1755.23 | bwd_allreduce_microstep: 12.12 | step_microstep: 112.80
[2024-12-05 13:55:30,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4508.14 | bwd: 9036.86 | bwd_inner: 9023.00 | bwd_allreduce: 12.77 | step: 113.91

 13%|█▎        | 50/384 [12:26<1:22:41, 14.85s/it]
                                                  
{'loss': 0.2065, 'grad_norm': 1.6957286596298218, 'learning_rate': 9.744735364682344e-07, 'epoch': 0.26}

 13%|█▎        | 50/384 [12:26<1:22:41, 14.85s/it][2024-12-05 13:55:30,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.77 | bwd_microstep: 457.58 | bwd_inner_microstep: 457.45 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:55:31,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.85 | bwd_microstep: 532.36 | bwd_inner_microstep: 532.22 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 13:55:32,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.44 | bwd_microstep: 558.42 | bwd_inner_microstep: 558.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:55:33,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.62 | bwd_microstep: 680.61 | bwd_inner_microstep: 680.47 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:55:34,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.31 | bwd_microstep: 479.85 | bwd_inner_microstep: 479.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:55:35,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.66 | bwd_microstep: 526.39 | bwd_inner_microstep: 526.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:55:36,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.64 | bwd_microstep: 481.22 | bwd_inner_microstep: 481.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:55:37,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.05 | bwd_microstep: 489.07 | bwd_inner_microstep: 488.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 13:55:38,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.14 | bwd_microstep: 430.56 | bwd_inner_microstep: 430.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:55:39,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.81 | bwd_microstep: 708.67 | bwd_inner_microstep: 708.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:55:39,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.66 | bwd_microstep: 415.55 | bwd_inner_microstep: 415.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:55:41,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 349.24 | bwd_microstep: 719.21 | bwd_inner_microstep: 719.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:55:42,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.63 | bwd_microstep: 618.25 | bwd_inner_microstep: 618.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:55:42,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.19 | bwd_microstep: 453.79 | bwd_inner_microstep: 453.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:55:43,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.35 | bwd_microstep: 554.41 | bwd_inner_microstep: 554.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:55:45,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.44 | optimizer_gradients: 14.13 | optimizer_step: 23.14
[2024-12-05 13:55:45,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.84 | bwd_microstep: 999.19 | bwd_inner_microstep: 987.06 | bwd_allreduce_microstep: 12.03 | step_microstep: 113.92
[2024-12-05 13:55:45,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4720.28 | bwd: 9105.16 | bwd_inner: 9091.38 | bwd_allreduce: 12.68 | step: 115.24

 13%|█▎        | 51/384 [12:41<1:23:00, 14.96s/it]
                                                  
{'loss': 0.1949, 'grad_norm': 1.7848544120788574, 'learning_rate': 9.731246845359184e-07, 'epoch': 0.27}

 13%|█▎        | 51/384 [12:41<1:23:00, 14.96s/it][2024-12-05 13:55:46,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.43 | bwd_microstep: 493.63 | bwd_inner_microstep: 493.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:55:47,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.42 | bwd_microstep: 455.29 | bwd_inner_microstep: 455.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:48,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.10 | bwd_microstep: 613.18 | bwd_inner_microstep: 613.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:48,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.72 | bwd_microstep: 462.67 | bwd_inner_microstep: 462.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:49,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.11 | bwd_microstep: 487.49 | bwd_inner_microstep: 487.35 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:55:50,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.15 | bwd_microstep: 459.65 | bwd_inner_microstep: 459.51 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:55:51,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.94 | bwd_microstep: 403.61 | bwd_inner_microstep: 403.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:52,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 388.25 | bwd_microstep: 836.97 | bwd_inner_microstep: 836.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:53,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.28 | bwd_microstep: 489.23 | bwd_inner_microstep: 489.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:54,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.96 | bwd_microstep: 565.14 | bwd_inner_microstep: 565.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:55,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 351.39 | bwd_microstep: 725.73 | bwd_inner_microstep: 725.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:56,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.92 | bwd_microstep: 616.46 | bwd_inner_microstep: 616.32 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:55:57,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.95 | bwd_microstep: 505.00 | bwd_inner_microstep: 504.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:55:58,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.67 | bwd_microstep: 450.60 | bwd_inner_microstep: 450.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:55:59,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 354.89 | bwd_microstep: 702.42 | bwd_inner_microstep: 702.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:00,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.26 | optimizer_gradients: 14.10 | optimizer_step: 23.11
[2024-12-05 13:56:00,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.15 | bwd_microstep: 881.74 | bwd_inner_microstep: 869.50 | bwd_allreduce_microstep: 12.14 | step_microstep: 112.00
[2024-12-05 13:56:00,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4789.41 | bwd: 9148.79 | bwd_inner: 9134.94 | bwd_allreduce: 12.80 | step: 113.08

 14%|█▎        | 52/384 [12:56<1:23:27, 15.08s/it]
                                                  
{'loss': 0.2135, 'grad_norm': 1.8262349367141724, 'learning_rate': 9.7174208935499e-07, 'epoch': 0.27}

 14%|█▎        | 52/384 [12:56<1:23:27, 15.08s/it][2024-12-05 13:56:01,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.22 | bwd_microstep: 424.23 | bwd_inner_microstep: 424.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:56:02,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.23 | bwd_microstep: 431.85 | bwd_inner_microstep: 431.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:56:02,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.89 | bwd_microstep: 356.28 | bwd_inner_microstep: 356.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:56:03,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.70 | bwd_microstep: 403.35 | bwd_inner_microstep: 403.20 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:56:04,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.76 | bwd_microstep: 439.06 | bwd_inner_microstep: 438.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:05,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.48 | bwd_microstep: 592.57 | bwd_inner_microstep: 592.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:06,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.83 | bwd_microstep: 433.53 | bwd_inner_microstep: 433.41 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:56:06,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.50 | bwd_microstep: 456.41 | bwd_inner_microstep: 456.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:08,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.81 | bwd_microstep: 691.75 | bwd_inner_microstep: 691.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:08,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.95 | bwd_microstep: 432.51 | bwd_inner_microstep: 432.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:09,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.61 | bwd_microstep: 613.03 | bwd_inner_microstep: 612.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:10,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.64 | bwd_microstep: 439.83 | bwd_inner_microstep: 439.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:56:11,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.16 | bwd_microstep: 365.94 | bwd_inner_microstep: 365.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:12,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.28 | bwd_microstep: 431.34 | bwd_inner_microstep: 431.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:12,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.54 | bwd_microstep: 431.52 | bwd_inner_microstep: 431.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:56:15,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.50 | optimizer_gradients: 14.07 | optimizer_step: 23.06
[2024-12-05 13:56:15,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.31 | bwd_microstep: 2263.11 | bwd_inner_microstep: 2250.88 | bwd_allreduce_microstep: 12.14 | step_microstep: 112.22
[2024-12-05 13:56:15,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4390.96 | bwd: 9206.32 | bwd_inner: 9192.44 | bwd_allreduce: 12.79 | step: 113.53

 14%|█▍        | 53/384 [13:11<1:22:50, 15.02s/it]
                                                  
{'loss': 0.1643, 'grad_norm': 1.6100451946258545, 'learning_rate': 9.703258495321265e-07, 'epoch': 0.28}

 14%|█▍        | 53/384 [13:11<1:22:50, 15.02s/it][2024-12-05 13:56:16,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.25 | bwd_microstep: 404.30 | bwd_inner_microstep: 404.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:17,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 356.55 | bwd_microstep: 734.44 | bwd_inner_microstep: 734.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:56:18,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.05 | bwd_microstep: 498.65 | bwd_inner_microstep: 498.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:56:19,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.18 | bwd_microstep: 357.09 | bwd_inner_microstep: 356.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:19,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.57 | bwd_microstep: 450.74 | bwd_inner_microstep: 450.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:56:20,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.15 | bwd_microstep: 398.16 | bwd_inner_microstep: 398.03 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:56:21,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.24 | bwd_microstep: 356.97 | bwd_inner_microstep: 356.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:22,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.54 | bwd_microstep: 544.86 | bwd_inner_microstep: 544.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:56:23,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.98 | bwd_microstep: 587.94 | bwd_inner_microstep: 587.77 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 13:56:24,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.47 | bwd_microstep: 481.11 | bwd_inner_microstep: 480.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:25,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.39 | bwd_microstep: 562.78 | bwd_inner_microstep: 562.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:56:25,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.44 | bwd_microstep: 435.24 | bwd_inner_microstep: 435.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:56:26,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.65 | bwd_microstep: 460.27 | bwd_inner_microstep: 460.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:56:27,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.14 | bwd_microstep: 439.54 | bwd_inner_microstep: 439.39 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 13:56:28,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.80 | bwd_microstep: 589.36 | bwd_inner_microstep: 589.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:56:30,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.26 | optimizer_gradients: 14.13 | optimizer_step: 23.29
[2024-12-05 13:56:30,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.48 | bwd_microstep: 1664.49 | bwd_inner_microstep: 1652.19 | bwd_allreduce_microstep: 12.21 | step_microstep: 112.31
[2024-12-05 13:56:30,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4492.00 | bwd: 8965.91 | bwd_inner: 8951.75 | bwd_allreduce: 12.89 | step: 113.55

 14%|█▍        | 54/384 [13:26<1:22:17, 14.96s/it]
                                                  
{'loss': 0.1796, 'grad_norm': 1.6476277112960815, 'learning_rate': 9.688760660735402e-07, 'epoch': 0.28}

 14%|█▍        | 54/384 [13:26<1:22:17, 14.96s/it][2024-12-05 13:56:31,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.98 | bwd_microstep: 387.37 | bwd_inner_microstep: 387.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:31,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.57 | bwd_microstep: 396.31 | bwd_inner_microstep: 396.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:32,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.06 | bwd_microstep: 338.92 | bwd_inner_microstep: 338.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:33,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.50 | bwd_microstep: 383.73 | bwd_inner_microstep: 383.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:34,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.67 | bwd_microstep: 437.97 | bwd_inner_microstep: 437.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:34,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.27 | bwd_microstep: 482.81 | bwd_inner_microstep: 482.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:35,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.33 | bwd_microstep: 434.59 | bwd_inner_microstep: 434.45 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:56:36,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.72 | bwd_microstep: 438.95 | bwd_inner_microstep: 438.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:37,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.42 | bwd_microstep: 522.35 | bwd_inner_microstep: 522.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:56:38,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.65 | bwd_microstep: 485.90 | bwd_inner_microstep: 485.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:56:39,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.15 | bwd_microstep: 599.88 | bwd_inner_microstep: 599.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:56:39,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.04 | bwd_microstep: 396.46 | bwd_inner_microstep: 396.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:40,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.89 | bwd_microstep: 418.75 | bwd_inner_microstep: 418.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:41,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 371.60 | bwd_microstep: 772.67 | bwd_inner_microstep: 772.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:56:42,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.82 | bwd_microstep: 624.21 | bwd_inner_microstep: 624.07 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 13:56:45,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.55 | optimizer_gradients: 14.05 | optimizer_step: 23.96
[2024-12-05 13:56:45,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.71 | bwd_microstep: 2061.80 | bwd_inner_microstep: 2049.69 | bwd_allreduce_microstep: 12.01 | step_microstep: 112.00
[2024-12-05 13:56:45,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4452.48 | bwd: 9182.64 | bwd_inner: 9168.86 | bwd_allreduce: 12.66 | step: 113.13

 14%|█▍        | 55/384 [13:41<1:22:14, 15.00s/it]
                                                  
{'loss': 0.2048, 'grad_norm': 1.6926974058151245, 'learning_rate': 9.673928423777756e-07, 'epoch': 0.29}

 14%|█▍        | 55/384 [13:41<1:22:14, 15.00s/it][2024-12-05 13:56:46,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.22 | bwd_microstep: 382.82 | bwd_inner_microstep: 382.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:56:47,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.62 | bwd_microstep: 464.76 | bwd_inner_microstep: 464.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:47,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.11 | bwd_microstep: 453.55 | bwd_inner_microstep: 453.11 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.16
[2024-12-05 13:56:48,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.99 | bwd_microstep: 491.74 | bwd_inner_microstep: 491.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:56:49,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.76 | bwd_microstep: 408.28 | bwd_inner_microstep: 408.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:50,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.62 | bwd_microstep: 387.46 | bwd_inner_microstep: 387.34 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 13:56:51,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.31 | bwd_microstep: 529.82 | bwd_inner_microstep: 529.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:56:51,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.69 | bwd_microstep: 418.48 | bwd_inner_microstep: 418.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:56:52,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.18 | bwd_microstep: 437.74 | bwd_inner_microstep: 437.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 13:56:53,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.89 | bwd_microstep: 439.20 | bwd_inner_microstep: 439.07 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:56:54,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.58 | bwd_microstep: 361.90 | bwd_inner_microstep: 361.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:54,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.65 | bwd_microstep: 443.33 | bwd_inner_microstep: 443.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:55,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.31 | bwd_microstep: 398.57 | bwd_inner_microstep: 398.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:56:56,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.93 | bwd_microstep: 442.04 | bwd_inner_microstep: 441.88 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:56:57,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.52 | bwd_microstep: 454.94 | bwd_inner_microstep: 454.80 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 13:57:00,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.53 | optimizer_gradients: 14.12 | optimizer_step: 23.14
[2024-12-05 13:57:00,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 243.32 | bwd_microstep: 2504.27 | bwd_inner_microstep: 2492.13 | bwd_allreduce_microstep: 12.05 | step_microstep: 112.45
[2024-12-05 13:57:00,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4217.78 | bwd: 9018.90 | bwd_inner: 9004.74 | bwd_allreduce: 12.87 | step: 113.67

 15%|█▍        | 56/384 [13:56<1:21:16, 14.87s/it]
                                                  
{'loss': 0.1953, 'grad_norm': 1.7797563076019287, 'learning_rate': 9.658762842283341e-07, 'epoch': 0.29}

 15%|█▍        | 56/384 [13:56<1:21:16, 14.87s/it][2024-12-05 13:57:00,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 238.68 | bwd_microstep: 330.10 | bwd_inner_microstep: 329.91 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 13:57:01,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.58 | bwd_microstep: 486.82 | bwd_inner_microstep: 486.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:02,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.42 | bwd_microstep: 456.74 | bwd_inner_microstep: 456.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:03,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 341.33 | bwd_microstep: 685.98 | bwd_inner_microstep: 685.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:04,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.52 | bwd_microstep: 567.37 | bwd_inner_microstep: 567.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:05,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.94 | bwd_microstep: 429.66 | bwd_inner_microstep: 429.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:57:06,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.30 | bwd_microstep: 568.92 | bwd_inner_microstep: 568.78 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:57:07,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.76 | bwd_microstep: 598.49 | bwd_inner_microstep: 598.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:08,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.69 | bwd_microstep: 455.75 | bwd_inner_microstep: 455.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:08,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.68 | bwd_microstep: 405.21 | bwd_inner_microstep: 405.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:57:09,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.27 | bwd_microstep: 536.97 | bwd_inner_microstep: 536.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:10,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.78 | bwd_microstep: 458.94 | bwd_inner_microstep: 458.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:11,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.45 | bwd_microstep: 407.47 | bwd_inner_microstep: 407.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:57:11,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.77 | bwd_microstep: 358.13 | bwd_inner_microstep: 358.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:57:12,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.66 | bwd_microstep: 360.78 | bwd_inner_microstep: 360.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:15,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.61 | optimizer_gradients: 14.14 | optimizer_step: 23.19
[2024-12-05 13:57:15,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.82 | bwd_microstep: 2008.37 | bwd_inner_microstep: 1996.25 | bwd_allreduce_microstep: 12.03 | step_microstep: 112.85
[2024-12-05 13:57:15,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4427.73 | bwd: 9115.71 | bwd_inner: 9101.87 | bwd_allreduce: 12.68 | step: 113.94

 15%|█▍        | 57/384 [14:11<1:21:14, 14.91s/it]
                                                  
{'loss': 0.2103, 'grad_norm': 1.7844983339309692, 'learning_rate': 9.643264997861312e-07, 'epoch': 0.3}

 15%|█▍        | 57/384 [14:11<1:21:14, 14.91s/it][2024-12-05 13:57:15,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.90 | bwd_microstep: 383.85 | bwd_inner_microstep: 383.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:16,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.21 | bwd_microstep: 406.73 | bwd_inner_microstep: 406.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:17,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.92 | bwd_microstep: 599.20 | bwd_inner_microstep: 599.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:57:18,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.20 | bwd_microstep: 442.31 | bwd_inner_microstep: 442.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:19,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.36 | bwd_microstep: 422.92 | bwd_inner_microstep: 422.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:19,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.50 | bwd_microstep: 493.09 | bwd_inner_microstep: 492.95 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:57:20,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.12 | bwd_microstep: 436.46 | bwd_inner_microstep: 436.32 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 13:57:21,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.76 | bwd_microstep: 362.11 | bwd_inner_microstep: 361.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:22,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.33 | bwd_microstep: 417.78 | bwd_inner_microstep: 417.63 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 13:57:23,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.27 | bwd_microstep: 533.09 | bwd_inner_microstep: 532.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:57:23,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.76 | bwd_microstep: 407.92 | bwd_inner_microstep: 407.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:24,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.81 | bwd_microstep: 444.59 | bwd_inner_microstep: 444.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:25,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.22 | bwd_microstep: 462.30 | bwd_inner_microstep: 462.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:57:26,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.37 | bwd_microstep: 453.08 | bwd_inner_microstep: 452.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:57:26,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.71 | bwd_microstep: 341.42 | bwd_inner_microstep: 341.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:57:29,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.96 | optimizer_gradients: 14.03 | optimizer_step: 23.40
[2024-12-05 13:57:29,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.61 | bwd_microstep: 1761.06 | bwd_inner_microstep: 1748.84 | bwd_allreduce_microstep: 12.14 | step_microstep: 111.89
[2024-12-05 13:57:29,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4252.12 | bwd: 8367.90 | bwd_inner: 8353.99 | bwd_allreduce: 12.82 | step: 113.05

 15%|█▌        | 58/384 [14:25<1:19:32, 14.64s/it]
                                                  
{'loss': 0.1854, 'grad_norm': 1.8552502393722534, 'learning_rate': 9.627435995817797e-07, 'epoch': 0.3}

 15%|█▌        | 58/384 [14:25<1:19:32, 14.64s/it][2024-12-05 13:57:29,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.64 | bwd_microstep: 424.97 | bwd_inner_microstep: 424.82 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:57:30,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.75 | bwd_microstep: 356.42 | bwd_inner_microstep: 356.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:57:31,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.64 | bwd_microstep: 440.80 | bwd_inner_microstep: 440.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:57:32,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.01 | bwd_microstep: 628.33 | bwd_inner_microstep: 628.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:57:33,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.58 | bwd_microstep: 448.17 | bwd_inner_microstep: 448.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:57:34,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.12 | bwd_microstep: 435.62 | bwd_inner_microstep: 435.49 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:57:34,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.20 | bwd_microstep: 406.59 | bwd_inner_microstep: 406.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:57:35,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.19 | bwd_microstep: 557.93 | bwd_inner_microstep: 557.80 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 13:57:36,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.06 | bwd_microstep: 458.95 | bwd_inner_microstep: 458.82 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:57:37,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.99 | bwd_microstep: 482.92 | bwd_inner_microstep: 482.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:57:38,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.03 | bwd_microstep: 431.03 | bwd_inner_microstep: 430.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:57:39,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.92 | bwd_microstep: 497.13 | bwd_inner_microstep: 497.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:57:39,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.96 | bwd_microstep: 565.62 | bwd_inner_microstep: 565.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:57:40,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.35 | bwd_microstep: 420.61 | bwd_inner_microstep: 420.49 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:57:41,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.59 | bwd_microstep: 416.46 | bwd_inner_microstep: 416.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:57:44,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.93 | optimizer_gradients: 14.10 | optimizer_step: 23.11
[2024-12-05 13:57:44,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.99 | bwd_microstep: 2273.26 | bwd_inner_microstep: 2261.07 | bwd_allreduce_microstep: 12.11 | step_microstep: 111.39
[2024-12-05 13:57:44,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4337.09 | bwd: 9244.81 | bwd_inner: 9230.93 | bwd_allreduce: 12.79 | step: 112.71

 15%|█▌        | 59/384 [14:40<1:20:00, 14.77s/it]
                                                  
{'loss': 0.1892, 'grad_norm': 1.8308689594268799, 'learning_rate': 9.611276965077097e-07, 'epoch': 0.31}

 15%|█▌        | 59/384 [14:40<1:20:00, 14.77s/it][2024-12-05 13:57:45,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.28 | bwd_microstep: 563.72 | bwd_inner_microstep: 563.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:57:46,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.72 | bwd_microstep: 720.05 | bwd_inner_microstep: 719.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:57:47,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.99 | bwd_microstep: 507.05 | bwd_inner_microstep: 506.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:57:47,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.83 | bwd_microstep: 450.94 | bwd_inner_microstep: 450.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:57:49,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 368.18 | bwd_microstep: 773.32 | bwd_inner_microstep: 773.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:57:49,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.10 | bwd_microstep: 454.57 | bwd_inner_microstep: 454.45 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:57:50,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.53 | bwd_microstep: 460.41 | bwd_inner_microstep: 460.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:57:51,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.48 | bwd_microstep: 447.11 | bwd_inner_microstep: 446.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:52,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.52 | bwd_microstep: 447.30 | bwd_inner_microstep: 447.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:53,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.11 | bwd_microstep: 565.79 | bwd_inner_microstep: 565.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:54,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.89 | bwd_microstep: 403.06 | bwd_inner_microstep: 402.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:57:54,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 239.76 | bwd_microstep: 365.84 | bwd_inner_microstep: 365.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:55,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 351.90 | bwd_microstep: 717.66 | bwd_inner_microstep: 717.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:56,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.76 | bwd_microstep: 587.57 | bwd_inner_microstep: 587.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:57,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.83 | bwd_microstep: 444.60 | bwd_inner_microstep: 444.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:57:59,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.35 | optimizer_gradients: 14.08 | optimizer_step: 23.22
[2024-12-05 13:57:59,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.76 | bwd_microstep: 899.44 | bwd_inner_microstep: 887.37 | bwd_allreduce_microstep: 11.97 | step_microstep: 112.16
[2024-12-05 13:57:59,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4663.70 | bwd: 8808.41 | bwd_inner: 8794.69 | bwd_allreduce: 12.62 | step: 113.34

 16%|█▌        | 60/384 [14:55<1:20:03, 14.82s/it]
                                                  
{'loss': 0.216, 'grad_norm': 1.8636093139648438, 'learning_rate': 9.594789058101153e-07, 'epoch': 0.31}

 16%|█▌        | 60/384 [14:55<1:20:03, 14.82s/it][2024-12-05 13:57:59,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.61 | bwd_microstep: 389.22 | bwd_inner_microstep: 389.05 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 13:58:01,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 377.07 | bwd_microstep: 758.41 | bwd_inner_microstep: 758.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:58:01,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.14 | bwd_microstep: 353.70 | bwd_inner_microstep: 353.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:58:02,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.00 | bwd_microstep: 407.33 | bwd_inner_microstep: 407.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:58:03,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.43 | bwd_microstep: 386.29 | bwd_inner_microstep: 386.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:58:04,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.73 | bwd_microstep: 437.52 | bwd_inner_microstep: 437.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:04,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.29 | bwd_microstep: 405.62 | bwd_inner_microstep: 405.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:05,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.32 | bwd_microstep: 437.53 | bwd_inner_microstep: 437.39 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:58:06,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.11 | bwd_microstep: 496.84 | bwd_inner_microstep: 496.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:07,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.06 | bwd_microstep: 706.35 | bwd_inner_microstep: 706.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:08,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.70 | bwd_microstep: 481.06 | bwd_inner_microstep: 480.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:09,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.74 | bwd_microstep: 414.53 | bwd_inner_microstep: 414.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 13:58:09,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.83 | bwd_microstep: 461.15 | bwd_inner_microstep: 461.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:58:10,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.85 | bwd_microstep: 459.55 | bwd_inner_microstep: 459.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:11,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.08 | bwd_microstep: 404.73 | bwd_inner_microstep: 404.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 13:58:13,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.25 | optimizer_gradients: 14.13 | optimizer_step: 23.33
[2024-12-05 13:58:13,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.26 | bwd_microstep: 1648.60 | bwd_inner_microstep: 1636.36 | bwd_allreduce_microstep: 12.16 | step_microstep: 112.36
[2024-12-05 13:58:13,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4409.28 | bwd: 8648.42 | bwd_inner: 8634.50 | bwd_allreduce: 12.82 | step: 113.59

 16%|█▌        | 61/384 [15:09<1:19:15, 14.72s/it]
                                                  
{'loss': 0.1924, 'grad_norm': 1.5857099294662476, 'learning_rate': 9.577973450807351e-07, 'epoch': 0.32}

 16%|█▌        | 61/384 [15:09<1:19:15, 14.72s/it][2024-12-05 13:58:14,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.01 | bwd_microstep: 331.43 | bwd_inner_microstep: 331.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:58:15,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.37 | bwd_microstep: 495.52 | bwd_inner_microstep: 495.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:58:15,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.51 | bwd_microstep: 457.51 | bwd_inner_microstep: 457.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:58:16,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.15 | bwd_microstep: 634.60 | bwd_inner_microstep: 634.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:58:18,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.58 | bwd_microstep: 693.29 | bwd_inner_microstep: 693.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:58:19,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.76 | bwd_microstep: 580.29 | bwd_inner_microstep: 580.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:19,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.33 | bwd_microstep: 463.94 | bwd_inner_microstep: 463.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:20,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.01 | bwd_microstep: 364.91 | bwd_inner_microstep: 364.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:21,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.98 | bwd_microstep: 454.99 | bwd_inner_microstep: 454.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:22,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.75 | bwd_microstep: 450.68 | bwd_inner_microstep: 450.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:22,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.55 | bwd_microstep: 413.30 | bwd_inner_microstep: 413.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:23,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 228.91 | bwd_microstep: 332.98 | bwd_inner_microstep: 332.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:58:24,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.81 | bwd_microstep: 603.53 | bwd_inner_microstep: 603.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:25,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.78 | bwd_microstep: 460.29 | bwd_inner_microstep: 460.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:26,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 341.09 | bwd_microstep: 690.06 | bwd_inner_microstep: 689.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:28,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.95 | optimizer_gradients: 14.24 | optimizer_step: 23.17
[2024-12-05 13:58:28,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.81 | bwd_microstep: 1826.57 | bwd_inner_microstep: 1814.40 | bwd_allreduce_microstep: 12.07 | step_microstep: 114.83
[2024-12-05 13:58:28,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4486.46 | bwd: 9253.89 | bwd_inner: 9240.13 | bwd_allreduce: 12.68 | step: 115.97

 16%|█▌        | 62/384 [15:25<1:19:51, 14.88s/it]
                                                  
{'loss': 0.2088, 'grad_norm': 1.670000433921814, 'learning_rate': 9.560831342484666e-07, 'epoch': 0.32}

 16%|█▌        | 62/384 [15:25<1:19:51, 14.88s/it][2024-12-05 13:58:29,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.06 | bwd_microstep: 372.81 | bwd_inner_microstep: 372.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:58:30,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 341.28 | bwd_microstep: 603.30 | bwd_inner_microstep: 603.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:31,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.47 | bwd_microstep: 420.94 | bwd_inner_microstep: 420.54 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.21
[2024-12-05 13:58:32,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.13 | bwd_microstep: 592.12 | bwd_inner_microstep: 591.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:33,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.91 | bwd_microstep: 438.53 | bwd_inner_microstep: 438.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:58:33,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.70 | bwd_microstep: 426.72 | bwd_inner_microstep: 426.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:58:34,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.35 | bwd_microstep: 610.92 | bwd_inner_microstep: 610.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:58:35,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.88 | bwd_microstep: 628.30 | bwd_inner_microstep: 628.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:36,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.36 | bwd_microstep: 443.10 | bwd_inner_microstep: 442.91 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 13:58:37,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.00 | bwd_microstep: 409.45 | bwd_inner_microstep: 409.30 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:58:38,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.68 | bwd_microstep: 436.01 | bwd_inner_microstep: 435.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:39,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.21 | bwd_microstep: 418.74 | bwd_inner_microstep: 418.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:39,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.92 | bwd_microstep: 442.40 | bwd_inner_microstep: 442.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:40,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 326.50 | bwd_microstep: 602.72 | bwd_inner_microstep: 602.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:58:41,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.60 | bwd_microstep: 451.70 | bwd_inner_microstep: 451.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:58:43,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.49 | optimizer_gradients: 14.17 | optimizer_step: 23.16
[2024-12-05 13:58:43,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.88 | bwd_microstep: 1766.32 | bwd_inner_microstep: 1754.12 | bwd_allreduce_microstep: 12.10 | step_microstep: 112.79
[2024-12-05 13:58:43,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4530.98 | bwd: 9064.11 | bwd_inner: 9049.93 | bwd_allreduce: 12.82 | step: 114.11

 16%|█▋        | 63/384 [15:40<1:19:56, 14.94s/it]
                                                  
{'loss': 0.1697, 'grad_norm': 1.8758388757705688, 'learning_rate': 9.543363955708124e-07, 'epoch': 0.33}

 16%|█▋        | 63/384 [15:40<1:19:56, 14.94s/it][2024-12-05 13:58:44,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.53 | bwd_microstep: 390.28 | bwd_inner_microstep: 390.14 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:58:45,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.77 | bwd_microstep: 485.45 | bwd_inner_microstep: 485.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:46,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.65 | bwd_microstep: 540.97 | bwd_inner_microstep: 540.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:47,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.53 | bwd_microstep: 439.26 | bwd_inner_microstep: 439.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:58:48,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.56 | bwd_microstep: 526.65 | bwd_inner_microstep: 526.51 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07
[2024-12-05 13:58:48,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.99 | bwd_microstep: 432.69 | bwd_inner_microstep: 432.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:58:49,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.84 | bwd_microstep: 354.21 | bwd_inner_microstep: 354.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:58:51,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 507.56 | bwd_microstep: 1268.21 | bwd_inner_microstep: 1268.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:58:52,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.89 | bwd_microstep: 421.53 | bwd_inner_microstep: 421.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:58:53,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.53 | bwd_microstep: 412.36 | bwd_inner_microstep: 412.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:58:53,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.37 | bwd_microstep: 457.02 | bwd_inner_microstep: 456.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:58:54,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.68 | bwd_microstep: 360.87 | bwd_inner_microstep: 360.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:58:55,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.69 | bwd_microstep: 454.17 | bwd_inner_microstep: 454.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:58:56,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.67 | bwd_microstep: 453.79 | bwd_inner_microstep: 453.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:58:56,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.83 | bwd_microstep: 431.39 | bwd_inner_microstep: 431.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:58:59,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.41 | optimizer_gradients: 14.13 | optimizer_step: 23.31
[2024-12-05 13:58:59,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.29 | bwd_microstep: 1654.75 | bwd_inner_microstep: 1642.41 | bwd_allreduce_microstep: 12.24 | step_microstep: 112.38
[2024-12-05 13:58:59,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4541.43 | bwd: 9083.58 | bwd_inner: 9069.57 | bwd_allreduce: 12.87 | step: 113.63

 17%|█▋        | 64/384 [15:55<1:20:01, 15.00s/it]
                                                  
{'loss': 0.2249, 'grad_norm': 2.020310401916504, 'learning_rate': 9.525572536251605e-07, 'epoch': 0.33}

 17%|█▋        | 64/384 [15:55<1:20:01, 15.00s/it][2024-12-05 13:58:59,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.02 | bwd_microstep: 387.10 | bwd_inner_microstep: 386.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:59:00,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.12 | bwd_microstep: 588.92 | bwd_inner_microstep: 588.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:59:01,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.49 | bwd_microstep: 439.58 | bwd_inner_microstep: 439.39 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 13:59:02,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.43 | bwd_microstep: 599.55 | bwd_inner_microstep: 599.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:59:03,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.60 | bwd_microstep: 494.70 | bwd_inner_microstep: 494.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:59:04,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.26 | bwd_microstep: 623.59 | bwd_inner_microstep: 623.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:59:05,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 383.06 | bwd_microstep: 844.25 | bwd_inner_microstep: 844.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:59:06,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.21 | bwd_microstep: 432.11 | bwd_inner_microstep: 431.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:59:07,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.53 | bwd_microstep: 585.36 | bwd_inner_microstep: 585.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:59:08,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.02 | bwd_microstep: 453.13 | bwd_inner_microstep: 453.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:59:09,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.91 | bwd_microstep: 379.74 | bwd_inner_microstep: 379.61 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:59:09,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.85 | bwd_microstep: 380.97 | bwd_inner_microstep: 380.85 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 13:59:10,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.90 | bwd_microstep: 609.22 | bwd_inner_microstep: 609.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:59:12,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.29 | bwd_microstep: 697.30 | bwd_inner_microstep: 697.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:59:12,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.38 | bwd_microstep: 432.21 | bwd_inner_microstep: 432.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:59:14,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.73 | optimizer_gradients: 14.04 | optimizer_step: 24.01
[2024-12-05 13:59:14,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.57 | bwd_microstep: 771.43 | bwd_inner_microstep: 759.33 | bwd_allreduce_microstep: 12.01 | step_microstep: 112.34
[2024-12-05 13:59:14,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4626.71 | bwd: 8719.17 | bwd_inner: 8705.38 | bwd_allreduce: 12.69 | step: 113.61

 17%|█▋        | 65/384 [16:10<1:19:39, 14.98s/it]
                                                  
{'loss': 0.1884, 'grad_norm': 1.8294882774353027, 'learning_rate': 9.507458352999001e-07, 'epoch': 0.34}

 17%|█▋        | 65/384 [16:10<1:19:39, 14.98s/it][2024-12-05 13:59:14,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.51 | bwd_microstep: 411.20 | bwd_inner_microstep: 411.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:15,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.57 | bwd_microstep: 494.32 | bwd_inner_microstep: 494.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:16,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.40 | bwd_microstep: 591.34 | bwd_inner_microstep: 591.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:17,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.99 | bwd_microstep: 437.55 | bwd_inner_microstep: 437.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:59:18,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.81 | bwd_microstep: 405.16 | bwd_inner_microstep: 405.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:59:19,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 379.34 | bwd_microstep: 776.80 | bwd_inner_microstep: 776.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:20,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.94 | bwd_microstep: 483.55 | bwd_inner_microstep: 483.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:21,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.20 | bwd_microstep: 486.26 | bwd_inner_microstep: 486.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:22,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.69 | bwd_microstep: 430.28 | bwd_inner_microstep: 430.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:59:22,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.49 | bwd_microstep: 449.12 | bwd_inner_microstep: 448.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:59:23,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.80 | bwd_microstep: 570.15 | bwd_inner_microstep: 570.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:59:24,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.48 | bwd_microstep: 540.06 | bwd_inner_microstep: 539.87 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 13:59:25,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.54 | bwd_microstep: 455.60 | bwd_inner_microstep: 455.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:59:26,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.95 | bwd_microstep: 426.90 | bwd_inner_microstep: 426.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:59:27,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.62 | bwd_microstep: 599.06 | bwd_inner_microstep: 598.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:59:29,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.25 | optimizer_gradients: 14.07 | optimizer_step: 23.33
[2024-12-05 13:59:29,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.96 | bwd_microstep: 1199.10 | bwd_inner_microstep: 1186.89 | bwd_allreduce_microstep: 12.12 | step_microstep: 113.72
[2024-12-05 13:59:29,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4636.35 | bwd: 8756.45 | bwd_inner: 8742.54 | bwd_allreduce: 12.77 | step: 114.95

 17%|█▋        | 66/384 [16:25<1:19:38, 15.03s/it]
                                                  
{'loss': 0.1761, 'grad_norm': 1.606002926826477, 'learning_rate': 9.489022697853708e-07, 'epoch': 0.34}

 17%|█▋        | 66/384 [16:25<1:19:38, 15.03s/it][2024-12-05 13:59:29,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.63 | bwd_microstep: 424.75 | bwd_inner_microstep: 424.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 13:59:31,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 349.20 | bwd_microstep: 716.46 | bwd_inner_microstep: 716.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:59:31,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.70 | bwd_microstep: 428.76 | bwd_inner_microstep: 428.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:32,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.55 | bwd_microstep: 462.22 | bwd_inner_microstep: 462.07 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:59:33,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.04 | bwd_microstep: 412.59 | bwd_inner_microstep: 412.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:34,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.56 | bwd_microstep: 418.89 | bwd_inner_microstep: 418.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:35,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.57 | bwd_microstep: 450.26 | bwd_inner_microstep: 450.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:35,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.57 | bwd_microstep: 463.26 | bwd_inner_microstep: 463.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:36,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.66 | bwd_microstep: 413.93 | bwd_inner_microstep: 413.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:37,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.23 | bwd_microstep: 484.69 | bwd_inner_microstep: 484.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 13:59:38,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.21 | bwd_microstep: 532.63 | bwd_inner_microstep: 532.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:39,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.13 | bwd_microstep: 687.92 | bwd_inner_microstep: 687.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:40,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.33 | bwd_microstep: 418.68 | bwd_inner_microstep: 418.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:41,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.92 | bwd_microstep: 623.65 | bwd_inner_microstep: 623.50 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:59:42,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 328.61 | bwd_microstep: 596.67 | bwd_inner_microstep: 596.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:44,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.36 | optimizer_gradients: 14.04 | optimizer_step: 23.25
[2024-12-05 13:59:44,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.24 | bwd_microstep: 1312.21 | bwd_inner_microstep: 1300.21 | bwd_allreduce_microstep: 11.90 | step_microstep: 112.13
[2024-12-05 13:59:44,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4545.21 | bwd: 8847.56 | bwd_inner: 8833.88 | bwd_allreduce: 12.55 | step: 113.22

 17%|█▋        | 67/384 [16:40<1:19:14, 15.00s/it]
                                                  
{'loss': 0.1871, 'grad_norm': 1.6073530912399292, 'learning_rate': 9.470266885646503e-07, 'epoch': 0.35}

 17%|█▋        | 67/384 [16:40<1:19:14, 15.00s/it][2024-12-05 13:59:44,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.74 | bwd_microstep: 516.51 | bwd_inner_microstep: 516.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:45,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.09 | bwd_microstep: 513.05 | bwd_inner_microstep: 512.91 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:59:46,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.40 | bwd_microstep: 410.12 | bwd_inner_microstep: 409.99 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:59:47,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.51 | bwd_microstep: 433.87 | bwd_inner_microstep: 433.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:48,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.77 | bwd_microstep: 441.06 | bwd_inner_microstep: 440.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:49,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.64 | bwd_microstep: 559.60 | bwd_inner_microstep: 559.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:50,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.46 | bwd_microstep: 444.51 | bwd_inner_microstep: 444.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:59:50,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.22 | bwd_microstep: 412.00 | bwd_inner_microstep: 411.86 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 13:59:51,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.78 | bwd_microstep: 467.59 | bwd_inner_microstep: 467.45 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:59:52,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.09 | bwd_microstep: 407.44 | bwd_inner_microstep: 407.25 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:59:53,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.51 | bwd_microstep: 494.19 | bwd_inner_microstep: 494.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 13:59:54,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 326.83 | bwd_microstep: 625.12 | bwd_inner_microstep: 624.95 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 13:59:55,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.53 | bwd_microstep: 409.82 | bwd_inner_microstep: 409.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.12
[2024-12-05 13:59:56,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.72 | bwd_microstep: 425.93 | bwd_inner_microstep: 425.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 13:59:56,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.82 | bwd_microstep: 456.54 | bwd_inner_microstep: 456.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 13:59:59,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.46 | optimizer_gradients: 14.04 | optimizer_step: 23.10
[2024-12-05 13:59:59,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.90 | bwd_microstep: 1823.21 | bwd_inner_microstep: 1810.95 | bwd_allreduce_microstep: 12.16 | step_microstep: 112.00
[2024-12-05 13:59:59,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4421.09 | bwd: 8840.53 | bwd_inner: 8826.49 | bwd_allreduce: 12.86 | step: 113.21

 18%|█▊        | 68/384 [16:55<1:19:04, 15.01s/it]
                                                  
{'loss': 0.173, 'grad_norm': 1.7087726593017578, 'learning_rate': 9.451192254041758e-07, 'epoch': 0.35}

 18%|█▊        | 68/384 [16:55<1:19:04, 15.01s/it][2024-12-05 13:59:59,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.31 | bwd_microstep: 332.04 | bwd_inner_microstep: 331.90 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:00:00,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.38 | bwd_microstep: 435.06 | bwd_inner_microstep: 434.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:00:01,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.94 | bwd_microstep: 451.74 | bwd_inner_microstep: 451.59 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 14:00:02,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.81 | bwd_microstep: 444.95 | bwd_inner_microstep: 444.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:00:03,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.04 | bwd_microstep: 438.15 | bwd_inner_microstep: 438.02 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:00:03,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.40 | bwd_microstep: 546.64 | bwd_inner_microstep: 546.48 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:00:05,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.06 | bwd_microstep: 627.67 | bwd_inner_microstep: 627.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:00:05,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.73 | bwd_microstep: 405.28 | bwd_inner_microstep: 405.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:00:06,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.66 | bwd_microstep: 456.87 | bwd_inner_microstep: 456.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:00:07,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.35 | bwd_microstep: 606.87 | bwd_inner_microstep: 606.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:00:09,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.53 | bwd_microstep: 688.81 | bwd_inner_microstep: 688.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:00:10,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.80 | bwd_microstep: 409.07 | bwd_inner_microstep: 408.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:00:10,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.10 | bwd_microstep: 447.25 | bwd_inner_microstep: 447.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:00:12,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.07 | bwd_microstep: 647.73 | bwd_inner_microstep: 647.53 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.12
[2024-12-05 14:00:13,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 351.63 | bwd_microstep: 725.55 | bwd_inner_microstep: 725.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:00:14,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.11 | optimizer_gradients: 14.10 | optimizer_step: 23.34
[2024-12-05 14:00:14,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.51 | bwd_microstep: 803.56 | bwd_inner_microstep: 791.25 | bwd_allreduce_microstep: 12.21 | step_microstep: 112.13
[2024-12-05 14:00:14,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4570.38 | bwd: 8467.24 | bwd_inner: 8453.13 | bwd_allreduce: 12.93 | step: 113.51

 18%|█▊        | 69/384 [17:10<1:19:18, 15.11s/it]
                                                  
{'loss': 0.1957, 'grad_norm': 1.7764538526535034, 'learning_rate': 9.431800163442041e-07, 'epoch': 0.36}

 18%|█▊        | 69/384 [17:10<1:19:18, 15.11s/it][2024-12-05 14:00:15,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 359.19 | bwd_microstep: 710.57 | bwd_inner_microstep: 710.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 14:00:16,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 354.88 | bwd_microstep: 716.14 | bwd_inner_microstep: 715.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:00:17,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.52 | bwd_microstep: 441.97 | bwd_inner_microstep: 441.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:00:18,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.70 | bwd_microstep: 425.18 | bwd_inner_microstep: 425.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:00:19,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.33 | bwd_microstep: 598.19 | bwd_inner_microstep: 598.06 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:00:20,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.69 | bwd_microstep: 474.64 | bwd_inner_microstep: 474.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:00:21,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.35 | bwd_microstep: 591.53 | bwd_inner_microstep: 591.34 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.11
[2024-12-05 14:00:22,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.21 | bwd_microstep: 487.63 | bwd_inner_microstep: 487.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:00:23,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.69 | bwd_microstep: 532.08 | bwd_inner_microstep: 531.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:00:24,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.35 | bwd_microstep: 686.53 | bwd_inner_microstep: 686.37 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:00:25,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.63 | bwd_microstep: 491.20 | bwd_inner_microstep: 491.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:00:27,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 391.23 | bwd_microstep: 870.26 | bwd_inner_microstep: 870.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:00:27,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.06 | bwd_microstep: 432.03 | bwd_inner_microstep: 431.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:00:28,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.44 | bwd_microstep: 481.73 | bwd_inner_microstep: 481.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:00:29,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.87 | bwd_microstep: 568.34 | bwd_inner_microstep: 568.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:00:31,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.14 | optimizer_gradients: 14.10 | optimizer_step: 23.18
[2024-12-05 14:00:31,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.16 | bwd_microstep: 1038.84 | bwd_inner_microstep: 1026.53 | bwd_allreduce_microstep: 12.20 | step_microstep: 112.00
[2024-12-05 14:00:31,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4836.36 | bwd: 9546.89 | bwd_inner: 9532.80 | bwd_allreduce: 12.91 | step: 113.37

 18%|█▊        | 70/384 [17:27<1:21:49, 15.63s/it]
                                                  
{'loss': 0.2154, 'grad_norm': 1.8759775161743164, 'learning_rate': 9.412091996891095e-07, 'epoch': 0.36}

 18%|█▊        | 70/384 [17:27<1:21:49, 15.63s/it][2024-12-05 14:00:32,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.07 | bwd_microstep: 457.78 | bwd_inner_microstep: 457.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:00:33,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.05 | bwd_microstep: 417.23 | bwd_inner_microstep: 417.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:00:33,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.63 | bwd_microstep: 452.02 | bwd_inner_microstep: 451.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:00:34,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.30 | bwd_microstep: 454.56 | bwd_inner_microstep: 454.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:00:35,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.29 | bwd_microstep: 433.75 | bwd_inner_microstep: 433.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:00:36,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.58 | bwd_microstep: 465.21 | bwd_inner_microstep: 465.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:00:37,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.40 | bwd_microstep: 436.71 | bwd_inner_microstep: 436.57 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 14:00:38,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.64 | bwd_microstep: 410.44 | bwd_inner_microstep: 410.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:00:39,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.33 | bwd_microstep: 435.72 | bwd_inner_microstep: 435.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:00:40,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.24 | bwd_microstep: 429.16 | bwd_inner_microstep: 429.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:00:41,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.84 | bwd_microstep: 613.60 | bwd_inner_microstep: 613.44 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:00:42,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.72 | bwd_microstep: 483.56 | bwd_inner_microstep: 483.44 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:00:42,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.88 | bwd_microstep: 374.45 | bwd_inner_microstep: 374.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:00:43,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.62 | bwd_microstep: 483.55 | bwd_inner_microstep: 483.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:00:44,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.67 | bwd_microstep: 464.61 | bwd_inner_microstep: 464.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:00:48,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.89 | optimizer_gradients: 14.04 | optimizer_step: 23.37
[2024-12-05 14:00:48,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.93 | bwd_microstep: 2948.57 | bwd_inner_microstep: 2936.45 | bwd_allreduce_microstep: 12.01 | step_microstep: 111.68
[2024-12-05 14:00:48,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4323.26 | bwd: 9760.90 | bwd_inner: 9747.15 | bwd_allreduce: 12.65 | step: 112.80

 18%|█▊        | 71/384 [17:44<1:23:47, 16.06s/it]
                                                  
{'loss': 0.1679, 'grad_norm': 1.8281443119049072, 'learning_rate': 9.392069159975198e-07, 'epoch': 0.37}

 18%|█▊        | 71/384 [17:44<1:23:47, 16.06s/it][2024-12-05 14:00:49,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.59 | bwd_microstep: 408.85 | bwd_inner_microstep: 408.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:00:50,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.30 | bwd_microstep: 478.02 | bwd_inner_microstep: 477.90 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:00:51,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.60 | bwd_microstep: 465.74 | bwd_inner_microstep: 465.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:00:52,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 348.38 | bwd_microstep: 691.32 | bwd_inner_microstep: 691.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:00:53,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.68 | bwd_microstep: 485.15 | bwd_inner_microstep: 484.99 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:00:54,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.11 | bwd_microstep: 338.52 | bwd_inner_microstep: 338.39 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:00:55,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 329.03 | bwd_microstep: 596.99 | bwd_inner_microstep: 596.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:00:56,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.25 | bwd_microstep: 452.24 | bwd_inner_microstep: 452.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:00:57,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.95 | bwd_microstep: 488.42 | bwd_inner_microstep: 488.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.12
[2024-12-05 14:00:58,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.56 | bwd_microstep: 416.98 | bwd_inner_microstep: 416.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:00:59,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.16 | bwd_microstep: 571.91 | bwd_inner_microstep: 571.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:01:00,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.11 | bwd_microstep: 415.19 | bwd_inner_microstep: 415.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:01:01,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 325.16 | bwd_microstep: 620.62 | bwd_inner_microstep: 620.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:01:02,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 240.85 | bwd_microstep: 360.53 | bwd_inner_microstep: 360.37 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09
[2024-12-05 14:01:02,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.45 | bwd_microstep: 430.24 | bwd_inner_microstep: 430.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:01:05,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.39 | optimizer_gradients: 14.06 | optimizer_step: 23.15
[2024-12-05 14:01:05,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.78 | bwd_microstep: 2351.19 | bwd_inner_microstep: 2338.96 | bwd_allreduce_microstep: 12.14 | step_microstep: 112.20
[2024-12-05 14:01:05,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4552.00 | bwd: 9571.92 | bwd_inner: 9558.00 | bwd_allreduce: 12.83 | step: 113.41

 19%|█▉        | 72/384 [18:02<1:25:39, 16.47s/it]
                                                  
{'loss': 0.1679, 'grad_norm': 1.7894129753112793, 'learning_rate': 9.37173308072291e-07, 'epoch': 0.37}

 19%|█▉        | 72/384 [18:02<1:25:39, 16.47s/it][2024-12-05 14:01:07,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.61 | bwd_microstep: 653.68 | bwd_inner_microstep: 653.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:01:08,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.18 | bwd_microstep: 441.66 | bwd_inner_microstep: 441.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:01:09,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.54 | bwd_microstep: 677.84 | bwd_inner_microstep: 677.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:01:10,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 238.78 | bwd_microstep: 361.70 | bwd_inner_microstep: 361.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:01:11,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.09 | bwd_microstep: 421.23 | bwd_inner_microstep: 421.10 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:01:12,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.92 | bwd_microstep: 568.06 | bwd_inner_microstep: 567.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:01:13,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.39 | bwd_microstep: 552.73 | bwd_inner_microstep: 552.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:01:14,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.42 | bwd_microstep: 620.83 | bwd_inner_microstep: 620.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:01:15,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.77 | bwd_microstep: 432.45 | bwd_inner_microstep: 432.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:01:16,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.84 | bwd_microstep: 533.68 | bwd_inner_microstep: 533.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:01:17,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.38 | bwd_microstep: 430.11 | bwd_inner_microstep: 429.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:01:19,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 369.89 | bwd_microstep: 779.77 | bwd_inner_microstep: 779.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:01:19,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.31 | bwd_microstep: 411.72 | bwd_inner_microstep: 411.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:01:20,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.47 | bwd_microstep: 465.59 | bwd_inner_microstep: 465.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:01:22,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 351.90 | bwd_microstep: 714.46 | bwd_inner_microstep: 714.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:01:23,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.82 | optimizer_gradients: 14.12 | optimizer_step: 23.53
[2024-12-05 14:01:23,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.82 | bwd_microstep: 898.93 | bwd_inner_microstep: 886.87 | bwd_allreduce_microstep: 11.97 | step_microstep: 115.90
[2024-12-05 14:01:23,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4771.39 | bwd: 8964.43 | bwd_inner: 8950.75 | bwd_allreduce: 12.61 | step: 116.97

 19%|█▉        | 73/384 [18:20<1:27:46, 16.93s/it]
                                                  
{'loss': 0.1833, 'grad_norm': 1.6492183208465576, 'learning_rate': 9.35108520950324e-07, 'epoch': 0.38}

 19%|█▉        | 73/384 [18:20<1:27:46, 16.93s/it][2024-12-05 14:01:24,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.52 | bwd_microstep: 387.27 | bwd_inner_microstep: 387.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:01:25,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.72 | bwd_microstep: 428.41 | bwd_inner_microstep: 428.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:01:26,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.77 | bwd_microstep: 380.63 | bwd_inner_microstep: 380.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:01:27,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.52 | bwd_microstep: 441.60 | bwd_inner_microstep: 441.46 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:01:28,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.62 | bwd_microstep: 415.85 | bwd_inner_microstep: 415.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:01:29,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.75 | bwd_microstep: 408.38 | bwd_inner_microstep: 408.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:01:30,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.62 | bwd_microstep: 617.03 | bwd_inner_microstep: 616.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:01:31,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.84 | bwd_microstep: 432.16 | bwd_inner_microstep: 432.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:01:33,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.45 | bwd_microstep: 405.18 | bwd_inner_microstep: 405.03 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.06
[2024-12-05 14:01:34,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 350.89 | bwd_microstep: 714.99 | bwd_inner_microstep: 714.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:01:35,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.47 | bwd_microstep: 400.59 | bwd_inner_microstep: 400.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:01:36,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.52 | bwd_microstep: 442.21 | bwd_inner_microstep: 442.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:01:37,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.44 | bwd_microstep: 484.62 | bwd_inner_microstep: 484.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:01:38,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.69 | bwd_microstep: 594.63 | bwd_inner_microstep: 594.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.12
[2024-12-05 14:01:39,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.27 | bwd_microstep: 455.98 | bwd_inner_microstep: 455.84 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:01:42,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.27 | optimizer_gradients: 14.03 | optimizer_step: 23.38
[2024-12-05 14:01:42,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.93 | bwd_microstep: 1914.65 | bwd_inner_microstep: 1902.66 | bwd_allreduce_microstep: 11.90 | step_microstep: 111.94
[2024-12-05 14:01:42,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4389.10 | bwd: 8924.18 | bwd_inner: 8910.54 | bwd_allreduce: 12.59 | step: 113.15

 19%|█▉        | 74/384 [18:38<1:29:57, 17.41s/it]
                                                  
{'loss': 0.1507, 'grad_norm': 1.6076239347457886, 'learning_rate': 9.330127018922193e-07, 'epoch': 0.38}

 19%|█▉        | 74/384 [18:38<1:29:57, 17.41s/it][2024-12-05 14:01:43,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.43 | bwd_microstep: 325.26 | bwd_inner_microstep: 325.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:01:44,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.15 | bwd_microstep: 478.56 | bwd_inner_microstep: 478.44 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:01:45,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.72 | bwd_microstep: 355.00 | bwd_inner_microstep: 354.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:01:46,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.81 | bwd_microstep: 479.41 | bwd_inner_microstep: 479.27 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07
[2024-12-05 14:01:47,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.81 | bwd_microstep: 496.83 | bwd_inner_microstep: 496.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:01:48,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.59 | bwd_microstep: 416.53 | bwd_inner_microstep: 416.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:01:49,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.34 | bwd_microstep: 599.54 | bwd_inner_microstep: 599.37 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:01:50,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.98 | bwd_microstep: 625.29 | bwd_inner_microstep: 625.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:01:51,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.51 | bwd_microstep: 363.07 | bwd_inner_microstep: 362.89 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:01:52,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.31 | bwd_microstep: 435.03 | bwd_inner_microstep: 434.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:01:53,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.71 | bwd_microstep: 488.26 | bwd_inner_microstep: 488.07 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 14:01:54,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.52 | bwd_microstep: 406.76 | bwd_inner_microstep: 406.58 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 14:01:55,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.45 | bwd_microstep: 494.95 | bwd_inner_microstep: 494.78 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:01:56,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.77 | bwd_microstep: 492.06 | bwd_inner_microstep: 491.85 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:01:57,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.61 | bwd_microstep: 410.88 | bwd_inner_microstep: 410.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:02:01,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.26 | optimizer_gradients: 14.25 | optimizer_step: 23.18
[2024-12-05 14:02:01,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.08 | bwd_microstep: 2932.36 | bwd_inner_microstep: 2920.20 | bwd_allreduce_microstep: 12.08 | step_microstep: 112.58
[2024-12-05 14:02:01,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4337.90 | bwd: 9799.85 | bwd_inner: 9785.68 | bwd_allreduce: 12.86 | step: 113.76

 20%|█▉        | 75/384 [18:57<1:31:36, 17.79s/it]
                                                  
{'loss': 0.1759, 'grad_norm': 1.8182544708251953, 'learning_rate': 9.308860003717748e-07, 'epoch': 0.39}

 20%|█▉        | 75/384 [18:57<1:31:36, 17.79s/it][2024-12-05 14:02:02,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.90 | bwd_microstep: 536.37 | bwd_inner_microstep: 536.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:02:03,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.30 | bwd_microstep: 415.17 | bwd_inner_microstep: 415.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:02:04,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.36 | bwd_microstep: 413.10 | bwd_inner_microstep: 412.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:02:05,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.86 | bwd_microstep: 521.56 | bwd_inner_microstep: 521.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:02:06,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.52 | bwd_microstep: 419.04 | bwd_inner_microstep: 418.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:02:07,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.30 | bwd_microstep: 465.42 | bwd_inner_microstep: 465.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:02:08,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.79 | bwd_microstep: 440.11 | bwd_inner_microstep: 439.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:02:09,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.32 | bwd_microstep: 485.09 | bwd_inner_microstep: 484.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:02:10,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.98 | bwd_microstep: 492.17 | bwd_inner_microstep: 492.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:02:11,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.33 | bwd_microstep: 411.43 | bwd_inner_microstep: 411.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:02:12,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.26 | bwd_microstep: 407.45 | bwd_inner_microstep: 407.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:02:14,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.18 | bwd_microstep: 362.08 | bwd_inner_microstep: 361.93 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:02:15,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.37 | bwd_microstep: 422.21 | bwd_inner_microstep: 422.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:02:16,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.69 | bwd_microstep: 468.80 | bwd_inner_microstep: 468.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:02:17,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.59 | bwd_microstep: 591.54 | bwd_inner_microstep: 591.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:02:19,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.23 | optimizer_gradients: 14.17 | optimizer_step: 23.30
[2024-12-05 14:02:19,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.65 | bwd_microstep: 1354.60 | bwd_inner_microstep: 1342.42 | bwd_allreduce_microstep: 12.09 | step_microstep: 112.35
[2024-12-05 14:02:19,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4385.48 | bwd: 8206.11 | bwd_inner: 8192.32 | bwd_allreduce: 12.74 | step: 113.50

 20%|█▉        | 76/384 [19:15<1:32:38, 18.05s/it]
                                                  
{'loss': 0.2069, 'grad_norm': 2.183497667312622, 'learning_rate': 9.287285680653254e-07, 'epoch': 0.4}

 20%|█▉        | 76/384 [19:15<1:32:38, 18.05s/it][2024-12-05 14:02:20,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.81 | bwd_microstep: 405.37 | bwd_inner_microstep: 405.00 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.22
[2024-12-05 14:02:21,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.92 | bwd_microstep: 547.04 | bwd_inner_microstep: 546.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:02:22,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.98 | bwd_microstep: 443.03 | bwd_inner_microstep: 442.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:02:23,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.56 | bwd_microstep: 404.67 | bwd_inner_microstep: 404.54 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:02:24,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.80 | bwd_microstep: 447.97 | bwd_inner_microstep: 447.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:02:25,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.42 | bwd_microstep: 436.98 | bwd_inner_microstep: 436.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:02:26,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.06 | bwd_microstep: 384.00 | bwd_inner_microstep: 383.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:02:27,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.50 | bwd_microstep: 445.51 | bwd_inner_microstep: 445.32 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 14:02:28,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.76 | bwd_microstep: 424.89 | bwd_inner_microstep: 424.74 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:02:29,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.70 | bwd_microstep: 447.79 | bwd_inner_microstep: 447.65 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:02:31,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.17 | bwd_microstep: 502.69 | bwd_inner_microstep: 502.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:02:32,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.59 | bwd_microstep: 586.00 | bwd_inner_microstep: 585.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:02:33,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.91 | bwd_microstep: 400.31 | bwd_inner_microstep: 400.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:02:34,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.77 | bwd_microstep: 413.59 | bwd_inner_microstep: 413.44 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:02:35,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.36 | bwd_microstep: 496.79 | bwd_inner_microstep: 496.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:02:39,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.93 | optimizer_gradients: 14.15 | optimizer_step: 23.18
[2024-12-05 14:02:39,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.87 | bwd_microstep: 3231.93 | bwd_inner_microstep: 3219.65 | bwd_allreduce_microstep: 12.16 | step_microstep: 114.75
[2024-12-05 14:02:39,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4295.25 | bwd: 10018.58 | bwd_inner: 10004.36 | bwd_allreduce: 12.94 | step: 116.13

 20%|██        | 77/384 [19:35<1:34:52, 18.54s/it]
                                                  
{'loss': 0.1813, 'grad_norm': 1.9281930923461914, 'learning_rate': 9.265405588409256e-07, 'epoch': 0.4}

 20%|██        | 77/384 [19:35<1:34:52, 18.54s/it][2024-12-05 14:02:40,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.99 | bwd_microstep: 421.35 | bwd_inner_microstep: 421.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:02:41,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.07 | bwd_microstep: 458.94 | bwd_inner_microstep: 458.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:02:42,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.88 | bwd_microstep: 437.41 | bwd_inner_microstep: 437.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:02:43,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.68 | bwd_microstep: 532.95 | bwd_inner_microstep: 532.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:02:45,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.14 | bwd_microstep: 530.68 | bwd_inner_microstep: 530.54 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:02:46,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.94 | bwd_microstep: 434.71 | bwd_inner_microstep: 434.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:02:47,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.03 | bwd_microstep: 439.92 | bwd_inner_microstep: 439.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:02:48,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.13 | bwd_microstep: 438.90 | bwd_inner_microstep: 438.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:02:49,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.21 | bwd_microstep: 454.56 | bwd_inner_microstep: 454.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:02:50,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.72 | bwd_microstep: 486.04 | bwd_inner_microstep: 485.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:02:51,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.62 | bwd_microstep: 436.73 | bwd_inner_microstep: 436.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:02:52,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.47 | bwd_microstep: 493.71 | bwd_inner_microstep: 493.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:02:53,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.40 | bwd_microstep: 434.97 | bwd_inner_microstep: 434.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:02:55,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.84 | bwd_microstep: 455.45 | bwd_inner_microstep: 455.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:02:56,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 349.12 | bwd_microstep: 716.39 | bwd_inner_microstep: 716.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:02:58,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.31 | optimizer_gradients: 14.11 | optimizer_step: 23.42
[2024-12-05 14:02:58,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.12 | bwd_microstep: 1586.15 | bwd_inner_microstep: 1573.88 | bwd_allreduce_microstep: 12.19 | step_microstep: 112.39
[2024-12-05 14:02:58,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4413.41 | bwd: 8758.86 | bwd_inner: 8744.96 | bwd_allreduce: 12.83 | step: 113.70

 20%|██        | 78/384 [19:54<1:35:35, 18.74s/it]
                                                  
{'loss': 0.1764, 'grad_norm': 1.7534650564193726, 'learning_rate': 9.243221287473755e-07, 'epoch': 0.41}

 20%|██        | 78/384 [19:54<1:35:35, 18.74s/it][2024-12-05 14:02:59,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.87 | bwd_microstep: 410.47 | bwd_inner_microstep: 410.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:03:00,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.83 | bwd_microstep: 477.39 | bwd_inner_microstep: 477.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:03:01,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.78 | bwd_microstep: 457.24 | bwd_inner_microstep: 457.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:03:02,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.69 | bwd_microstep: 452.50 | bwd_inner_microstep: 452.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:03:04,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.80 | bwd_microstep: 451.53 | bwd_inner_microstep: 451.38 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:03:05,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 374.67 | bwd_microstep: 783.10 | bwd_inner_microstep: 782.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:03:06,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 239.23 | bwd_microstep: 359.00 | bwd_inner_microstep: 358.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:03:07,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 374.91 | bwd_microstep: 785.89 | bwd_inner_microstep: 785.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:03:08,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 239.17 | bwd_microstep: 354.35 | bwd_inner_microstep: 354.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:03:09,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.08 | bwd_microstep: 433.11 | bwd_inner_microstep: 432.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:03:11,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.84 | bwd_microstep: 594.34 | bwd_inner_microstep: 594.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:03:12,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.36 | bwd_microstep: 433.80 | bwd_inner_microstep: 433.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:03:13,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.05 | bwd_microstep: 438.89 | bwd_inner_microstep: 438.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 14:03:14,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.49 | bwd_microstep: 429.67 | bwd_inner_microstep: 429.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:03:15,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.11 | bwd_microstep: 377.88 | bwd_inner_microstep: 377.75 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:03:17,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.16 | optimizer_gradients: 14.05 | optimizer_step: 23.28
[2024-12-05 14:03:17,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.78 | bwd_microstep: 1695.95 | bwd_inner_microstep: 1683.72 | bwd_allreduce_microstep: 12.15 | step_microstep: 112.27
[2024-12-05 14:03:17,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4536.73 | bwd: 8935.11 | bwd_inner: 8921.22 | bwd_allreduce: 12.82 | step: 113.48

 21%|██        | 79/384 [20:13<1:35:43, 18.83s/it]
                                                  
{'loss': 0.1863, 'grad_norm': 1.7174078226089478, 'learning_rate': 9.220734360030906e-07, 'epoch': 0.41}

 21%|██        | 79/384 [20:13<1:35:43, 18.83s/it][2024-12-05 14:03:18,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.92 | bwd_microstep: 373.12 | bwd_inner_microstep: 373.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:03:19,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.84 | bwd_microstep: 439.72 | bwd_inner_microstep: 439.58 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:03:20,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.52 | bwd_microstep: 402.29 | bwd_inner_microstep: 402.13 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:03:21,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.22 | bwd_microstep: 383.92 | bwd_inner_microstep: 383.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:03:22,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.87 | bwd_microstep: 403.18 | bwd_inner_microstep: 403.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:03:23,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.82 | bwd_microstep: 431.50 | bwd_inner_microstep: 431.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:03:25,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.27 | bwd_microstep: 684.04 | bwd_inner_microstep: 683.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:03:25,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.66 | bwd_microstep: 527.36 | bwd_inner_microstep: 527.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:03:27,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.37 | bwd_microstep: 442.57 | bwd_inner_microstep: 442.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:03:27,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.37 | bwd_microstep: 451.21 | bwd_inner_microstep: 451.05 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:03:29,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 409.66 | bwd_microstep: 908.25 | bwd_inner_microstep: 908.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:03:30,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.78 | bwd_microstep: 414.86 | bwd_inner_microstep: 414.69 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08
[2024-12-05 14:03:31,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.01 | bwd_microstep: 402.27 | bwd_inner_microstep: 402.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:03:32,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.92 | bwd_microstep: 617.80 | bwd_inner_microstep: 617.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:03:34,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 348.57 | bwd_microstep: 714.28 | bwd_inner_microstep: 714.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:03:36,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.41 | optimizer_gradients: 14.08 | optimizer_step: 23.25
[2024-12-05 14:03:36,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.05 | bwd_microstep: 2240.33 | bwd_inner_microstep: 2228.16 | bwd_allreduce_microstep: 12.07 | step_microstep: 112.36
[2024-12-05 14:03:36,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4552.94 | bwd: 9836.67 | bwd_inner: 9822.75 | bwd_allreduce: 12.80 | step: 113.64

 21%|██        | 80/384 [20:33<1:36:09, 18.98s/it]
                                                  
{'loss': 0.1718, 'grad_norm': 1.7550305128097534, 'learning_rate': 9.197946409848194e-07, 'epoch': 0.42}

 21%|██        | 80/384 [20:33<1:36:09, 18.98s/it][2024-12-05 14:03:37,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.35 | bwd_microstep: 345.40 | bwd_inner_microstep: 345.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:03:39,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.56 | bwd_microstep: 430.40 | bwd_inner_microstep: 430.26 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:03:40,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.97 | bwd_microstep: 674.51 | bwd_inner_microstep: 674.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:03:41,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.42 | bwd_microstep: 437.29 | bwd_inner_microstep: 437.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:03:42,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.59 | bwd_microstep: 620.74 | bwd_inner_microstep: 620.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:03:44,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.35 | bwd_microstep: 440.12 | bwd_inner_microstep: 439.98 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:03:45,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.59 | bwd_microstep: 403.08 | bwd_inner_microstep: 402.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 14:03:46,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.09 | bwd_microstep: 437.35 | bwd_inner_microstep: 437.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:03:47,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.14 | bwd_microstep: 466.49 | bwd_inner_microstep: 466.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:03:48,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.98 | bwd_microstep: 544.12 | bwd_inner_microstep: 544.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:03:49,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.59 | bwd_microstep: 429.51 | bwd_inner_microstep: 429.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:03:50,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.48 | bwd_microstep: 402.18 | bwd_inner_microstep: 402.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:03:51,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.43 | bwd_microstep: 501.20 | bwd_inner_microstep: 501.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:03:52,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.59 | bwd_microstep: 408.73 | bwd_inner_microstep: 408.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:03:53,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.55 | bwd_microstep: 441.43 | bwd_inner_microstep: 441.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:03:55,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.56 | optimizer_gradients: 14.13 | optimizer_step: 23.45
[2024-12-05 14:03:55,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.69 | bwd_microstep: 1164.33 | bwd_inner_microstep: 1152.13 | bwd_allreduce_microstep: 12.11 | step_microstep: 111.65
[2024-12-05 14:03:55,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4389.45 | bwd: 8146.87 | bwd_inner: 8133.06 | bwd_allreduce: 12.75 | step: 112.80

 21%|██        | 81/384 [20:51<1:35:30, 18.91s/it]
                                                  
{'loss': 0.156, 'grad_norm': 1.4776816368103027, 'learning_rate': 9.174859062162037e-07, 'epoch': 0.42}

 21%|██        | 81/384 [20:51<1:35:30, 18.91s/it][2024-12-05 14:03:56,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.98 | bwd_microstep: 368.27 | bwd_inner_microstep: 368.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:03:57,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.63 | bwd_microstep: 430.50 | bwd_inner_microstep: 430.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:03:58,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.76 | bwd_microstep: 418.24 | bwd_inner_microstep: 418.08 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09
[2024-12-05 14:04:00,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 397.35 | bwd_microstep: 870.68 | bwd_inner_microstep: 870.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:04:01,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.87 | bwd_microstep: 447.70 | bwd_inner_microstep: 447.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:04:02,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 239.96 | bwd_microstep: 354.45 | bwd_inner_microstep: 354.30 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:04:04,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.63 | bwd_microstep: 453.88 | bwd_inner_microstep: 453.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:04:05,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.02 | bwd_microstep: 462.31 | bwd_inner_microstep: 462.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:04:06,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.31 | bwd_microstep: 457.87 | bwd_inner_microstep: 457.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:04:07,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.56 | bwd_microstep: 494.65 | bwd_inner_microstep: 494.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.13
[2024-12-05 14:04:08,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.26 | bwd_microstep: 461.26 | bwd_inner_microstep: 461.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:04:09,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.45 | bwd_microstep: 406.20 | bwd_inner_microstep: 406.06 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:04:10,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.07 | bwd_microstep: 402.05 | bwd_inner_microstep: 401.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:04:11,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.02 | bwd_microstep: 459.78 | bwd_inner_microstep: 459.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:04:12,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.36 | bwd_microstep: 406.68 | bwd_inner_microstep: 406.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:15,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.62 | optimizer_gradients: 14.11 | optimizer_step: 23.15
[2024-12-05 14:04:15,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.69 | bwd_microstep: 1897.77 | bwd_inner_microstep: 1885.74 | bwd_allreduce_microstep: 11.92 | step_microstep: 113.95
[2024-12-05 14:04:15,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4361.99 | bwd: 8792.26 | bwd_inner: 8778.59 | bwd_allreduce: 12.61 | step: 115.34

 21%|██▏       | 82/384 [21:11<1:36:31, 19.18s/it]
                                                  
{'loss': 0.1821, 'grad_norm': 1.7932229042053223, 'learning_rate': 9.151473963561882e-07, 'epoch': 0.43}

 21%|██▏       | 82/384 [21:11<1:36:31, 19.18s/it][2024-12-05 14:04:16,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.70 | bwd_microstep: 414.88 | bwd_inner_microstep: 414.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:04:17,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.60 | bwd_microstep: 433.36 | bwd_inner_microstep: 433.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:04:19,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 371.51 | bwd_microstep: 778.97 | bwd_inner_microstep: 778.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:20,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.23 | bwd_microstep: 427.09 | bwd_inner_microstep: 426.96 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:04:21,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 329.71 | bwd_microstep: 599.80 | bwd_inner_microstep: 599.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:04:22,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.33 | bwd_microstep: 440.80 | bwd_inner_microstep: 440.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:24,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.80 | bwd_microstep: 612.87 | bwd_inner_microstep: 612.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:24,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.54 | bwd_microstep: 418.69 | bwd_inner_microstep: 418.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:25,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.20 | bwd_microstep: 440.48 | bwd_inner_microstep: 440.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:27,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.08 | bwd_microstep: 592.24 | bwd_inner_microstep: 592.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:28,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.93 | bwd_microstep: 410.94 | bwd_inner_microstep: 410.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:04:29,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.06 | bwd_microstep: 332.18 | bwd_inner_microstep: 332.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:04:30,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.12 | bwd_microstep: 561.21 | bwd_inner_microstep: 561.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:31,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.05 | bwd_microstep: 407.41 | bwd_inner_microstep: 407.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:32,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.58 | bwd_microstep: 539.40 | bwd_inner_microstep: 539.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:34,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.17 | optimizer_gradients: 14.09 | optimizer_step: 23.39
[2024-12-05 14:04:34,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.09 | bwd_microstep: 1272.59 | bwd_inner_microstep: 1260.61 | bwd_allreduce_microstep: 11.90 | step_microstep: 112.32
[2024-12-05 14:04:34,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4542.61 | bwd: 8682.91 | bwd_inner: 8669.25 | bwd_allreduce: 12.53 | step: 113.42

 22%|██▏       | 83/384 [21:30<1:35:42, 19.08s/it]
                                                  
{'loss': 0.1749, 'grad_norm': 1.6103583574295044, 'learning_rate': 9.127792781872768e-07, 'epoch': 0.43}

 22%|██▏       | 83/384 [21:30<1:35:42, 19.08s/it][2024-12-05 14:04:35,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.86 | bwd_microstep: 420.30 | bwd_inner_microstep: 420.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:04:36,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.52 | bwd_microstep: 409.45 | bwd_inner_microstep: 409.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:04:37,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.27 | bwd_microstep: 606.06 | bwd_inner_microstep: 605.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:38,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.98 | bwd_microstep: 396.13 | bwd_inner_microstep: 396.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:39,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.76 | bwd_microstep: 407.29 | bwd_inner_microstep: 407.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:04:40,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.12 | bwd_microstep: 418.83 | bwd_inner_microstep: 418.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:04:41,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.93 | bwd_microstep: 421.77 | bwd_inner_microstep: 421.56 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09
[2024-12-05 14:04:42,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.80 | bwd_microstep: 369.27 | bwd_inner_microstep: 369.12 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:04:43,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.71 | bwd_microstep: 489.01 | bwd_inner_microstep: 488.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:45,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.84 | bwd_microstep: 468.27 | bwd_inner_microstep: 468.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:46,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.86 | bwd_microstep: 445.14 | bwd_inner_microstep: 444.99 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:04:46,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.07 | bwd_microstep: 409.46 | bwd_inner_microstep: 409.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:47,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.18 | bwd_microstep: 408.60 | bwd_inner_microstep: 408.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:04:49,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.21 | bwd_microstep: 496.81 | bwd_inner_microstep: 496.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:49,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.21 | bwd_microstep: 455.92 | bwd_inner_microstep: 455.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:53,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.30 | optimizer_gradients: 14.04 | optimizer_step: 23.34
[2024-12-05 14:04:53,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.04 | bwd_microstep: 2837.35 | bwd_inner_microstep: 2825.33 | bwd_allreduce_microstep: 11.90 | step_microstep: 112.17
[2024-12-05 14:04:53,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4227.46 | bwd: 9459.66 | bwd_inner: 9445.85 | bwd_allreduce: 12.59 | step: 113.27

 22%|██▏       | 84/384 [21:49<1:35:38, 19.13s/it]
                                                  
{'loss': 0.177, 'grad_norm': 1.8216729164123535, 'learning_rate': 9.103817206036382e-07, 'epoch': 0.44}

 22%|██▏       | 84/384 [21:49<1:35:38, 19.13s/it][2024-12-05 14:04:54,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.33 | bwd_microstep: 528.64 | bwd_inner_microstep: 528.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:55,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.39 | bwd_microstep: 433.15 | bwd_inner_microstep: 432.99 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:04:57,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 329.30 | bwd_microstep: 569.20 | bwd_inner_microstep: 569.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:58,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.69 | bwd_microstep: 418.68 | bwd_inner_microstep: 418.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:04:59,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.94 | bwd_microstep: 453.06 | bwd_inner_microstep: 452.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:00,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.54 | bwd_microstep: 443.13 | bwd_inner_microstep: 443.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:05:01,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.46 | bwd_microstep: 560.62 | bwd_inner_microstep: 560.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:05:02,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.92 | bwd_microstep: 407.61 | bwd_inner_microstep: 407.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:03,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.26 | bwd_microstep: 434.25 | bwd_inner_microstep: 434.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:05:04,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.96 | bwd_microstep: 416.98 | bwd_inner_microstep: 416.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:05:06,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.35 | bwd_microstep: 591.95 | bwd_inner_microstep: 591.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:06,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 240.03 | bwd_microstep: 378.44 | bwd_inner_microstep: 378.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:05:08,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 325.66 | bwd_microstep: 624.07 | bwd_inner_microstep: 623.93 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:05:09,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.65 | bwd_microstep: 490.21 | bwd_inner_microstep: 490.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:05:10,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 351.95 | bwd_microstep: 713.73 | bwd_inner_microstep: 713.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:12,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.40 | optimizer_gradients: 14.06 | optimizer_step: 23.27
[2024-12-05 14:05:12,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.06 | bwd_microstep: 1726.74 | bwd_inner_microstep: 1714.67 | bwd_allreduce_microstep: 11.99 | step_microstep: 112.96
[2024-12-05 14:05:12,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4584.54 | bwd: 9190.41 | bwd_inner: 9176.69 | bwd_allreduce: 12.63 | step: 114.08

 22%|██▏       | 85/384 [22:09<1:35:29, 19.16s/it]
                                                  
{'loss': 0.1845, 'grad_norm': 1.7169886827468872, 'learning_rate': 9.079548945990592e-07, 'epoch': 0.44}

 22%|██▏       | 85/384 [22:09<1:35:29, 19.16s/it][2024-12-05 14:05:14,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.97 | bwd_microstep: 568.10 | bwd_inner_microstep: 567.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:05:15,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.56 | bwd_microstep: 534.57 | bwd_inner_microstep: 534.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:05:16,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.12 | bwd_microstep: 373.74 | bwd_inner_microstep: 373.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:05:17,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.03 | bwd_microstep: 449.73 | bwd_inner_microstep: 449.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:18,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.01 | bwd_microstep: 377.66 | bwd_inner_microstep: 377.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:05:19,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.98 | bwd_microstep: 362.25 | bwd_inner_microstep: 362.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:05:20,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.16 | bwd_microstep: 419.32 | bwd_inner_microstep: 419.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:05:22,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.00 | bwd_microstep: 420.37 | bwd_inner_microstep: 420.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:22,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.76 | bwd_microstep: 461.73 | bwd_inner_microstep: 461.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:05:23,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.67 | bwd_microstep: 445.02 | bwd_inner_microstep: 444.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:24,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.64 | bwd_microstep: 411.77 | bwd_inner_microstep: 411.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:25,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.66 | bwd_microstep: 434.21 | bwd_inner_microstep: 434.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:05:27,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.27 | bwd_microstep: 419.90 | bwd_inner_microstep: 419.77 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:05:28,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.61 | bwd_microstep: 461.32 | bwd_inner_microstep: 461.18 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:05:28,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.59 | bwd_microstep: 415.43 | bwd_inner_microstep: 415.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:32,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.45 | optimizer_gradients: 14.30 | optimizer_step: 23.30
[2024-12-05 14:05:32,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.84 | bwd_microstep: 2405.80 | bwd_inner_microstep: 2393.20 | bwd_allreduce_microstep: 12.49 | step_microstep: 115.95
[2024-12-05 14:05:32,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4244.94 | bwd: 8960.93 | bwd_inner: 8946.66 | bwd_allreduce: 13.14 | step: 117.09

 22%|██▏       | 86/384 [22:28<1:35:16, 19.18s/it]
                                                  
{'loss': 0.1518, 'grad_norm': 1.4935150146484375, 'learning_rate': 9.054989732547506e-07, 'epoch': 0.45}

 22%|██▏       | 86/384 [22:28<1:35:16, 19.18s/it][2024-12-05 14:05:33,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.87 | bwd_microstep: 409.94 | bwd_inner_microstep: 409.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:05:34,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.00 | bwd_microstep: 587.34 | bwd_inner_microstep: 587.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:35,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.07 | bwd_microstep: 488.85 | bwd_inner_microstep: 488.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:36,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.52 | bwd_microstep: 431.55 | bwd_inner_microstep: 431.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:37,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.58 | bwd_microstep: 437.54 | bwd_inner_microstep: 437.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:38,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 357.51 | bwd_microstep: 743.54 | bwd_inner_microstep: 743.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:39,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.11 | bwd_microstep: 438.49 | bwd_inner_microstep: 438.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:40,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.24 | bwd_microstep: 443.66 | bwd_inner_microstep: 443.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:05:41,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.56 | bwd_microstep: 452.06 | bwd_inner_microstep: 451.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:42,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.77 | bwd_microstep: 559.99 | bwd_inner_microstep: 559.85 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:05:44,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.94 | bwd_microstep: 439.74 | bwd_inner_microstep: 439.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:05:44,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.49 | bwd_microstep: 416.28 | bwd_inner_microstep: 416.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:45,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.08 | bwd_microstep: 438.49 | bwd_inner_microstep: 438.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:05:46,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 243.28 | bwd_microstep: 361.93 | bwd_inner_microstep: 361.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:47,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.36 | bwd_microstep: 404.02 | bwd_inner_microstep: 403.86 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:05:51,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.85 | optimizer_gradients: 14.27 | optimizer_step: 23.31
[2024-12-05 14:05:51,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.67 | bwd_microstep: 3228.58 | bwd_inner_microstep: 3216.02 | bwd_allreduce_microstep: 12.44 | step_microstep: 115.20
[2024-12-05 14:05:51,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4409.13 | bwd: 10282.01 | bwd_inner: 10267.76 | bwd_allreduce: 13.11 | step: 116.40

 23%|██▎       | 87/384 [22:47<1:35:01, 19.20s/it]
                                                  
{'loss': 0.1651, 'grad_norm': 1.7215607166290283, 'learning_rate': 9.030141317270025e-07, 'epoch': 0.45}

 23%|██▎       | 87/384 [22:47<1:35:01, 19.20s/it][2024-12-05 14:05:52,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.05 | bwd_microstep: 379.02 | bwd_inner_microstep: 378.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:53,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.34 | bwd_microstep: 484.44 | bwd_inner_microstep: 484.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:55,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.98 | bwd_microstep: 619.05 | bwd_inner_microstep: 618.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:05:56,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.66 | bwd_microstep: 418.63 | bwd_inner_microstep: 418.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:05:56,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.53 | bwd_microstep: 431.42 | bwd_inner_microstep: 431.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:05:57,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.02 | bwd_microstep: 436.02 | bwd_inner_microstep: 435.88 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.06
[2024-12-05 14:05:58,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.88 | bwd_microstep: 479.40 | bwd_inner_microstep: 479.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:05:59,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.15 | bwd_microstep: 524.90 | bwd_inner_microstep: 524.78 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:06:01,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.94 | bwd_microstep: 426.21 | bwd_inner_microstep: 426.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:06:02,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 361.59 | bwd_microstep: 738.94 | bwd_inner_microstep: 738.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:06:03,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.10 | bwd_microstep: 438.71 | bwd_inner_microstep: 438.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:06:04,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.48 | bwd_microstep: 436.56 | bwd_inner_microstep: 436.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:06:05,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.10 | bwd_microstep: 439.71 | bwd_inner_microstep: 439.58 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:06:06,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.21 | bwd_microstep: 556.70 | bwd_inner_microstep: 556.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:06:07,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.48 | bwd_microstep: 481.33 | bwd_inner_microstep: 481.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:06:10,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.31 | optimizer_gradients: 14.05 | optimizer_step: 23.10
[2024-12-05 14:06:10,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.77 | bwd_microstep: 2055.33 | bwd_inner_microstep: 2043.15 | bwd_allreduce_microstep: 12.09 | step_microstep: 111.99
[2024-12-05 14:06:10,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4502.33 | bwd: 9346.36 | bwd_inner: 9332.56 | bwd_allreduce: 12.72 | step: 113.07

 23%|██▎       | 88/384 [23:06<1:34:46, 19.21s/it]
                                                  
{'loss': 0.1862, 'grad_norm': 1.885299801826477, 'learning_rate': 9.005005472346923e-07, 'epoch': 0.46}

 23%|██▎       | 88/384 [23:06<1:34:46, 19.21s/it][2024-12-05 14:06:11,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.74 | bwd_microstep: 405.03 | bwd_inner_microstep: 404.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:06:12,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.32 | bwd_microstep: 580.52 | bwd_inner_microstep: 580.39 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:06:13,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.74 | bwd_microstep: 448.48 | bwd_inner_microstep: 448.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:06:14,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.96 | bwd_microstep: 524.96 | bwd_inner_microstep: 524.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:06:15,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.44 | bwd_microstep: 504.30 | bwd_inner_microstep: 504.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:06:16,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.95 | bwd_microstep: 582.59 | bwd_inner_microstep: 582.44 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:06:17,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.16 | bwd_microstep: 479.06 | bwd_inner_microstep: 478.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:06:18,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.95 | bwd_microstep: 425.55 | bwd_inner_microstep: 425.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:06:19,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.21 | bwd_microstep: 435.10 | bwd_inner_microstep: 434.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:06:21,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.98 | bwd_microstep: 490.07 | bwd_inner_microstep: 489.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:06:22,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.53 | bwd_microstep: 410.15 | bwd_inner_microstep: 410.01 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:06:23,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.55 | bwd_microstep: 480.60 | bwd_inner_microstep: 480.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:06:24,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.39 | bwd_microstep: 485.71 | bwd_inner_microstep: 485.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:06:25,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.64 | bwd_microstep: 469.99 | bwd_inner_microstep: 469.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:06:26,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.26 | bwd_microstep: 575.80 | bwd_inner_microstep: 575.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:06:30,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.20 | optimizer_gradients: 14.10 | optimizer_step: 23.90
[2024-12-05 14:06:30,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.88 | bwd_microstep: 3025.37 | bwd_inner_microstep: 3013.12 | bwd_allreduce_microstep: 12.16 | step_microstep: 113.02
[2024-12-05 14:06:30,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4537.79 | bwd: 10323.26 | bwd_inner: 10309.31 | bwd_allreduce: 12.80 | step: 114.11

 23%|██▎       | 89/384 [23:26<1:34:59, 19.32s/it]
                                                  
{'loss': 0.1834, 'grad_norm': 1.6924781799316406, 'learning_rate': 8.979583990466452e-07, 'epoch': 0.46}

 23%|██▎       | 89/384 [23:26<1:34:59, 19.32s/it][2024-12-05 14:06:31,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.35 | bwd_microstep: 365.08 | bwd_inner_microstep: 364.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:06:32,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.80 | bwd_microstep: 556.70 | bwd_inner_microstep: 556.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:06:33,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.82 | bwd_microstep: 353.98 | bwd_inner_microstep: 353.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:06:34,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.84 | bwd_microstep: 461.73 | bwd_inner_microstep: 461.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:06:35,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.20 | bwd_microstep: 483.20 | bwd_inner_microstep: 483.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:06:36,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.49 | bwd_microstep: 527.78 | bwd_inner_microstep: 527.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:06:37,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.93 | bwd_microstep: 411.77 | bwd_inner_microstep: 411.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:06:38,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.58 | bwd_microstep: 433.87 | bwd_inner_microstep: 433.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:06:39,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.92 | bwd_microstep: 490.02 | bwd_inner_microstep: 489.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:06:40,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.86 | bwd_microstep: 360.03 | bwd_inner_microstep: 359.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:06:41,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.49 | bwd_microstep: 693.18 | bwd_inner_microstep: 693.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:06:43,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.34 | bwd_microstep: 688.39 | bwd_inner_microstep: 688.27 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:06:44,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.23 | bwd_microstep: 456.48 | bwd_inner_microstep: 456.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:06:44,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.90 | bwd_microstep: 463.07 | bwd_inner_microstep: 462.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:06:46,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.90 | bwd_microstep: 434.28 | bwd_inner_microstep: 434.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:06:49,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.08 | optimizer_gradients: 14.05 | optimizer_step: 23.39
[2024-12-05 14:06:49,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.08 | bwd_microstep: 3031.43 | bwd_inner_microstep: 3019.39 | bwd_allreduce_microstep: 11.95 | step_microstep: 111.89
[2024-12-05 14:06:49,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4393.77 | bwd: 10210.99 | bwd_inner: 10197.38 | bwd_allreduce: 12.57 | step: 113.10

 23%|██▎       | 90/384 [23:45<1:34:48, 19.35s/it]
                                                  
{'loss': 0.1736, 'grad_norm': 1.6620601415634155, 'learning_rate': 8.953878684688492e-07, 'epoch': 0.47}

 23%|██▎       | 90/384 [23:45<1:34:48, 19.35s/it][2024-12-05 14:06:50,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.92 | bwd_microstep: 467.29 | bwd_inner_microstep: 467.14 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:06:52,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.27 | bwd_microstep: 630.89 | bwd_inner_microstep: 630.71 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08
[2024-12-05 14:06:53,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.66 | bwd_microstep: 611.46 | bwd_inner_microstep: 611.27 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:06:54,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.45 | bwd_microstep: 586.36 | bwd_inner_microstep: 586.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:06:55,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.21 | bwd_microstep: 453.69 | bwd_inner_microstep: 453.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:06:56,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.01 | bwd_microstep: 444.43 | bwd_inner_microstep: 444.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:06:58,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.06 | bwd_microstep: 623.63 | bwd_inner_microstep: 623.48 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 14:06:59,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.56 | bwd_microstep: 489.61 | bwd_inner_microstep: 489.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:07:00,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.24 | bwd_microstep: 409.28 | bwd_inner_microstep: 409.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:07:01,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.44 | bwd_microstep: 442.53 | bwd_inner_microstep: 442.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:07:03,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.79 | bwd_microstep: 448.84 | bwd_inner_microstep: 448.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:07:04,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.44 | bwd_microstep: 601.56 | bwd_inner_microstep: 601.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:07:04,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.09 | bwd_microstep: 438.70 | bwd_inner_microstep: 438.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:07:06,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.13 | bwd_microstep: 702.58 | bwd_inner_microstep: 702.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:07:07,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.22 | bwd_microstep: 463.77 | bwd_inner_microstep: 463.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:07:09,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.40 | optimizer_gradients: 14.06 | optimizer_step: 23.09
[2024-12-05 14:07:09,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.01 | bwd_microstep: 1420.61 | bwd_inner_microstep: 1408.41 | bwd_allreduce_microstep: 12.11 | step_microstep: 112.14
[2024-12-05 14:07:09,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4699.57 | bwd: 9235.23 | bwd_inner: 9221.25 | bwd_allreduce: 12.83 | step: 113.30

 24%|██▎       | 91/384 [24:05<1:35:44, 19.60s/it]
                                                  
{'loss': 0.1792, 'grad_norm': 1.7256325483322144, 'learning_rate': 8.92789138831524e-07, 'epoch': 0.47}

 24%|██▎       | 91/384 [24:05<1:35:44, 19.60s/it][2024-12-05 14:07:11,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.99 | bwd_microstep: 567.03 | bwd_inner_microstep: 566.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:07:12,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.45 | bwd_microstep: 438.74 | bwd_inner_microstep: 438.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:07:13,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.68 | bwd_microstep: 409.36 | bwd_inner_microstep: 409.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:07:14,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.17 | bwd_microstep: 488.38 | bwd_inner_microstep: 488.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:07:15,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.00 | bwd_microstep: 623.41 | bwd_inner_microstep: 623.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:07:16,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.47 | bwd_microstep: 479.91 | bwd_inner_microstep: 479.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:07:18,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.87 | bwd_microstep: 539.79 | bwd_inner_microstep: 539.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:07:18,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.54 | bwd_microstep: 425.37 | bwd_inner_microstep: 425.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:07:20,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.14 | bwd_microstep: 684.38 | bwd_inner_microstep: 684.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:07:21,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.27 | bwd_microstep: 526.56 | bwd_inner_microstep: 526.42 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:07:22,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.84 | bwd_microstep: 480.54 | bwd_inner_microstep: 480.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:07:23,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.34 | bwd_microstep: 560.86 | bwd_inner_microstep: 560.73 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:07:24,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.29 | bwd_microstep: 465.87 | bwd_inner_microstep: 465.72 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:07:25,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.75 | bwd_microstep: 433.14 | bwd_inner_microstep: 433.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:07:27,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.13 | bwd_microstep: 417.13 | bwd_inner_microstep: 416.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:07:29,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 73.54 | optimizer_gradients: 14.04 | optimizer_step: 23.38
[2024-12-05 14:07:29,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.43 | bwd_microstep: 1258.71 | bwd_inner_microstep: 1246.46 | bwd_allreduce_microstep: 12.15 | step_microstep: 127.44
[2024-12-05 14:07:29,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4558.42 | bwd: 8799.19 | bwd_inner: 8785.24 | bwd_allreduce: 12.80 | step: 128.65

 24%|██▍       | 92/384 [24:25<1:35:04, 19.53s/it]
                                                  
{'loss': 0.1704, 'grad_norm': 1.6039340496063232, 'learning_rate': 8.901623954760459e-07, 'epoch': 0.48}

 24%|██▍       | 92/384 [24:25<1:35:04, 19.53s/it][2024-12-05 14:07:29,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.83 | bwd_microstep: 420.23 | bwd_inner_microstep: 420.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:07:31,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.94 | bwd_microstep: 597.58 | bwd_inner_microstep: 597.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:07:32,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.12 | bwd_microstep: 429.22 | bwd_inner_microstep: 429.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:07:33,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.04 | bwd_microstep: 452.16 | bwd_inner_microstep: 452.00 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:07:34,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.97 | bwd_microstep: 458.67 | bwd_inner_microstep: 458.51 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 14:07:35,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.13 | bwd_microstep: 531.49 | bwd_inner_microstep: 531.32 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:07:36,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.27 | bwd_microstep: 439.95 | bwd_inner_microstep: 439.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:07:37,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.97 | bwd_microstep: 453.06 | bwd_inner_microstep: 452.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:07:38,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.97 | bwd_microstep: 488.37 | bwd_inner_microstep: 488.25 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:07:39,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.00 | bwd_microstep: 353.66 | bwd_inner_microstep: 353.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:07:40,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.84 | bwd_microstep: 457.97 | bwd_inner_microstep: 457.83 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:07:41,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.61 | bwd_microstep: 495.07 | bwd_inner_microstep: 494.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:07:42,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.82 | bwd_microstep: 439.15 | bwd_inner_microstep: 439.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:07:44,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.85 | bwd_microstep: 362.65 | bwd_inner_microstep: 362.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:07:45,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 238.80 | bwd_microstep: 364.99 | bwd_inner_microstep: 364.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:07:48,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.22 | optimizer_gradients: 14.13 | optimizer_step: 23.08
[2024-12-05 14:07:48,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.37 | bwd_microstep: 2321.46 | bwd_inner_microstep: 2309.25 | bwd_allreduce_microstep: 12.12 | step_microstep: 111.89
[2024-12-05 14:07:48,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4302.61 | bwd: 9065.68 | bwd_inner: 9051.75 | bwd_allreduce: 12.79 | step: 113.06

 24%|██▍       | 93/384 [24:44<1:34:27, 19.48s/it]
                                                  
{'loss': 0.1621, 'grad_norm': 1.6422524452209473, 'learning_rate': 8.875078257417294e-07, 'epoch': 0.48}

 24%|██▍       | 93/384 [24:44<1:34:27, 19.48s/it][2024-12-05 14:07:49,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.71 | bwd_microstep: 658.01 | bwd_inner_microstep: 657.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:07:50,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.71 | bwd_microstep: 563.90 | bwd_inner_microstep: 563.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:07:51,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.73 | bwd_microstep: 455.21 | bwd_inner_microstep: 455.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:07:52,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.99 | bwd_microstep: 439.18 | bwd_inner_microstep: 439.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:07:53,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.06 | bwd_microstep: 453.84 | bwd_inner_microstep: 453.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:07:54,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.92 | bwd_microstep: 465.69 | bwd_inner_microstep: 465.55 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:07:55,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.22 | bwd_microstep: 437.43 | bwd_inner_microstep: 437.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:07:56,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.80 | bwd_microstep: 460.18 | bwd_inner_microstep: 460.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:07:57,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.71 | bwd_microstep: 489.29 | bwd_inner_microstep: 489.14 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 14:07:59,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.97 | bwd_microstep: 585.19 | bwd_inner_microstep: 585.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:08:00,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.24 | bwd_microstep: 437.94 | bwd_inner_microstep: 437.78 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 14:08:01,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.63 | bwd_microstep: 428.48 | bwd_inner_microstep: 428.36 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:08:02,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.76 | bwd_microstep: 427.80 | bwd_inner_microstep: 427.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:08:03,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.93 | bwd_microstep: 479.63 | bwd_inner_microstep: 479.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:08:04,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.65 | bwd_microstep: 488.76 | bwd_inner_microstep: 488.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:08:07,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.61 | optimizer_gradients: 14.14 | optimizer_step: 23.08
[2024-12-05 14:08:07,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.27 | bwd_microstep: 2599.87 | bwd_inner_microstep: 2587.44 | bwd_allreduce_microstep: 12.31 | step_microstep: 112.53
[2024-12-05 14:08:07,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4492.37 | bwd: 9870.39 | bwd_inner: 9856.29 | bwd_allreduce: 12.98 | step: 113.83

 24%|██▍       | 94/384 [25:04<1:34:06, 19.47s/it]
                                                  
{'loss': 0.183, 'grad_norm': 1.6837060451507568, 'learning_rate': 8.84825618952466e-07, 'epoch': 0.49}

 24%|██▍       | 94/384 [25:04<1:34:06, 19.47s/it][2024-12-05 14:08:08,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.69 | bwd_microstep: 382.28 | bwd_inner_microstep: 382.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:08:10,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.49 | bwd_microstep: 584.79 | bwd_inner_microstep: 584.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:08:11,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.01 | bwd_microstep: 452.02 | bwd_inner_microstep: 451.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:08:12,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.73 | bwd_microstep: 436.87 | bwd_inner_microstep: 436.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:08:13,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.02 | bwd_microstep: 624.31 | bwd_inner_microstep: 624.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:08:14,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.40 | bwd_microstep: 431.53 | bwd_inner_microstep: 431.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:08:15,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.18 | bwd_microstep: 404.01 | bwd_inner_microstep: 403.82 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.12
[2024-12-05 14:08:15,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.72 | bwd_microstep: 405.10 | bwd_inner_microstep: 404.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:08:16,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.41 | bwd_microstep: 494.72 | bwd_inner_microstep: 494.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:08:17,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.77 | bwd_microstep: 448.81 | bwd_inner_microstep: 448.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:08:18,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.19 | bwd_microstep: 494.58 | bwd_inner_microstep: 494.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:08:19,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.05 | bwd_microstep: 441.98 | bwd_inner_microstep: 441.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:08:21,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.39 | bwd_microstep: 455.62 | bwd_inner_microstep: 455.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:08:21,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.47 | bwd_microstep: 483.10 | bwd_inner_microstep: 482.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:08:23,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.48 | bwd_microstep: 507.76 | bwd_inner_microstep: 507.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:08:27,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.24 | optimizer_gradients: 14.07 | optimizer_step: 23.12
[2024-12-05 14:08:27,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.28 | bwd_microstep: 3287.13 | bwd_inner_microstep: 3274.91 | bwd_allreduce_microstep: 12.12 | step_microstep: 112.04
[2024-12-05 14:08:27,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4433.33 | bwd: 10334.61 | bwd_inner: 10320.76 | bwd_allreduce: 12.78 | step: 113.26

 25%|██▍       | 95/384 [25:23<1:33:33, 19.42s/it]
                                                  
{'loss': 0.1689, 'grad_norm': 1.750653862953186, 'learning_rate': 8.821159664032223e-07, 'epoch': 0.49}

 25%|██▍       | 95/384 [25:23<1:33:33, 19.42s/it][2024-12-05 14:08:28,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.15 | bwd_microstep: 431.65 | bwd_inner_microstep: 431.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:08:29,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.23 | bwd_microstep: 461.20 | bwd_inner_microstep: 461.04 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08
[2024-12-05 14:08:30,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.44 | bwd_microstep: 480.01 | bwd_inner_microstep: 479.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:08:31,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.40 | bwd_microstep: 437.94 | bwd_inner_microstep: 437.81 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:08:33,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.54 | bwd_microstep: 525.64 | bwd_inner_microstep: 525.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:08:34,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.15 | bwd_microstep: 460.13 | bwd_inner_microstep: 460.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:08:35,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.92 | bwd_microstep: 408.71 | bwd_inner_microstep: 408.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:08:36,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.54 | bwd_microstep: 535.91 | bwd_inner_microstep: 535.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:08:37,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.21 | bwd_microstep: 416.63 | bwd_inner_microstep: 416.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:08:38,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.79 | bwd_microstep: 409.03 | bwd_inner_microstep: 408.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:08:39,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 344.31 | bwd_microstep: 584.38 | bwd_inner_microstep: 584.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:08:41,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.33 | bwd_microstep: 470.15 | bwd_inner_microstep: 469.99 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:08:42,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.56 | bwd_microstep: 560.33 | bwd_inner_microstep: 560.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:08:43,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.48 | bwd_microstep: 449.21 | bwd_inner_microstep: 449.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:08:44,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.56 | bwd_microstep: 430.86 | bwd_inner_microstep: 430.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:08:47,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.48 | optimizer_gradients: 14.22 | optimizer_step: 23.23
[2024-12-05 14:08:47,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.47 | bwd_microstep: 2421.53 | bwd_inner_microstep: 2408.95 | bwd_allreduce_microstep: 12.46 | step_microstep: 113.86
[2024-12-05 14:08:47,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4440.15 | bwd: 9483.30 | bwd_inner: 9469.03 | bwd_allreduce: 13.13 | step: 114.91

 25%|██▌       | 96/384 [25:43<1:34:40, 19.72s/it]
                                                  
{'loss': 0.1394, 'grad_norm': 1.6462229490280151, 'learning_rate': 8.793790613463954e-07, 'epoch': 0.5}

 25%|██▌       | 96/384 [25:43<1:34:40, 19.72s/it][2024-12-05 14:08:48,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.28 | bwd_microstep: 456.21 | bwd_inner_microstep: 456.05 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:08:49,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.23 | bwd_microstep: 404.07 | bwd_inner_microstep: 403.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:08:50,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.42 | bwd_microstep: 611.59 | bwd_inner_microstep: 611.44 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:08:52,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.00 | bwd_microstep: 435.24 | bwd_inner_microstep: 435.09 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:08:53,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.41 | bwd_microstep: 438.30 | bwd_inner_microstep: 438.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:08:54,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.65 | bwd_microstep: 407.46 | bwd_inner_microstep: 407.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:08:54,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.25 | bwd_microstep: 438.89 | bwd_inner_microstep: 438.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:08:56,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.39 | bwd_microstep: 462.50 | bwd_inner_microstep: 462.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:08:57,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.30 | bwd_microstep: 578.75 | bwd_inner_microstep: 578.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:08:58,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.77 | bwd_microstep: 521.34 | bwd_inner_microstep: 521.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:08:59,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.98 | bwd_microstep: 454.84 | bwd_inner_microstep: 454.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:09:00,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.22 | bwd_microstep: 444.98 | bwd_inner_microstep: 444.84 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:09:02,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.04 | bwd_microstep: 512.87 | bwd_inner_microstep: 512.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:09:03,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.38 | bwd_microstep: 421.73 | bwd_inner_microstep: 421.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:09:04,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.70 | bwd_microstep: 428.24 | bwd_inner_microstep: 428.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:09:06,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.64 | optimizer_gradients: 14.21 | optimizer_step: 23.17
[2024-12-05 14:09:06,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.31 | bwd_microstep: 1864.57 | bwd_inner_microstep: 1852.33 | bwd_allreduce_microstep: 12.14 | step_microstep: 112.72
[2024-12-05 14:09:06,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4368.41 | bwd: 8881.56 | bwd_inner: 8867.65 | bwd_allreduce: 12.77 | step: 113.85

 25%|██▌       | 97/384 [26:03<1:33:43, 19.59s/it]
                                                  
{'loss': 0.1581, 'grad_norm': 1.7336857318878174, 'learning_rate': 8.766150989780317e-07, 'epoch': 0.5}

 25%|██▌       | 97/384 [26:03<1:33:43, 19.59s/it][2024-12-05 14:09:08,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.67 | bwd_microstep: 540.39 | bwd_inner_microstep: 540.25 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:09:09,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.06 | bwd_microstep: 407.06 | bwd_inner_microstep: 406.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:09:10,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.32 | bwd_microstep: 542.99 | bwd_inner_microstep: 542.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:09:11,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.68 | bwd_microstep: 497.42 | bwd_inner_microstep: 497.27 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:09:12,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.93 | bwd_microstep: 452.76 | bwd_inner_microstep: 452.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:09:13,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.54 | bwd_microstep: 476.11 | bwd_inner_microstep: 475.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:09:14,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.39 | bwd_microstep: 427.34 | bwd_inner_microstep: 427.20 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:09:16,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.06 | bwd_microstep: 683.35 | bwd_inner_microstep: 683.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:09:17,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.47 | bwd_microstep: 488.62 | bwd_inner_microstep: 488.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:09:18,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.41 | bwd_microstep: 583.33 | bwd_inner_microstep: 583.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:09:19,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.80 | bwd_microstep: 477.60 | bwd_inner_microstep: 477.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:09:20,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.39 | bwd_microstep: 493.74 | bwd_inner_microstep: 493.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:09:22,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.11 | bwd_microstep: 565.95 | bwd_inner_microstep: 565.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:09:23,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.61 | bwd_microstep: 527.41 | bwd_inner_microstep: 527.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:09:24,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 363.86 | bwd_microstep: 744.29 | bwd_inner_microstep: 744.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:09:26,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.63 | optimizer_gradients: 14.10 | optimizer_step: 23.09
[2024-12-05 14:09:26,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.52 | bwd_microstep: 972.89 | bwd_inner_microstep: 960.65 | bwd_allreduce_microstep: 12.15 | step_microstep: 114.55
[2024-12-05 14:09:26,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4680.90 | bwd: 8881.23 | bwd_inner: 8867.36 | bwd_allreduce: 12.77 | step: 115.70

 26%|██▌       | 98/384 [26:22<1:33:20, 19.58s/it]
                                                  
{'loss': 0.1918, 'grad_norm': 1.8384933471679688, 'learning_rate': 8.738242764239046e-07, 'epoch': 0.51}

 26%|██▌       | 98/384 [26:22<1:33:20, 19.58s/it][2024-12-05 14:09:27,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.28 | bwd_microstep: 373.17 | bwd_inner_microstep: 373.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:09:28,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.88 | bwd_microstep: 355.53 | bwd_inner_microstep: 355.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:09:29,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.57 | bwd_microstep: 429.68 | bwd_inner_microstep: 429.50 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 14:09:31,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.77 | bwd_microstep: 433.32 | bwd_inner_microstep: 433.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:09:32,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 381.60 | bwd_microstep: 837.70 | bwd_inner_microstep: 837.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:09:33,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.03 | bwd_microstep: 409.97 | bwd_inner_microstep: 409.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:09:35,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.49 | bwd_microstep: 432.96 | bwd_inner_microstep: 432.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:09:35,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.11 | bwd_microstep: 406.56 | bwd_inner_microstep: 406.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:09:36,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.77 | bwd_microstep: 433.39 | bwd_inner_microstep: 433.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:09:37,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.77 | bwd_microstep: 482.89 | bwd_inner_microstep: 482.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:09:39,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.94 | bwd_microstep: 439.60 | bwd_inner_microstep: 439.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:09:40,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.50 | bwd_microstep: 602.78 | bwd_inner_microstep: 602.64 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:09:41,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.09 | bwd_microstep: 427.54 | bwd_inner_microstep: 427.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:09:42,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.70 | bwd_microstep: 414.72 | bwd_inner_microstep: 414.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:09:43,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.20 | bwd_microstep: 484.40 | bwd_inner_microstep: 484.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:09:47,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.43 | optimizer_gradients: 14.12 | optimizer_step: 23.22
[2024-12-05 14:09:47,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.40 | bwd_microstep: 2431.26 | bwd_inner_microstep: 2419.04 | bwd_allreduce_microstep: 12.11 | step_microstep: 112.50
[2024-12-05 14:09:47,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4354.16 | bwd: 9395.48 | bwd_inner: 9381.55 | bwd_allreduce: 12.76 | step: 113.77

 26%|██▌       | 99/384 [26:43<1:34:28, 19.89s/it]
                                                  
{'loss': 0.1737, 'grad_norm': 1.723486065864563, 'learning_rate': 8.710067927254554e-07, 'epoch': 0.51}

 26%|██▌       | 99/384 [26:43<1:34:28, 19.89s/it][2024-12-05 14:09:48,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.48 | bwd_microstep: 375.95 | bwd_inner_microstep: 375.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:09:50,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.64 | bwd_microstep: 458.35 | bwd_inner_microstep: 458.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:09:51,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.82 | bwd_microstep: 409.89 | bwd_inner_microstep: 409.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:09:52,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.68 | bwd_microstep: 353.62 | bwd_inner_microstep: 353.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:09:53,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.72 | bwd_microstep: 385.81 | bwd_inner_microstep: 385.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:09:55,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.31 | bwd_microstep: 535.39 | bwd_inner_microstep: 535.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:09:56,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.21 | bwd_microstep: 527.06 | bwd_inner_microstep: 526.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:09:57,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.55 | bwd_microstep: 434.78 | bwd_inner_microstep: 434.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:09:58,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.36 | bwd_microstep: 523.49 | bwd_inner_microstep: 523.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:00,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.17 | bwd_microstep: 630.14 | bwd_inner_microstep: 629.99 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 14:10:01,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.25 | bwd_microstep: 413.22 | bwd_inner_microstep: 413.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:10:02,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.09 | bwd_microstep: 562.94 | bwd_inner_microstep: 562.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:03,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.63 | bwd_microstep: 432.55 | bwd_inner_microstep: 432.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:10:04,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.99 | bwd_microstep: 505.91 | bwd_inner_microstep: 505.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:10:05,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.86 | bwd_microstep: 402.75 | bwd_inner_microstep: 402.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:10:08,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.40 | optimizer_gradients: 14.09 | optimizer_step: 23.45
[2024-12-05 14:10:08,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 240.96 | bwd_microstep: 2203.05 | bwd_inner_microstep: 2190.78 | bwd_allreduce_microstep: 12.18 | step_microstep: 112.72
[2024-12-05 14:10:08,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4310.79 | bwd: 9154.88 | bwd_inner: 9140.98 | bwd_allreduce: 12.79 | step: 113.91

 26%|██▌       | 100/384 [27:05<1:36:41, 20.43s/it]
                                                   
{'loss': 0.1728, 'grad_norm': 1.9092669486999512, 'learning_rate': 8.681628488255986e-07, 'epoch': 0.52}

 26%|██▌       | 100/384 [27:05<1:36:41, 20.43s/it][2024-12-05 14:10:10,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 344.83 | bwd_microstep: 647.87 | bwd_inner_microstep: 647.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 14:10:11,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.83 | bwd_microstep: 436.94 | bwd_inner_microstep: 436.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:10:12,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.84 | bwd_microstep: 361.30 | bwd_inner_microstep: 361.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:10:13,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.17 | bwd_microstep: 588.27 | bwd_inner_microstep: 588.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:10:14,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.14 | bwd_microstep: 450.45 | bwd_inner_microstep: 450.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:16,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.59 | bwd_microstep: 444.00 | bwd_inner_microstep: 443.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:17,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.95 | bwd_microstep: 561.14 | bwd_inner_microstep: 560.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:18,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.82 | bwd_microstep: 462.81 | bwd_inner_microstep: 462.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:19,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.68 | bwd_microstep: 430.19 | bwd_inner_microstep: 430.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:20,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.92 | bwd_microstep: 468.80 | bwd_inner_microstep: 468.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:10:21,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.53 | bwd_microstep: 438.78 | bwd_inner_microstep: 438.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:10:22,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.02 | bwd_microstep: 597.24 | bwd_inner_microstep: 597.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:23,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.94 | bwd_microstep: 359.05 | bwd_inner_microstep: 358.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:25,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.28 | bwd_microstep: 565.03 | bwd_inner_microstep: 564.89 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 14:10:26,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.87 | bwd_microstep: 411.47 | bwd_inner_microstep: 411.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:10:27,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.17 | optimizer_gradients: 14.11 | optimizer_step: 23.26
[2024-12-05 14:10:27,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.93 | bwd_microstep: 1002.50 | bwd_inner_microstep: 990.18 | bwd_allreduce_microstep: 12.20 | step_microstep: 112.25
[2024-12-05 14:10:27,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4394.42 | bwd: 8225.86 | bwd_inner: 8211.95 | bwd_allreduce: 12.85 | step: 113.59

 26%|██▋       | 101/384 [27:24<1:34:34, 20.05s/it]
                                                   
{'loss': 0.162, 'grad_norm': 1.729762315750122, 'learning_rate': 8.652926475543898e-07, 'epoch': 0.53}

 26%|██▋       | 101/384 [27:24<1:34:34, 20.05s/it][2024-12-05 14:10:28,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.11 | bwd_microstep: 355.78 | bwd_inner_microstep: 355.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:10:29,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.70 | bwd_microstep: 542.99 | bwd_inner_microstep: 542.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:30,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.46 | bwd_microstep: 409.59 | bwd_inner_microstep: 409.42 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 14:10:32,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.65 | bwd_microstep: 493.46 | bwd_inner_microstep: 493.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:33,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.76 | bwd_microstep: 421.67 | bwd_inner_microstep: 421.52 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:10:33,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.37 | bwd_microstep: 406.24 | bwd_inner_microstep: 406.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:10:35,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.24 | bwd_microstep: 455.28 | bwd_inner_microstep: 455.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:36,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.43 | bwd_microstep: 443.92 | bwd_inner_microstep: 443.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:10:37,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.94 | bwd_microstep: 433.55 | bwd_inner_microstep: 433.41 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:10:37,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.81 | bwd_microstep: 409.26 | bwd_inner_microstep: 409.10 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 14:10:38,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.02 | bwd_microstep: 411.37 | bwd_inner_microstep: 411.20 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:10:39,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.86 | bwd_microstep: 358.67 | bwd_inner_microstep: 358.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:40,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.99 | bwd_microstep: 452.47 | bwd_inner_microstep: 452.34 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:10:41,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 227.49 | bwd_microstep: 340.84 | bwd_inner_microstep: 340.68 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 14:10:42,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.82 | bwd_microstep: 443.19 | bwd_inner_microstep: 443.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:46,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.03 | optimizer_gradients: 14.23 | optimizer_step: 23.23
[2024-12-05 14:10:46,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.46 | bwd_microstep: 3879.34 | bwd_inner_microstep: 3866.75 | bwd_allreduce_microstep: 12.47 | step_microstep: 113.19
[2024-12-05 14:10:46,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4182.20 | bwd: 10257.64 | bwd_inner: 10243.24 | bwd_allreduce: 13.19 | step: 114.53

 27%|██▋       | 102/384 [27:42<1:32:27, 19.67s/it]
                                                   
{'loss': 0.1658, 'grad_norm': 1.7867392301559448, 'learning_rate': 8.623963936145599e-07, 'epoch': 0.53}

 27%|██▋       | 102/384 [27:42<1:32:27, 19.67s/it][2024-12-05 14:10:48,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.86 | bwd_microstep: 517.89 | bwd_inner_microstep: 517.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:49,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.31 | bwd_microstep: 418.57 | bwd_inner_microstep: 418.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:49,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 225.39 | bwd_microstep: 338.80 | bwd_inner_microstep: 338.66 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:10:51,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.88 | bwd_microstep: 459.86 | bwd_inner_microstep: 459.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:52,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.51 | bwd_microstep: 559.22 | bwd_inner_microstep: 559.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:53,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.63 | bwd_microstep: 438.89 | bwd_inner_microstep: 438.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:10:54,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.03 | bwd_microstep: 423.29 | bwd_inner_microstep: 423.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:55,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.06 | bwd_microstep: 439.91 | bwd_inner_microstep: 439.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:10:56,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.76 | bwd_microstep: 475.12 | bwd_inner_microstep: 474.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:10:57,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 382.33 | bwd_microstep: 851.32 | bwd_inner_microstep: 851.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:10:58,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.42 | bwd_microstep: 459.96 | bwd_inner_microstep: 459.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:10:59,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.26 | bwd_microstep: 601.96 | bwd_inner_microstep: 601.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:11:01,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.93 | bwd_microstep: 415.15 | bwd_inner_microstep: 415.01 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:11:02,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.83 | bwd_microstep: 397.86 | bwd_inner_microstep: 397.72 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 14:11:03,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.03 | bwd_microstep: 541.88 | bwd_inner_microstep: 541.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:11:05,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.13 | optimizer_gradients: 14.06 | optimizer_step: 23.08
[2024-12-05 14:11:05,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.55 | bwd_microstep: 1728.05 | bwd_inner_microstep: 1715.80 | bwd_allreduce_microstep: 12.14 | step_microstep: 111.66
[2024-12-05 14:11:05,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4439.85 | bwd: 9067.74 | bwd_inner: 9053.80 | bwd_allreduce: 12.83 | step: 112.94

 27%|██▋       | 103/384 [28:01<1:30:47, 19.39s/it]
                                                   
{'loss': 0.1865, 'grad_norm': 2.0217678546905518, 'learning_rate': 8.594742935669164e-07, 'epoch': 0.54}

 27%|██▋       | 103/384 [28:01<1:30:47, 19.39s/it][2024-12-05 14:11:06,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.55 | bwd_microstep: 388.19 | bwd_inner_microstep: 388.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:11:07,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.01 | bwd_microstep: 447.01 | bwd_inner_microstep: 446.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:11:08,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.58 | bwd_microstep: 462.87 | bwd_inner_microstep: 462.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:11:09,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.80 | bwd_microstep: 432.86 | bwd_inner_microstep: 432.67 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.11
[2024-12-05 14:11:10,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 328.03 | bwd_microstep: 622.53 | bwd_inner_microstep: 622.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:11:11,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.89 | bwd_microstep: 455.38 | bwd_inner_microstep: 455.23 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:11:12,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.69 | bwd_microstep: 441.73 | bwd_inner_microstep: 441.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:11:13,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.72 | bwd_microstep: 414.32 | bwd_inner_microstep: 414.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:11:14,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.15 | bwd_microstep: 452.12 | bwd_inner_microstep: 452.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:11:16,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.29 | bwd_microstep: 549.69 | bwd_inner_microstep: 549.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:11:17,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.58 | bwd_microstep: 442.66 | bwd_inner_microstep: 442.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:11:18,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.89 | bwd_microstep: 415.26 | bwd_inner_microstep: 415.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:11:19,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.27 | bwd_microstep: 431.54 | bwd_inner_microstep: 431.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:11:20,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.52 | bwd_microstep: 630.21 | bwd_inner_microstep: 630.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:11:21,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.88 | bwd_microstep: 407.59 | bwd_inner_microstep: 407.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:11:24,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.58 | optimizer_gradients: 14.34 | optimizer_step: 23.17
[2024-12-05 14:11:24,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.60 | bwd_microstep: 2296.30 | bwd_inner_microstep: 2284.27 | bwd_allreduce_microstep: 11.95 | step_microstep: 113.47
[2024-12-05 14:11:24,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4397.51 | bwd: 9290.28 | bwd_inner: 9276.55 | bwd_allreduce: 12.60 | step: 114.74

 27%|██▋       | 104/384 [28:21<1:30:26, 19.38s/it]
                                                   
{'loss': 0.1535, 'grad_norm': 1.7473349571228027, 'learning_rate': 8.565265558156101e-07, 'epoch': 0.54}

 27%|██▋       | 104/384 [28:21<1:30:26, 19.38s/it][2024-12-05 14:11:25,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.13 | bwd_microstep: 415.25 | bwd_inner_microstep: 415.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:11:27,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.84 | bwd_microstep: 565.96 | bwd_inner_microstep: 565.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:11:28,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.80 | bwd_microstep: 418.60 | bwd_inner_microstep: 418.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:11:29,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.66 | bwd_microstep: 459.92 | bwd_inner_microstep: 459.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:11:30,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.96 | bwd_microstep: 486.64 | bwd_inner_microstep: 486.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:11:31,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.05 | bwd_microstep: 490.30 | bwd_inner_microstep: 490.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:11:32,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.60 | bwd_microstep: 628.90 | bwd_inner_microstep: 628.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:11:33,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.31 | bwd_microstep: 434.68 | bwd_inner_microstep: 434.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:11:34,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.99 | bwd_microstep: 436.01 | bwd_inner_microstep: 435.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:11:35,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.03 | bwd_microstep: 531.87 | bwd_inner_microstep: 531.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:11:36,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.70 | bwd_microstep: 435.69 | bwd_inner_microstep: 435.54 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:11:37,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.45 | bwd_microstep: 482.26 | bwd_inner_microstep: 482.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:11:38,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.42 | bwd_microstep: 438.11 | bwd_inner_microstep: 437.96 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:11:40,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.43 | bwd_microstep: 703.43 | bwd_inner_microstep: 703.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:11:41,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.00 | bwd_microstep: 462.72 | bwd_inner_microstep: 462.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:11:44,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.27 | optimizer_gradients: 14.05 | optimizer_step: 23.07
[2024-12-05 14:11:44,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.77 | bwd_microstep: 2992.44 | bwd_inner_microstep: 2980.23 | bwd_allreduce_microstep: 12.11 | step_microstep: 114.59
[2024-12-05 14:11:44,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4465.22 | bwd: 10382.76 | bwd_inner: 10368.90 | bwd_allreduce: 12.77 | step: 115.67

 27%|██▋       | 105/384 [28:40<1:30:32, 19.47s/it]
                                                   
{'loss': 0.1559, 'grad_norm': 1.5292036533355713, 'learning_rate': 8.535533905932737e-07, 'epoch': 0.55}

 27%|██▋       | 105/384 [28:40<1:30:32, 19.47s/it][2024-12-05 14:11:45,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.39 | bwd_microstep: 405.92 | bwd_inner_microstep: 405.71 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11
[2024-12-05 14:11:46,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.55 | bwd_microstep: 438.42 | bwd_inner_microstep: 438.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:11:47,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.87 | bwd_microstep: 568.56 | bwd_inner_microstep: 568.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:11:48,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.78 | bwd_microstep: 356.75 | bwd_inner_microstep: 356.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:11:50,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.00 | bwd_microstep: 564.31 | bwd_inner_microstep: 564.10 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.12
[2024-12-05 14:11:51,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.36 | bwd_microstep: 488.16 | bwd_inner_microstep: 488.01 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:11:52,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.02 | bwd_microstep: 432.62 | bwd_inner_microstep: 432.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:11:53,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.61 | bwd_microstep: 361.46 | bwd_inner_microstep: 361.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:11:54,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.48 | bwd_microstep: 525.04 | bwd_inner_microstep: 524.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:11:55,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.09 | bwd_microstep: 626.99 | bwd_inner_microstep: 626.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:11:56,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 400.99 | bwd_microstep: 898.91 | bwd_inner_microstep: 898.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:11:58,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.77 | bwd_microstep: 437.13 | bwd_inner_microstep: 437.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:11:59,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.76 | bwd_microstep: 410.09 | bwd_inner_microstep: 409.91 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09
[2024-12-05 14:12:00,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.33 | bwd_microstep: 458.13 | bwd_inner_microstep: 457.99 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:12:01,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.44 | bwd_microstep: 440.63 | bwd_inner_microstep: 440.44 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 14:12:03,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.36 | optimizer_gradients: 14.06 | optimizer_step: 23.10
[2024-12-05 14:12:03,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.55 | bwd_microstep: 1668.79 | bwd_inner_microstep: 1656.66 | bwd_allreduce_microstep: 12.05 | step_microstep: 111.99
[2024-12-05 14:12:03,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4579.05 | bwd: 9081.91 | bwd_inner: 9067.93 | bwd_allreduce: 12.82 | step: 113.35

 28%|██▊       | 106/384 [28:59<1:29:45, 19.37s/it]
                                                   
{'loss': 0.1423, 'grad_norm': 1.5472049713134766, 'learning_rate': 8.505550099460263e-07, 'epoch': 0.55}

 28%|██▊       | 106/384 [28:59<1:29:45, 19.37s/it][2024-12-05 14:12:04,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.52 | bwd_microstep: 404.99 | bwd_inner_microstep: 404.82 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:12:05,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.05 | bwd_microstep: 434.93 | bwd_inner_microstep: 434.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:12:06,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.65 | bwd_microstep: 483.33 | bwd_inner_microstep: 483.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:12:07,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.46 | bwd_microstep: 483.16 | bwd_inner_microstep: 483.02 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:12:09,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 348.07 | bwd_microstep: 711.87 | bwd_inner_microstep: 711.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:12:10,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.12 | bwd_microstep: 477.86 | bwd_inner_microstep: 477.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:12:10,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.01 | bwd_microstep: 444.59 | bwd_inner_microstep: 444.46 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:12:11,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.62 | bwd_microstep: 614.19 | bwd_inner_microstep: 614.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:12:13,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.58 | bwd_microstep: 680.37 | bwd_inner_microstep: 680.17 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10
[2024-12-05 14:12:13,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.33 | bwd_microstep: 412.92 | bwd_inner_microstep: 412.79 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:12:14,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.92 | bwd_microstep: 442.43 | bwd_inner_microstep: 442.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:12:16,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.75 | bwd_microstep: 439.16 | bwd_inner_microstep: 439.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:12:16,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.11 | bwd_microstep: 353.04 | bwd_inner_microstep: 352.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:12:17,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.38 | bwd_microstep: 492.24 | bwd_inner_microstep: 492.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:12:19,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.57 | bwd_microstep: 594.67 | bwd_inner_microstep: 594.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:12:22,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.10 | optimizer_gradients: 14.08 | optimizer_step: 23.29
[2024-12-05 14:12:22,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.23 | bwd_microstep: 2624.75 | bwd_inner_microstep: 2612.59 | bwd_allreduce_microstep: 12.06 | step_microstep: 112.02
[2024-12-05 14:12:22,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4513.41 | bwd: 10094.52 | bwd_inner: 10080.62 | bwd_allreduce: 12.79 | step: 113.26

 28%|██▊       | 107/384 [29:18<1:28:41, 19.21s/it]
                                                   
{'loss': 0.1747, 'grad_norm': 1.636443853378296, 'learning_rate': 8.475316277183508e-07, 'epoch': 0.56}

 28%|██▊       | 107/384 [29:18<1:28:41, 19.21s/it][2024-12-05 14:12:23,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.05 | bwd_microstep: 419.64 | bwd_inner_microstep: 419.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:12:24,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.87 | bwd_microstep: 428.71 | bwd_inner_microstep: 428.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:12:25,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.65 | bwd_microstep: 449.18 | bwd_inner_microstep: 449.00 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.11
[2024-12-05 14:12:26,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.44 | bwd_microstep: 413.45 | bwd_inner_microstep: 413.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:12:27,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.76 | bwd_microstep: 349.67 | bwd_inner_microstep: 349.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:12:28,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.14 | bwd_microstep: 407.96 | bwd_inner_microstep: 407.83 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:12:29,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.97 | bwd_microstep: 431.75 | bwd_inner_microstep: 431.61 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:12:29,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.40 | bwd_microstep: 424.25 | bwd_inner_microstep: 424.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:12:30,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.92 | bwd_microstep: 456.79 | bwd_inner_microstep: 456.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:12:31,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.76 | bwd_microstep: 562.13 | bwd_inner_microstep: 561.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:12:32,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.76 | bwd_microstep: 414.40 | bwd_inner_microstep: 414.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:12:33,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.71 | bwd_microstep: 400.11 | bwd_inner_microstep: 399.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:12:34,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.31 | bwd_microstep: 430.72 | bwd_inner_microstep: 430.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:12:35,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.69 | bwd_microstep: 608.51 | bwd_inner_microstep: 608.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:12:36,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.51 | bwd_microstep: 448.26 | bwd_inner_microstep: 448.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:12:42,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.17 | optimizer_gradients: 14.07 | optimizer_step: 23.18
[2024-12-05 14:12:42,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 370.62 | bwd_microstep: 5084.52 | bwd_inner_microstep: 5072.33 | bwd_allreduce_microstep: 12.10 | step_microstep: 112.13
[2024-12-05 14:12:42,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4417.62 | bwd: 11730.03 | bwd_inner: 11716.09 | bwd_allreduce: 12.76 | step: 113.43

 28%|██▊       | 108/384 [29:38<1:28:53, 19.32s/it]
                                                   
{'loss': 0.1766, 'grad_norm': 1.5992189645767212, 'learning_rate': 8.444834595378433e-07, 'epoch': 0.56}

 28%|██▊       | 108/384 [29:38<1:28:53, 19.32s/it][2024-12-05 14:12:43,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.01 | bwd_microstep: 416.88 | bwd_inner_microstep: 416.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:12:43,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.87 | bwd_microstep: 434.01 | bwd_inner_microstep: 433.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:12:45,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 359.00 | bwd_microstep: 750.19 | bwd_inner_microstep: 750.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:12:46,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.95 | bwd_microstep: 458.89 | bwd_inner_microstep: 458.74 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:12:47,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 238.12 | bwd_microstep: 359.24 | bwd_inner_microstep: 359.06 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 14:12:48,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.56 | bwd_microstep: 594.23 | bwd_inner_microstep: 594.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:12:49,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.69 | bwd_microstep: 402.00 | bwd_inner_microstep: 401.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:12:50,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.36 | bwd_microstep: 418.98 | bwd_inner_microstep: 418.82 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:12:51,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.01 | bwd_microstep: 590.73 | bwd_inner_microstep: 590.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:12:53,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.97 | bwd_microstep: 684.65 | bwd_inner_microstep: 684.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:12:54,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.21 | bwd_microstep: 473.97 | bwd_inner_microstep: 473.84 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:12:55,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.73 | bwd_microstep: 432.45 | bwd_inner_microstep: 432.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:12:56,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.46 | bwd_microstep: 534.15 | bwd_inner_microstep: 534.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:12:58,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.16 | bwd_microstep: 479.32 | bwd_inner_microstep: 479.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:12:58,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.13 | bwd_microstep: 428.06 | bwd_inner_microstep: 427.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:13:01,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.83 | optimizer_gradients: 14.04 | optimizer_step: 23.13
[2024-12-05 14:13:01,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 342.66 | bwd_microstep: 2349.99 | bwd_inner_microstep: 2337.85 | bwd_allreduce_microstep: 12.06 | step_microstep: 113.51
[2024-12-05 14:13:01,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4618.98 | bwd: 9807.74 | bwd_inner: 9793.89 | bwd_allreduce: 12.72 | step: 114.61

 28%|██▊       | 109/384 [29:58<1:29:11, 19.46s/it]
                                                   
{'loss': 0.1421, 'grad_norm': 1.6766347885131836, 'learning_rate': 8.414107227998328e-07, 'epoch': 0.57}

 28%|██▊       | 109/384 [29:58<1:29:11, 19.46s/it][2024-12-05 14:13:03,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.80 | bwd_microstep: 651.11 | bwd_inner_microstep: 650.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:13:05,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 401.96 | bwd_microstep: 905.50 | bwd_inner_microstep: 905.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:13:06,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.62 | bwd_microstep: 596.07 | bwd_inner_microstep: 595.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:13:07,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.24 | bwd_microstep: 429.78 | bwd_inner_microstep: 429.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:13:08,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 357.29 | bwd_microstep: 739.25 | bwd_inner_microstep: 739.11 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:13:09,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 229.75 | bwd_microstep: 332.00 | bwd_inner_microstep: 331.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:13:10,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.75 | bwd_microstep: 438.17 | bwd_inner_microstep: 438.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:13:11,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.94 | bwd_microstep: 455.85 | bwd_inner_microstep: 455.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:13:12,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.14 | bwd_microstep: 484.64 | bwd_inner_microstep: 484.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:13:13,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.83 | bwd_microstep: 409.03 | bwd_inner_microstep: 408.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:13:14,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.94 | bwd_microstep: 592.72 | bwd_inner_microstep: 592.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:13:16,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.48 | bwd_microstep: 377.85 | bwd_inner_microstep: 377.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:13:17,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.48 | bwd_microstep: 535.82 | bwd_inner_microstep: 535.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:13:18,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.40 | bwd_microstep: 525.72 | bwd_inner_microstep: 525.60 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:13:19,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.82 | bwd_microstep: 410.92 | bwd_inner_microstep: 410.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:13:21,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.61 | optimizer_gradients: 14.08 | optimizer_step: 23.12
[2024-12-05 14:13:21,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.78 | bwd_microstep: 988.50 | bwd_inner_microstep: 976.27 | bwd_allreduce_microstep: 12.13 | step_microstep: 112.49
[2024-12-05 14:13:21,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4665.32 | bwd: 8872.92 | bwd_inner: 8859.01 | bwd_allreduce: 12.76 | step: 113.61

 29%|██▊       | 110/384 [30:17<1:29:18, 19.55s/it]
                                                   
{'loss': 0.1752, 'grad_norm': 1.7345399856567383, 'learning_rate': 8.383136366518787e-07, 'epoch': 0.57}

 29%|██▊       | 110/384 [30:17<1:29:18, 19.55s/it][2024-12-05 14:13:22,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.39 | bwd_microstep: 452.68 | bwd_inner_microstep: 452.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:13:24,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 379.13 | bwd_microstep: 840.79 | bwd_inner_microstep: 840.66 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:13:25,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.75 | bwd_microstep: 444.98 | bwd_inner_microstep: 444.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:13:26,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.09 | bwd_microstep: 457.23 | bwd_inner_microstep: 457.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:13:27,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.59 | bwd_microstep: 456.33 | bwd_inner_microstep: 456.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:13:28,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.36 | bwd_microstep: 546.49 | bwd_inner_microstep: 546.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:13:29,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.29 | bwd_microstep: 426.60 | bwd_inner_microstep: 426.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:13:30,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.34 | bwd_microstep: 459.16 | bwd_inner_microstep: 459.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:13:31,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.72 | bwd_microstep: 441.30 | bwd_inner_microstep: 441.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:13:32,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.28 | bwd_microstep: 360.16 | bwd_inner_microstep: 360.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:13:33,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.49 | bwd_microstep: 403.88 | bwd_inner_microstep: 403.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:13:34,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.68 | bwd_microstep: 570.84 | bwd_inner_microstep: 570.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:13:35,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.64 | bwd_microstep: 584.32 | bwd_inner_microstep: 584.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:13:36,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.99 | bwd_microstep: 484.65 | bwd_inner_microstep: 484.49 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:13:37,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.63 | bwd_microstep: 451.04 | bwd_inner_microstep: 450.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:13:40,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.50 | optimizer_gradients: 14.05 | optimizer_step: 23.23
[2024-12-05 14:13:40,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 238.74 | bwd_microstep: 1795.24 | bwd_inner_microstep: 1783.07 | bwd_allreduce_microstep: 12.07 | step_microstep: 112.46
[2024-12-05 14:13:40,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4402.16 | bwd: 9175.68 | bwd_inner: 9161.88 | bwd_allreduce: 12.71 | step: 113.72

 29%|██▉       | 111/384 [30:36<1:27:23, 19.21s/it]
                                                   
{'loss': 0.1661, 'grad_norm': 1.669264793395996, 'learning_rate': 8.351924219781392e-07, 'epoch': 0.58}

 29%|██▉       | 111/384 [30:36<1:27:23, 19.21s/it][2024-12-05 14:13:41,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.87 | bwd_microstep: 506.60 | bwd_inner_microstep: 506.47 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:13:42,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.81 | bwd_microstep: 623.50 | bwd_inner_microstep: 623.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:13:43,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.14 | bwd_microstep: 430.63 | bwd_inner_microstep: 430.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:13:44,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.09 | bwd_microstep: 449.75 | bwd_inner_microstep: 449.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:13:45,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.89 | bwd_microstep: 480.71 | bwd_inner_microstep: 480.58 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:13:47,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.07 | bwd_microstep: 572.16 | bwd_inner_microstep: 572.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:13:48,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.19 | bwd_microstep: 359.72 | bwd_inner_microstep: 359.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:13:49,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.27 | bwd_microstep: 427.09 | bwd_inner_microstep: 426.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:13:50,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.51 | bwd_microstep: 480.26 | bwd_inner_microstep: 480.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:13:51,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.50 | bwd_microstep: 413.33 | bwd_inner_microstep: 413.14 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 14:13:52,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.75 | bwd_microstep: 400.37 | bwd_inner_microstep: 400.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:13:53,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.71 | bwd_microstep: 449.26 | bwd_inner_microstep: 449.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:13:54,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 362.94 | bwd_microstep: 748.49 | bwd_inner_microstep: 748.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:13:55,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.16 | bwd_microstep: 492.56 | bwd_inner_microstep: 492.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:13:56,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.47 | bwd_microstep: 416.73 | bwd_inner_microstep: 416.59 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:13:58,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.81 | optimizer_gradients: 14.02 | optimizer_step: 23.31
[2024-12-05 14:13:58,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.53 | bwd_microstep: 1505.81 | bwd_inner_microstep: 1493.65 | bwd_allreduce_microstep: 12.06 | step_microstep: 113.99
[2024-12-05 14:13:58,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4414.96 | bwd: 8756.95 | bwd_inner: 8743.08 | bwd_allreduce: 12.73 | step: 115.16

 29%|██▉       | 112/384 [30:55<1:26:41, 19.12s/it]
                                                   
{'loss': 0.1892, 'grad_norm': 1.7636350393295288, 'learning_rate': 8.320473013836195e-07, 'epoch': 0.58}

 29%|██▉       | 112/384 [30:55<1:26:41, 19.12s/it][2024-12-05 14:13:59,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.64 | bwd_microstep: 504.47 | bwd_inner_microstep: 504.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:14:00,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.14 | bwd_microstep: 431.74 | bwd_inner_microstep: 431.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:14:01,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.99 | bwd_microstep: 409.79 | bwd_inner_microstep: 409.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:14:03,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.22 | bwd_microstep: 584.92 | bwd_inner_microstep: 584.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:14:04,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.99 | bwd_microstep: 490.85 | bwd_inner_microstep: 490.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:14:05,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.22 | bwd_microstep: 414.73 | bwd_inner_microstep: 414.60 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:14:06,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.86 | bwd_microstep: 434.17 | bwd_inner_microstep: 434.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:14:07,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.79 | bwd_microstep: 454.47 | bwd_inner_microstep: 454.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:14:08,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 353.81 | bwd_microstep: 704.33 | bwd_inner_microstep: 704.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:14:09,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.74 | bwd_microstep: 485.47 | bwd_inner_microstep: 485.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:14:10,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.75 | bwd_microstep: 487.64 | bwd_inner_microstep: 487.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:14:11,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.73 | bwd_microstep: 430.37 | bwd_inner_microstep: 430.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:14:12,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.57 | bwd_microstep: 450.22 | bwd_inner_microstep: 450.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:14:13,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.10 | bwd_microstep: 417.70 | bwd_inner_microstep: 417.55 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:14:14,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.95 | bwd_microstep: 548.88 | bwd_inner_microstep: 548.71 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 14:14:17,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.91 | optimizer_gradients: 14.16 | optimizer_step: 23.60
[2024-12-05 14:14:17,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.57 | bwd_microstep: 1689.31 | bwd_inner_microstep: 1677.06 | bwd_allreduce_microstep: 12.14 | step_microstep: 114.17
[2024-12-05 14:14:17,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4472.13 | bwd: 8939.06 | bwd_inner: 8925.13 | bwd_allreduce: 12.81 | step: 115.38

 29%|██▉       | 113/384 [31:13<1:25:20, 18.90s/it]
                                                   
{'loss': 0.1883, 'grad_norm': 1.8429635763168335, 'learning_rate': 8.288784991782945e-07, 'epoch': 0.59}

 29%|██▉       | 113/384 [31:13<1:25:20, 18.90s/it][2024-12-05 14:14:18,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.29 | bwd_microstep: 372.99 | bwd_inner_microstep: 372.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:14:19,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.09 | bwd_microstep: 350.53 | bwd_inner_microstep: 350.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:14:20,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.31 | bwd_microstep: 449.26 | bwd_inner_microstep: 449.08 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:14:21,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 326.79 | bwd_microstep: 593.73 | bwd_inner_microstep: 593.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:14:22,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.86 | bwd_microstep: 434.40 | bwd_inner_microstep: 434.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:14:23,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.62 | bwd_microstep: 448.80 | bwd_inner_microstep: 448.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:14:24,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.94 | bwd_microstep: 439.92 | bwd_inner_microstep: 439.77 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:14:25,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.70 | bwd_microstep: 413.97 | bwd_inner_microstep: 413.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:14:26,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.55 | bwd_microstep: 542.74 | bwd_inner_microstep: 542.59 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:14:27,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.26 | bwd_microstep: 447.55 | bwd_inner_microstep: 447.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:14:29,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.95 | bwd_microstep: 621.33 | bwd_inner_microstep: 621.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:14:29,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.91 | bwd_microstep: 360.38 | bwd_inner_microstep: 360.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:14:31,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.63 | bwd_microstep: 460.07 | bwd_inner_microstep: 459.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:14:32,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.40 | bwd_microstep: 561.24 | bwd_inner_microstep: 561.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:14:33,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.46 | bwd_microstep: 418.05 | bwd_inner_microstep: 417.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:14:36,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.52 | optimizer_gradients: 14.05 | optimizer_step: 23.06
[2024-12-05 14:14:36,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.02 | bwd_microstep: 2012.70 | bwd_inner_microstep: 2000.57 | bwd_allreduce_microstep: 12.03 | step_microstep: 112.14
[2024-12-05 14:14:36,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4397.83 | bwd: 8927.68 | bwd_inner: 8913.78 | bwd_allreduce: 12.66 | step: 113.22

 30%|██▉       | 114/384 [31:32<1:25:03, 18.90s/it]
                                                   
{'loss': 0.1472, 'grad_norm': 1.5329152345657349, 'learning_rate': 8.256862413611112e-07, 'epoch': 0.59}

 30%|██▉       | 114/384 [31:32<1:25:03, 18.90s/it][2024-12-05 14:14:37,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.90 | bwd_microstep: 430.07 | bwd_inner_microstep: 429.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:14:38,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 388.08 | bwd_microstep: 851.56 | bwd_inner_microstep: 851.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:14:40,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.81 | bwd_microstep: 540.68 | bwd_inner_microstep: 540.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:14:41,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.29 | bwd_microstep: 584.93 | bwd_inner_microstep: 584.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:14:41,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.39 | bwd_microstep: 439.57 | bwd_inner_microstep: 439.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:14:42,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.85 | bwd_microstep: 413.84 | bwd_inner_microstep: 413.71 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:14:43,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.85 | bwd_microstep: 451.51 | bwd_inner_microstep: 451.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:14:44,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.57 | bwd_microstep: 398.86 | bwd_inner_microstep: 398.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:14:45,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.59 | bwd_microstep: 441.11 | bwd_inner_microstep: 440.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:14:47,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.90 | bwd_microstep: 628.64 | bwd_inner_microstep: 628.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:14:47,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.68 | bwd_microstep: 441.09 | bwd_inner_microstep: 440.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:14:49,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 402.29 | bwd_microstep: 877.42 | bwd_inner_microstep: 877.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:14:50,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.97 | bwd_microstep: 380.33 | bwd_inner_microstep: 380.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:14:51,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.11 | bwd_microstep: 430.25 | bwd_inner_microstep: 430.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:14:52,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.22 | bwd_microstep: 449.42 | bwd_inner_microstep: 449.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:14:54,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.27 | optimizer_gradients: 14.32 | optimizer_step: 23.37
[2024-12-05 14:14:54,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.01 | bwd_microstep: 1188.51 | bwd_inner_microstep: 1175.94 | bwd_allreduce_microstep: 12.45 | step_microstep: 114.06
[2024-12-05 14:14:54,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4694.53 | bwd: 8947.80 | bwd_inner: 8933.56 | bwd_allreduce: 13.10 | step: 115.25

 30%|██▉       | 115/384 [31:50<1:24:10, 18.78s/it]
                                                   
{'loss': 0.1714, 'grad_norm': 1.9208284616470337, 'learning_rate': 8.22470755603871e-07, 'epoch': 0.6}

 30%|██▉       | 115/384 [31:50<1:24:10, 18.78s/it][2024-12-05 14:14:55,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.20 | bwd_microstep: 420.66 | bwd_inner_microstep: 420.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:14:56,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.77 | bwd_microstep: 448.94 | bwd_inner_microstep: 448.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:14:58,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.62 | bwd_microstep: 416.04 | bwd_inner_microstep: 415.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:14:59,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.61 | bwd_microstep: 430.27 | bwd_inner_microstep: 430.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:15:00,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.03 | bwd_microstep: 358.49 | bwd_inner_microstep: 358.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:15:01,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.02 | bwd_microstep: 525.35 | bwd_inner_microstep: 525.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:15:02,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.54 | bwd_microstep: 623.25 | bwd_inner_microstep: 623.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:15:03,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.10 | bwd_microstep: 687.23 | bwd_inner_microstep: 687.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:15:05,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.75 | bwd_microstep: 488.13 | bwd_inner_microstep: 488.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:15:06,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.46 | bwd_microstep: 435.41 | bwd_inner_microstep: 435.27 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:15:07,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.72 | bwd_microstep: 523.70 | bwd_inner_microstep: 523.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:15:08,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.39 | bwd_microstep: 545.83 | bwd_inner_microstep: 545.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:15:09,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.86 | bwd_microstep: 412.16 | bwd_inner_microstep: 412.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:15:11,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 367.42 | bwd_microstep: 782.43 | bwd_inner_microstep: 782.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:15:12,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.56 | bwd_microstep: 337.93 | bwd_inner_microstep: 337.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:15:14,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.10 | optimizer_gradients: 14.02 | optimizer_step: 23.30
[2024-12-05 14:15:14,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.03 | bwd_microstep: 1080.56 | bwd_inner_microstep: 1068.61 | bwd_allreduce_microstep: 11.88 | step_microstep: 112.04
[2024-12-05 14:15:14,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4490.14 | bwd: 8516.37 | bwd_inner: 8502.78 | bwd_allreduce: 12.52 | step: 113.25

 30%|███       | 116/384 [32:10<1:25:00, 19.03s/it]
                                                   
{'loss': 0.1806, 'grad_norm': 1.6381752490997314, 'learning_rate': 8.192322712349917e-07, 'epoch': 0.6}

 30%|███       | 116/384 [32:10<1:25:00, 19.03s/it][2024-12-05 14:15:15,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.28 | bwd_microstep: 373.27 | bwd_inner_microstep: 373.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:15:16,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.27 | bwd_microstep: 585.12 | bwd_inner_microstep: 585.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:15:16,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.07 | bwd_microstep: 428.60 | bwd_inner_microstep: 428.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:15:18,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 332.93 | bwd_microstep: 676.32 | bwd_inner_microstep: 676.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:15:19,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.45 | bwd_microstep: 358.87 | bwd_inner_microstep: 358.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:15:20,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.04 | bwd_microstep: 568.18 | bwd_inner_microstep: 568.06 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:15:21,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.47 | bwd_microstep: 452.74 | bwd_inner_microstep: 452.61 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:15:22,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.16 | bwd_microstep: 500.84 | bwd_inner_microstep: 500.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:15:23,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.85 | bwd_microstep: 547.58 | bwd_inner_microstep: 547.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:15:24,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.41 | bwd_microstep: 340.12 | bwd_inner_microstep: 339.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:15:25,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.87 | bwd_microstep: 489.51 | bwd_inner_microstep: 489.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:15:26,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.05 | bwd_microstep: 446.11 | bwd_inner_microstep: 445.89 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10
[2024-12-05 14:15:27,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.98 | bwd_microstep: 412.59 | bwd_inner_microstep: 412.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:15:28,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.10 | bwd_microstep: 360.90 | bwd_inner_microstep: 360.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:15:29,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.17 | bwd_microstep: 458.83 | bwd_inner_microstep: 458.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:15:34,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.40 | optimizer_gradients: 13.99 | optimizer_step: 23.18
[2024-12-05 14:15:34,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.86 | bwd_microstep: 4136.54 | bwd_inner_microstep: 4124.55 | bwd_allreduce_microstep: 11.89 | step_microstep: 111.91
[2024-12-05 14:15:34,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4371.04 | bwd: 11136.10 | bwd_inner: 11122.44 | bwd_allreduce: 12.55 | step: 112.99

 30%|███       | 117/384 [32:30<1:25:41, 19.26s/it]
                                                   
{'loss': 0.1653, 'grad_norm': 1.5502922534942627, 'learning_rate': 8.159710192231519e-07, 'epoch': 0.61}

 30%|███       | 117/384 [32:30<1:25:41, 19.26s/it][2024-12-05 14:15:34,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.88 | bwd_microstep: 374.39 | bwd_inner_microstep: 374.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:15:36,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.56 | bwd_microstep: 587.82 | bwd_inner_microstep: 587.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:15:37,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.53 | bwd_microstep: 561.02 | bwd_inner_microstep: 560.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:15:38,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.04 | bwd_microstep: 562.19 | bwd_inner_microstep: 562.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:15:39,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 345.62 | bwd_microstep: 450.41 | bwd_inner_microstep: 450.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:15:40,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.99 | bwd_microstep: 580.94 | bwd_inner_microstep: 580.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:15:41,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.21 | bwd_microstep: 403.37 | bwd_inner_microstep: 403.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:15:42,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.49 | bwd_microstep: 436.65 | bwd_inner_microstep: 436.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:15:43,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.15 | bwd_microstep: 480.90 | bwd_inner_microstep: 480.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:15:44,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.89 | bwd_microstep: 426.75 | bwd_inner_microstep: 426.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:15:45,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.46 | bwd_microstep: 591.66 | bwd_inner_microstep: 591.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:15:46,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.70 | bwd_microstep: 486.12 | bwd_inner_microstep: 485.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:15:47,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.87 | bwd_microstep: 531.49 | bwd_inner_microstep: 531.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:15:48,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.09 | bwd_microstep: 564.06 | bwd_inner_microstep: 563.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:15:49,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.93 | bwd_microstep: 460.77 | bwd_inner_microstep: 460.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:15:53,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.99 | optimizer_gradients: 14.08 | optimizer_step: 23.34
[2024-12-05 14:15:53,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.70 | bwd_microstep: 2602.91 | bwd_inner_microstep: 2590.71 | bwd_allreduce_microstep: 12.12 | step_microstep: 112.20
[2024-12-05 14:15:53,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4601.18 | bwd: 10101.43 | bwd_inner: 10087.63 | bwd_allreduce: 12.73 | step: 113.27

 31%|███       | 118/384 [32:49<1:24:53, 19.15s/it]
                                                   
{'loss': 0.1478, 'grad_norm': 1.604650616645813, 'learning_rate': 8.126872321608183e-07, 'epoch': 0.61}

 31%|███       | 118/384 [32:49<1:24:53, 19.15s/it][2024-12-05 14:15:54,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.59 | bwd_microstep: 651.56 | bwd_inner_microstep: 651.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:15:55,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.67 | bwd_microstep: 482.68 | bwd_inner_microstep: 482.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:15:56,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.09 | bwd_microstep: 433.21 | bwd_inner_microstep: 433.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:15:56,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.89 | bwd_microstep: 539.06 | bwd_inner_microstep: 538.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:15:57,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.22 | bwd_microstep: 402.29 | bwd_inner_microstep: 402.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:15:58,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.93 | bwd_microstep: 401.48 | bwd_inner_microstep: 401.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:15:59,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.75 | bwd_microstep: 422.22 | bwd_inner_microstep: 422.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:16:00,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.45 | bwd_microstep: 640.25 | bwd_inner_microstep: 640.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:16:01,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.37 | bwd_microstep: 409.45 | bwd_inner_microstep: 409.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:16:02,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.05 | bwd_microstep: 688.02 | bwd_inner_microstep: 687.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:16:04,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 328.99 | bwd_microstep: 618.48 | bwd_inner_microstep: 618.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:16:05,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.48 | bwd_microstep: 523.20 | bwd_inner_microstep: 523.08 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:16:06,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.49 | bwd_microstep: 492.21 | bwd_inner_microstep: 492.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:16:07,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.69 | bwd_microstep: 418.69 | bwd_inner_microstep: 418.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:16:08,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.34 | bwd_microstep: 403.65 | bwd_inner_microstep: 403.49 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:16:11,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.96 | optimizer_gradients: 14.09 | optimizer_step: 23.43
[2024-12-05 14:16:11,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.38 | bwd_microstep: 2722.15 | bwd_inner_microstep: 2709.97 | bwd_allreduce_microstep: 12.06 | step_microstep: 113.58
[2024-12-05 14:16:11,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4584.46 | bwd: 10248.62 | bwd_inner: 10234.76 | bwd_allreduce: 12.69 | step: 114.66

 31%|███       | 119/384 [33:07<1:23:34, 18.92s/it]
                                                   
{'loss': 0.1639, 'grad_norm': 1.6860443353652954, 'learning_rate': 8.093811442476572e-07, 'epoch': 0.62}

 31%|███       | 119/384 [33:07<1:23:34, 18.92s/it][2024-12-05 14:16:12,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.06 | bwd_microstep: 427.65 | bwd_inner_microstep: 427.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:16:13,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.81 | bwd_microstep: 675.02 | bwd_inner_microstep: 674.85 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:16:15,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.97 | bwd_microstep: 594.09 | bwd_inner_microstep: 593.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:16:16,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.92 | bwd_microstep: 428.02 | bwd_inner_microstep: 427.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:16:17,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.88 | bwd_microstep: 433.99 | bwd_inner_microstep: 433.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:16:18,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.71 | bwd_microstep: 396.99 | bwd_inner_microstep: 396.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:16:19,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.83 | bwd_microstep: 432.85 | bwd_inner_microstep: 432.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:16:20,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.42 | bwd_microstep: 476.91 | bwd_inner_microstep: 476.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:16:20,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.25 | bwd_microstep: 400.41 | bwd_inner_microstep: 400.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:16:21,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.37 | bwd_microstep: 453.23 | bwd_inner_microstep: 453.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:16:23,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.37 | bwd_microstep: 449.76 | bwd_inner_microstep: 449.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:16:24,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.20 | bwd_microstep: 457.14 | bwd_inner_microstep: 457.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:16:25,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 348.67 | bwd_microstep: 708.10 | bwd_inner_microstep: 707.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:16:27,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 396.53 | bwd_microstep: 879.42 | bwd_inner_microstep: 879.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:16:28,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.25 | bwd_microstep: 417.23 | bwd_inner_microstep: 417.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:16:29,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.93 | optimizer_gradients: 14.10 | optimizer_step: 23.06
[2024-12-05 14:16:29,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.08 | bwd_microstep: 1048.77 | bwd_inner_microstep: 1036.62 | bwd_allreduce_microstep: 12.07 | step_microstep: 112.65
[2024-12-05 14:16:29,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4548.39 | bwd: 8679.61 | bwd_inner: 8665.75 | bwd_allreduce: 12.72 | step: 113.88

 31%|███▏      | 120/384 [33:26<1:22:38, 18.78s/it]
                                                   
{'loss': 0.1511, 'grad_norm': 1.5915076732635498, 'learning_rate': 8.060529912738314e-07, 'epoch': 0.62}

 31%|███▏      | 120/384 [33:26<1:22:38, 18.78s/it][2024-12-05 14:16:30,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.31 | bwd_microstep: 387.40 | bwd_inner_microstep: 387.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:16:31,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.59 | bwd_microstep: 404.08 | bwd_inner_microstep: 403.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:16:32,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.59 | bwd_microstep: 410.44 | bwd_inner_microstep: 410.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:16:33,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.80 | bwd_microstep: 451.92 | bwd_inner_microstep: 451.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:16:34,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.77 | bwd_microstep: 461.26 | bwd_inner_microstep: 461.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:16:36,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.42 | bwd_microstep: 631.54 | bwd_inner_microstep: 631.35 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.13
[2024-12-05 14:16:37,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.59 | bwd_microstep: 430.95 | bwd_inner_microstep: 430.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:16:38,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.29 | bwd_microstep: 482.61 | bwd_inner_microstep: 482.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:16:39,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.62 | bwd_microstep: 461.23 | bwd_inner_microstep: 461.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:16:40,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.77 | bwd_microstep: 439.03 | bwd_inner_microstep: 438.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:16:41,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 408.19 | bwd_microstep: 570.70 | bwd_inner_microstep: 570.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:16:42,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.10 | bwd_microstep: 404.23 | bwd_inner_microstep: 404.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:16:43,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.31 | bwd_microstep: 412.98 | bwd_inner_microstep: 412.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:16:44,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.94 | bwd_microstep: 404.57 | bwd_inner_microstep: 404.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:16:46,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.32 | bwd_microstep: 569.78 | bwd_inner_microstep: 569.63 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:16:48,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.63 | optimizer_gradients: 14.01 | optimizer_step: 23.13
[2024-12-05 14:16:48,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.24 | bwd_microstep: 1648.26 | bwd_inner_microstep: 1636.33 | bwd_allreduce_microstep: 11.83 | step_microstep: 112.20
[2024-12-05 14:16:48,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4444.95 | bwd: 8570.99 | bwd_inner: 8557.36 | bwd_allreduce: 12.50 | step: 113.45

 32%|███▏      | 121/384 [33:44<1:22:01, 18.71s/it]
                                                   
{'loss': 0.1848, 'grad_norm': 1.7241225242614746, 'learning_rate': 8.027030106031835e-07, 'epoch': 0.63}

 32%|███▏      | 121/384 [33:44<1:22:01, 18.71s/it][2024-12-05 14:16:49,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.22 | bwd_microstep: 385.11 | bwd_inner_microstep: 384.95 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:16:50,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.71 | bwd_microstep: 547.54 | bwd_inner_microstep: 547.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:16:51,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.54 | bwd_microstep: 500.29 | bwd_inner_microstep: 500.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:16:52,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.33 | bwd_microstep: 413.80 | bwd_inner_microstep: 413.68 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:16:53,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.31 | bwd_microstep: 412.81 | bwd_inner_microstep: 412.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:16:54,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.21 | bwd_microstep: 400.75 | bwd_inner_microstep: 400.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:16:55,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.61 | bwd_microstep: 447.45 | bwd_inner_microstep: 447.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:16:56,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.59 | bwd_microstep: 445.69 | bwd_inner_microstep: 445.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:16:57,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.11 | bwd_microstep: 431.11 | bwd_inner_microstep: 431.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:16:58,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.27 | bwd_microstep: 433.95 | bwd_inner_microstep: 433.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:16:59,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.38 | bwd_microstep: 487.86 | bwd_inner_microstep: 487.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:17:00,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.80 | bwd_microstep: 447.18 | bwd_inner_microstep: 447.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:17:01,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.55 | bwd_microstep: 427.29 | bwd_inner_microstep: 427.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:17:02,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 332.78 | bwd_microstep: 673.54 | bwd_inner_microstep: 673.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:17:04,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.02 | bwd_microstep: 594.90 | bwd_inner_microstep: 594.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:17:06,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.45 | optimizer_gradients: 13.99 | optimizer_step: 23.07
[2024-12-05 14:17:06,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.03 | bwd_microstep: 1745.68 | bwd_inner_microstep: 1733.75 | bwd_allreduce_microstep: 11.83 | step_microstep: 112.11
[2024-12-05 14:17:06,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4444.51 | bwd: 8794.94 | bwd_inner: 8781.40 | bwd_allreduce: 12.47 | step: 113.23

 32%|███▏      | 122/384 [34:03<1:21:20, 18.63s/it]
                                                   
{'loss': 0.1816, 'grad_norm': 1.7747095823287964, 'learning_rate': 7.993314411563075e-07, 'epoch': 0.63}

 32%|███▏      | 122/384 [34:03<1:21:20, 18.63s/it][2024-12-05 14:17:07,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.68 | bwd_microstep: 418.09 | bwd_inner_microstep: 417.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:17:09,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.01 | bwd_microstep: 479.61 | bwd_inner_microstep: 479.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:17:10,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.57 | bwd_microstep: 524.72 | bwd_inner_microstep: 524.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 14:17:10,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.70 | bwd_microstep: 484.45 | bwd_inner_microstep: 484.30 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:17:11,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.78 | bwd_microstep: 443.77 | bwd_inner_microstep: 443.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:17:12,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.42 | bwd_microstep: 406.25 | bwd_inner_microstep: 406.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:17:13,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.02 | bwd_microstep: 483.75 | bwd_inner_microstep: 483.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:17:14,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.97 | bwd_microstep: 433.70 | bwd_inner_microstep: 433.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:17:15,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.21 | bwd_microstep: 433.43 | bwd_inner_microstep: 433.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:17:17,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.74 | bwd_microstep: 482.63 | bwd_inner_microstep: 482.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:17:18,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.31 | bwd_microstep: 375.88 | bwd_inner_microstep: 375.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:17:19,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.62 | bwd_microstep: 402.13 | bwd_inner_microstep: 402.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:17:20,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.65 | bwd_microstep: 395.24 | bwd_inner_microstep: 395.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:17:22,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.62 | bwd_microstep: 565.14 | bwd_inner_microstep: 565.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:17:23,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.85 | bwd_microstep: 563.30 | bwd_inner_microstep: 563.17 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:17:26,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.43 | optimizer_gradients: 14.03 | optimizer_step: 23.23
[2024-12-05 14:17:26,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.71 | bwd_microstep: 1776.28 | bwd_inner_microstep: 1764.28 | bwd_allreduce_microstep: 11.91 | step_microstep: 112.74
[2024-12-05 14:17:26,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4399.94 | bwd: 8668.37 | bwd_inner: 8654.73 | bwd_allreduce: 12.54 | step: 113.97

 32%|███▏      | 123/384 [34:22<1:21:55, 18.83s/it]
                                                   
{'loss': 0.1696, 'grad_norm': 1.6497771739959717, 'learning_rate': 7.959385233935085e-07, 'epoch': 0.64}

 32%|███▏      | 123/384 [34:22<1:21:55, 18.83s/it][2024-12-05 14:17:27,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 332.07 | bwd_microstep: 643.99 | bwd_inner_microstep: 643.86 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 14:17:28,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.14 | bwd_microstep: 398.50 | bwd_inner_microstep: 398.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:17:29,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.39 | bwd_microstep: 452.68 | bwd_inner_microstep: 452.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:17:30,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.85 | bwd_microstep: 434.08 | bwd_inner_microstep: 433.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:17:32,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.34 | bwd_microstep: 596.83 | bwd_inner_microstep: 596.71 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:17:33,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.95 | bwd_microstep: 401.25 | bwd_inner_microstep: 401.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:17:33,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.04 | bwd_microstep: 450.34 | bwd_inner_microstep: 450.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:17:34,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.43 | bwd_microstep: 442.81 | bwd_inner_microstep: 442.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:17:35,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 345.45 | bwd_microstep: 713.02 | bwd_inner_microstep: 712.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:17:37,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.91 | bwd_microstep: 480.01 | bwd_inner_microstep: 479.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:17:38,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.87 | bwd_microstep: 447.81 | bwd_inner_microstep: 447.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:17:39,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.98 | bwd_microstep: 592.04 | bwd_inner_microstep: 591.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:17:40,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.47 | bwd_microstep: 428.93 | bwd_inner_microstep: 428.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:17:41,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.41 | bwd_microstep: 409.58 | bwd_inner_microstep: 409.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:17:42,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.51 | bwd_microstep: 442.73 | bwd_inner_microstep: 442.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:17:45,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.63 | optimizer_gradients: 14.07 | optimizer_step: 23.38
[2024-12-05 14:17:45,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.39 | bwd_microstep: 1702.54 | bwd_inner_microstep: 1690.40 | bwd_allreduce_microstep: 12.05 | step_microstep: 112.68
[2024-12-05 14:17:45,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4514.26 | bwd: 9037.14 | bwd_inner: 9023.36 | bwd_allreduce: 12.70 | step: 113.76

 32%|███▏      | 124/384 [34:41<1:22:02, 18.93s/it]
                                                   
{'loss': 0.1297, 'grad_norm': 1.4712307453155518, 'learning_rate': 7.925244992976537e-07, 'epoch': 0.64}

 32%|███▏      | 124/384 [34:41<1:22:02, 18.93s/it][2024-12-05 14:17:46,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.32 | bwd_microstep: 419.18 | bwd_inner_microstep: 419.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:17:47,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.20 | bwd_microstep: 426.83 | bwd_inner_microstep: 426.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:17:48,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.39 | bwd_microstep: 453.45 | bwd_inner_microstep: 453.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:17:49,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.74 | bwd_microstep: 381.10 | bwd_inner_microstep: 380.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:17:50,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.52 | bwd_microstep: 376.82 | bwd_inner_microstep: 376.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:17:51,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.14 | bwd_microstep: 446.40 | bwd_inner_microstep: 446.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:17:52,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.26 | bwd_microstep: 455.99 | bwd_inner_microstep: 455.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:17:53,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.59 | bwd_microstep: 432.80 | bwd_inner_microstep: 432.65 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:17:54,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.04 | bwd_microstep: 425.57 | bwd_inner_microstep: 425.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:17:55,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.43 | bwd_microstep: 557.29 | bwd_inner_microstep: 557.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:17:56,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.10 | bwd_microstep: 718.53 | bwd_inner_microstep: 718.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:17:57,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.48 | bwd_microstep: 434.55 | bwd_inner_microstep: 434.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:17:58,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.75 | bwd_microstep: 422.48 | bwd_inner_microstep: 422.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:18:00,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.22 | bwd_microstep: 692.72 | bwd_inner_microstep: 692.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:18:01,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.47 | bwd_microstep: 606.11 | bwd_inner_microstep: 605.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:18:03,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.21 | optimizer_gradients: 14.04 | optimizer_step: 23.31
[2024-12-05 14:18:03,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.91 | bwd_microstep: 1395.27 | bwd_inner_microstep: 1383.33 | bwd_allreduce_microstep: 11.85 | step_microstep: 112.51
[2024-12-05 14:18:03,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4506.62 | bwd: 8645.09 | bwd_inner: 8631.55 | bwd_allreduce: 12.46 | step: 113.59

 33%|███▎      | 125/384 [34:59<1:20:34, 18.67s/it]
                                                   
{'loss': 0.1708, 'grad_norm': 1.618713140487671, 'learning_rate': 7.890896123569135e-07, 'epoch': 0.65}

 33%|███▎      | 125/384 [34:59<1:20:34, 18.67s/it][2024-12-05 14:18:05,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 354.99 | bwd_microstep: 701.50 | bwd_inner_microstep: 701.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:18:06,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.45 | bwd_microstep: 442.79 | bwd_inner_microstep: 442.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:18:07,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 326.98 | bwd_microstep: 617.91 | bwd_inner_microstep: 617.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:18:08,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.42 | bwd_microstep: 448.37 | bwd_inner_microstep: 448.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:18:09,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.93 | bwd_microstep: 455.59 | bwd_inner_microstep: 455.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:18:09,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.30 | bwd_microstep: 481.65 | bwd_inner_microstep: 481.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:18:10,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.09 | bwd_microstep: 453.95 | bwd_inner_microstep: 453.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:18:11,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.14 | bwd_microstep: 478.27 | bwd_inner_microstep: 478.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:18:12,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.76 | bwd_microstep: 588.92 | bwd_inner_microstep: 588.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:18:14,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.69 | bwd_microstep: 491.00 | bwd_inner_microstep: 490.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:18:15,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.43 | bwd_microstep: 433.06 | bwd_inner_microstep: 432.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:18:16,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.42 | bwd_microstep: 485.64 | bwd_inner_microstep: 485.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:18:17,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.55 | bwd_microstep: 406.51 | bwd_inner_microstep: 406.35 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08
[2024-12-05 14:18:18,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.54 | bwd_microstep: 436.60 | bwd_inner_microstep: 436.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:18:19,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.94 | bwd_microstep: 407.75 | bwd_inner_microstep: 407.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:18:21,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.69 | optimizer_gradients: 14.04 | optimizer_step: 23.19
[2024-12-05 14:18:21,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.47 | bwd_microstep: 1984.73 | bwd_inner_microstep: 1972.65 | bwd_allreduce_microstep: 11.95 | step_microstep: 113.99
[2024-12-05 14:18:21,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4568.18 | bwd: 9314.24 | bwd_inner: 9300.49 | bwd_allreduce: 12.63 | step: 115.31

 33%|███▎      | 126/384 [35:18<1:20:06, 18.63s/it]
                                                   
{'loss': 0.1646, 'grad_norm': 1.8550593852996826, 'learning_rate': 7.856341075473961e-07, 'epoch': 0.65}

 33%|███▎      | 126/384 [35:18<1:20:06, 18.63s/it][2024-12-05 14:18:22,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.61 | bwd_microstep: 418.72 | bwd_inner_microstep: 418.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:18:23,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.59 | bwd_microstep: 379.15 | bwd_inner_microstep: 379.02 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:18:24,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.24 | bwd_microstep: 566.50 | bwd_inner_microstep: 566.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:18:26,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.13 | bwd_microstep: 449.42 | bwd_inner_microstep: 449.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:18:27,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 407.55 | bwd_microstep: 893.13 | bwd_inner_microstep: 892.95 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 14:18:29,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 351.17 | bwd_microstep: 722.35 | bwd_inner_microstep: 722.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:18:31,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 498.19 | bwd_microstep: 1191.91 | bwd_inner_microstep: 1191.75 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09
[2024-12-05 14:18:32,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.36 | bwd_microstep: 447.90 | bwd_inner_microstep: 447.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:18:33,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.39 | bwd_microstep: 388.83 | bwd_inner_microstep: 388.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:18:34,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.06 | bwd_microstep: 539.43 | bwd_inner_microstep: 539.31 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:18:35,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.94 | bwd_microstep: 449.21 | bwd_inner_microstep: 449.08 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:18:37,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.46 | bwd_microstep: 433.64 | bwd_inner_microstep: 433.51 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:18:38,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.62 | bwd_microstep: 398.01 | bwd_inner_microstep: 397.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:18:39,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.80 | bwd_microstep: 576.79 | bwd_inner_microstep: 576.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:18:40,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.62 | bwd_microstep: 427.37 | bwd_inner_microstep: 427.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:18:41,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.35 | optimizer_gradients: 14.04 | optimizer_step: 23.18
[2024-12-05 14:18:41,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.69 | bwd_microstep: 777.48 | bwd_inner_microstep: 765.33 | bwd_allreduce_microstep: 12.07 | step_microstep: 112.19
[2024-12-05 14:18:41,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4822.52 | bwd: 9059.86 | bwd_inner: 9045.94 | bwd_allreduce: 12.74 | step: 113.27

 33%|███▎      | 127/384 [35:38<1:21:30, 19.03s/it]
                                                   
{'loss': 0.1555, 'grad_norm': 1.7929205894470215, 'learning_rate': 7.821582313156763e-07, 'epoch': 0.66}

 33%|███▎      | 127/384 [35:38<1:21:30, 19.03s/it][2024-12-05 14:18:42,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.28 | bwd_microstep: 422.55 | bwd_inner_microstep: 422.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:18:44,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.29 | bwd_microstep: 444.09 | bwd_inner_microstep: 443.89 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10
[2024-12-05 14:18:45,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 329.99 | bwd_microstep: 621.47 | bwd_inner_microstep: 621.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:18:46,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.17 | bwd_microstep: 688.76 | bwd_inner_microstep: 688.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:18:47,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.35 | bwd_microstep: 482.66 | bwd_inner_microstep: 482.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:18:48,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.11 | bwd_microstep: 429.09 | bwd_inner_microstep: 428.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:18:49,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.80 | bwd_microstep: 403.22 | bwd_inner_microstep: 403.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:18:50,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.16 | bwd_microstep: 554.02 | bwd_inner_microstep: 553.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:18:52,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.05 | bwd_microstep: 436.21 | bwd_inner_microstep: 436.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:18:53,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.16 | bwd_microstep: 685.65 | bwd_inner_microstep: 685.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:18:53,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.54 | bwd_microstep: 413.92 | bwd_inner_microstep: 413.79 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:18:55,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 349.00 | bwd_microstep: 711.38 | bwd_inner_microstep: 711.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:18:56,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.02 | bwd_microstep: 481.42 | bwd_inner_microstep: 481.25 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:18:57,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.09 | bwd_microstep: 459.36 | bwd_inner_microstep: 459.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:18:58,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.79 | bwd_microstep: 577.76 | bwd_inner_microstep: 577.59 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:19:00,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.39 | optimizer_gradients: 14.17 | optimizer_step: 23.45
[2024-12-05 14:19:00,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.75 | bwd_microstep: 837.37 | bwd_inner_microstep: 825.35 | bwd_allreduce_microstep: 11.93 | step_microstep: 114.97
[2024-12-05 14:19:00,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4716.62 | bwd: 8648.92 | bwd_inner: 8635.12 | bwd_allreduce: 12.62 | step: 116.14

 33%|███▎      | 128/384 [35:56<1:20:02, 18.76s/it]
                                                   
{'loss': 0.1882, 'grad_norm': 1.8011633157730103, 'learning_rate': 7.786622315612181e-07, 'epoch': 0.67}

 33%|███▎      | 128/384 [35:56<1:20:02, 18.76s/it][2024-12-05 14:19:01,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.48 | bwd_microstep: 438.15 | bwd_inner_microstep: 438.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:19:01,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.83 | bwd_microstep: 416.67 | bwd_inner_microstep: 416.53 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:19:02,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.92 | bwd_microstep: 428.20 | bwd_inner_microstep: 428.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:19:04,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 373.43 | bwd_microstep: 774.13 | bwd_inner_microstep: 774.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:19:05,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.19 | bwd_microstep: 404.20 | bwd_inner_microstep: 404.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:19:06,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 387.54 | bwd_microstep: 868.66 | bwd_inner_microstep: 868.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:19:07,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 343.66 | bwd_microstep: 430.18 | bwd_inner_microstep: 430.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:19:08,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.19 | bwd_microstep: 435.09 | bwd_inner_microstep: 434.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:19:09,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.66 | bwd_microstep: 505.88 | bwd_inner_microstep: 505.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:19:10,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.44 | bwd_microstep: 419.26 | bwd_inner_microstep: 419.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:19:11,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.85 | bwd_microstep: 595.20 | bwd_inner_microstep: 595.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:19:13,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.75 | bwd_microstep: 680.21 | bwd_inner_microstep: 680.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:19:14,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.57 | bwd_microstep: 682.44 | bwd_inner_microstep: 682.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:19:15,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.26 | bwd_microstep: 482.24 | bwd_inner_microstep: 482.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:19:16,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.08 | bwd_microstep: 549.70 | bwd_inner_microstep: 549.56 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:19:19,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.46 | optimizer_gradients: 14.07 | optimizer_step: 23.10
[2024-12-05 14:19:19,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 360.63 | bwd_microstep: 1768.20 | bwd_inner_microstep: 1756.16 | bwd_allreduce_microstep: 11.94 | step_microstep: 112.10
[2024-12-05 14:19:19,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4889.60 | bwd: 9878.38 | bwd_inner: 9864.76 | bwd_allreduce: 12.58 | step: 113.20

 34%|███▎      | 129/384 [36:15<1:20:21, 18.91s/it]
                                                   
{'loss': 0.1659, 'grad_norm': 1.642986536026001, 'learning_rate': 7.751463576186957e-07, 'epoch': 0.67}

 34%|███▎      | 129/384 [36:15<1:20:21, 18.91s/it][2024-12-05 14:19:20,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.08 | bwd_microstep: 666.67 | bwd_inner_microstep: 666.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:19:23,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.17 | bwd_microstep: 444.54 | bwd_inner_microstep: 444.39 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:19:24,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.01 | bwd_microstep: 351.59 | bwd_inner_microstep: 351.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:19:25,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.76 | bwd_microstep: 486.63 | bwd_inner_microstep: 486.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:19:27,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.48 | bwd_microstep: 486.03 | bwd_inner_microstep: 485.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:19:28,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.25 | bwd_microstep: 435.28 | bwd_inner_microstep: 435.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:19:29,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.28 | bwd_microstep: 403.29 | bwd_inner_microstep: 403.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:19:30,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.68 | bwd_microstep: 357.18 | bwd_inner_microstep: 357.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:19:31,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.96 | bwd_microstep: 457.92 | bwd_inner_microstep: 457.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:19:32,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.25 | bwd_microstep: 430.03 | bwd_inner_microstep: 429.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:19:33,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.26 | bwd_microstep: 413.30 | bwd_inner_microstep: 413.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:19:34,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.71 | bwd_microstep: 354.80 | bwd_inner_microstep: 354.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:19:35,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.87 | bwd_microstep: 525.72 | bwd_inner_microstep: 525.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:19:37,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.63 | bwd_microstep: 680.91 | bwd_inner_microstep: 680.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:19:38,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.24 | bwd_microstep: 404.64 | bwd_inner_microstep: 404.51 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:19:41,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.22 | optimizer_gradients: 14.06 | optimizer_step: 23.21
[2024-12-05 14:19:41,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.18 | bwd_microstep: 2280.47 | bwd_inner_microstep: 2268.42 | bwd_allreduce_microstep: 11.95 | step_microstep: 112.49
[2024-12-05 14:19:41,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4353.86 | bwd: 9179.01 | bwd_inner: 9165.28 | bwd_allreduce: 12.58 | step: 113.53

 34%|███▍      | 130/384 [36:37<1:24:07, 19.87s/it]
                                                   
{'loss': 0.1479, 'grad_norm': 1.547602653503418, 'learning_rate': 7.716108602402094e-07, 'epoch': 0.68}

 34%|███▍      | 130/384 [36:37<1:24:07, 19.87s/it][2024-12-05 14:19:42,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.41 | bwd_microstep: 396.37 | bwd_inner_microstep: 396.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:19:43,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.01 | bwd_microstep: 485.85 | bwd_inner_microstep: 485.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:19:44,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.74 | bwd_microstep: 449.94 | bwd_inner_microstep: 449.81 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:19:45,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.58 | bwd_microstep: 439.07 | bwd_inner_microstep: 438.92 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:19:47,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.39 | bwd_microstep: 487.94 | bwd_inner_microstep: 487.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:19:48,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.23 | bwd_microstep: 357.70 | bwd_inner_microstep: 357.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:19:49,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.49 | bwd_microstep: 455.62 | bwd_inner_microstep: 455.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:19:50,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.66 | bwd_microstep: 416.42 | bwd_inner_microstep: 416.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:19:51,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.09 | bwd_microstep: 429.58 | bwd_inner_microstep: 429.44 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:19:52,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.60 | bwd_microstep: 536.87 | bwd_inner_microstep: 536.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:19:53,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.62 | bwd_microstep: 452.56 | bwd_inner_microstep: 452.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:19:54,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.79 | bwd_microstep: 589.32 | bwd_inner_microstep: 589.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:19:55,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.33 | bwd_microstep: 569.76 | bwd_inner_microstep: 569.62 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:19:57,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.11 | bwd_microstep: 441.02 | bwd_inner_microstep: 440.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:19:58,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.53 | bwd_microstep: 583.96 | bwd_inner_microstep: 583.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:20:00,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.59 | optimizer_gradients: 14.08 | optimizer_step: 23.57
[2024-12-05 14:20:00,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.59 | bwd_microstep: 1815.58 | bwd_inner_microstep: 1803.50 | bwd_allreduce_microstep: 12.00 | step_microstep: 113.83
[2024-12-05 14:20:00,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4363.22 | bwd: 8907.52 | bwd_inner: 8893.78 | bwd_allreduce: 12.66 | step: 114.96

 34%|███▍      | 131/384 [36:56<1:22:39, 19.60s/it]
                                                   
{'loss': 0.1627, 'grad_norm': 1.6602659225463867, 'learning_rate': 7.680559915774033e-07, 'epoch': 0.68}

 34%|███▍      | 131/384 [36:56<1:22:39, 19.60s/it][2024-12-05 14:20:01,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.71 | bwd_microstep: 409.18 | bwd_inner_microstep: 409.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:20:02,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.44 | bwd_microstep: 594.70 | bwd_inner_microstep: 594.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:20:03,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.93 | bwd_microstep: 434.97 | bwd_inner_microstep: 434.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:20:04,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.09 | bwd_microstep: 399.93 | bwd_inner_microstep: 399.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:20:05,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.18 | bwd_microstep: 489.19 | bwd_inner_microstep: 489.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:20:06,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.08 | bwd_microstep: 437.88 | bwd_inner_microstep: 437.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:20:07,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.71 | bwd_microstep: 434.02 | bwd_inner_microstep: 433.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:20:08,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.93 | bwd_microstep: 355.83 | bwd_inner_microstep: 355.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:20:09,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.11 | bwd_microstep: 447.39 | bwd_inner_microstep: 447.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:20:10,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 243.87 | bwd_microstep: 376.27 | bwd_inner_microstep: 376.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:20:11,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.11 | bwd_microstep: 416.52 | bwd_inner_microstep: 416.37 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:20:11,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.57 | bwd_microstep: 444.49 | bwd_inner_microstep: 444.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:20:12,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.21 | bwd_microstep: 416.64 | bwd_inner_microstep: 416.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:20:14,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.91 | bwd_microstep: 467.21 | bwd_inner_microstep: 467.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:20:15,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.45 | bwd_microstep: 485.20 | bwd_inner_microstep: 484.99 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:20:18,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.18 | optimizer_gradients: 14.04 | optimizer_step: 23.20
[2024-12-05 14:20:18,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.69 | bwd_microstep: 2557.44 | bwd_inner_microstep: 2545.32 | bwd_allreduce_microstep: 12.02 | step_microstep: 113.51
[2024-12-05 14:20:18,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4287.02 | bwd: 9166.85 | bwd_inner: 9153.03 | bwd_allreduce: 12.71 | step: 114.74

 34%|███▍      | 132/384 [37:14<1:20:30, 19.17s/it]
                                                   
{'loss': 0.1637, 'grad_norm': 1.8091386556625366, 'learning_rate': 7.644820051634812e-07, 'epoch': 0.69}

 34%|███▍      | 132/384 [37:14<1:20:30, 19.17s/it][2024-12-05 14:20:19,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.42 | bwd_microstep: 399.60 | bwd_inner_microstep: 399.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:20:20,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.02 | bwd_microstep: 536.48 | bwd_inner_microstep: 536.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:20:21,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.59 | bwd_microstep: 483.29 | bwd_inner_microstep: 483.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:20:23,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 393.53 | bwd_microstep: 876.65 | bwd_inner_microstep: 876.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:20:24,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.75 | bwd_microstep: 436.90 | bwd_inner_microstep: 436.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:20:25,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.05 | bwd_microstep: 591.36 | bwd_inner_microstep: 591.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:20:26,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.85 | bwd_microstep: 697.86 | bwd_inner_microstep: 697.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:20:27,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.27 | bwd_microstep: 437.10 | bwd_inner_microstep: 436.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:20:28,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.16 | bwd_microstep: 349.81 | bwd_inner_microstep: 349.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:20:29,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.73 | bwd_microstep: 413.35 | bwd_inner_microstep: 413.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:20:31,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 350.32 | bwd_microstep: 720.10 | bwd_inner_microstep: 719.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:20:32,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.61 | bwd_microstep: 415.46 | bwd_inner_microstep: 415.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:20:33,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.56 | bwd_microstep: 399.54 | bwd_inner_microstep: 399.41 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:20:34,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.52 | bwd_microstep: 490.36 | bwd_inner_microstep: 490.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:20:34,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.20 | bwd_microstep: 454.90 | bwd_inner_microstep: 454.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:20:36,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.52 | optimizer_gradients: 14.23 | optimizer_step: 23.18
[2024-12-05 14:20:36,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.30 | bwd_microstep: 1183.60 | bwd_inner_microstep: 1171.35 | bwd_allreduce_microstep: 12.15 | step_microstep: 112.65
[2024-12-05 14:20:36,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4572.96 | bwd: 8886.36 | bwd_inner: 8872.48 | bwd_allreduce: 12.78 | step: 113.76

 35%|███▍      | 133/384 [37:32<1:18:47, 18.83s/it]
                                                   
{'loss': 0.1599, 'grad_norm': 1.669487476348877, 'learning_rate': 7.608891558951248e-07, 'epoch': 0.69}

 35%|███▍      | 133/384 [37:32<1:18:47, 18.83s/it][2024-12-05 14:20:37,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.81 | bwd_microstep: 404.05 | bwd_inner_microstep: 403.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:20:38,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.34 | bwd_microstep: 355.49 | bwd_inner_microstep: 355.35 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:20:39,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.84 | bwd_microstep: 391.32 | bwd_inner_microstep: 391.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:20:40,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.80 | bwd_microstep: 407.57 | bwd_inner_microstep: 407.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:20:41,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.66 | bwd_microstep: 401.58 | bwd_inner_microstep: 401.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:20:42,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.17 | bwd_microstep: 484.99 | bwd_inner_microstep: 484.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:20:43,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.50 | bwd_microstep: 522.46 | bwd_inner_microstep: 522.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:20:44,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 228.35 | bwd_microstep: 329.57 | bwd_inner_microstep: 329.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:20:45,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.57 | bwd_microstep: 499.72 | bwd_inner_microstep: 499.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:20:46,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.12 | bwd_microstep: 413.48 | bwd_inner_microstep: 413.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:20:47,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.70 | bwd_microstep: 451.02 | bwd_inner_microstep: 450.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:20:47,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.24 | bwd_microstep: 447.94 | bwd_inner_microstep: 447.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:20:48,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.49 | bwd_microstep: 564.35 | bwd_inner_microstep: 564.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:20:50,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.07 | bwd_microstep: 620.10 | bwd_inner_microstep: 619.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:20:51,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.50 | bwd_microstep: 413.15 | bwd_inner_microstep: 413.01 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.12
[2024-12-05 14:20:54,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.86 | optimizer_gradients: 14.05 | optimizer_step: 23.19
[2024-12-05 14:20:54,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.44 | bwd_microstep: 2352.86 | bwd_inner_microstep: 2340.78 | bwd_allreduce_microstep: 11.98 | step_microstep: 112.66
[2024-12-05 14:20:54,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4307.65 | bwd: 9059.65 | bwd_inner: 9045.95 | bwd_allreduce: 12.61 | step: 113.76

 35%|███▍      | 134/384 [37:50<1:17:01, 18.49s/it]
                                                   
{'loss': 0.1654, 'grad_norm': 1.9016000032424927, 'learning_rate': 7.572777000143145e-07, 'epoch': 0.7}

 35%|███▍      | 134/384 [37:50<1:17:01, 18.49s/it][2024-12-05 14:20:55,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 359.39 | bwd_microstep: 711.97 | bwd_inner_microstep: 711.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:20:56,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.91 | bwd_microstep: 549.55 | bwd_inner_microstep: 549.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:20:58,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.30 | bwd_microstep: 593.30 | bwd_inner_microstep: 593.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:20:59,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.67 | bwd_microstep: 431.03 | bwd_inner_microstep: 430.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:00,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.89 | bwd_microstep: 540.26 | bwd_inner_microstep: 540.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:01,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 238.43 | bwd_microstep: 362.87 | bwd_inner_microstep: 362.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:21:02,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.34 | bwd_microstep: 405.15 | bwd_inner_microstep: 405.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:03,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.08 | bwd_microstep: 431.84 | bwd_inner_microstep: 431.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:21:04,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.16 | bwd_microstep: 405.06 | bwd_inner_microstep: 404.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:21:05,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.46 | bwd_microstep: 431.75 | bwd_inner_microstep: 431.61 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:21:06,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.96 | bwd_microstep: 416.39 | bwd_inner_microstep: 416.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:21:07,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.47 | bwd_microstep: 449.88 | bwd_inner_microstep: 449.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:21:08,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.60 | bwd_microstep: 546.03 | bwd_inner_microstep: 545.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:21:09,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.62 | bwd_microstep: 442.52 | bwd_inner_microstep: 442.37 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:21:10,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.66 | bwd_microstep: 564.14 | bwd_inner_microstep: 563.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:12,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.40 | optimizer_gradients: 14.11 | optimizer_step: 23.14
[2024-12-05 14:21:12,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.40 | bwd_microstep: 1425.67 | bwd_inner_microstep: 1413.63 | bwd_allreduce_microstep: 11.95 | step_microstep: 112.23
[2024-12-05 14:21:12,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4447.40 | bwd: 8707.41 | bwd_inner: 8693.64 | bwd_allreduce: 12.60 | step: 113.31

 35%|███▌      | 135/384 [38:08<1:16:36, 18.46s/it]
                                                   
{'loss': 0.1482, 'grad_norm': 1.4672502279281616, 'learning_rate': 7.536478950900536e-07, 'epoch': 0.7}

 35%|███▌      | 135/384 [38:08<1:16:36, 18.46s/it][2024-12-05 14:21:13,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.61 | bwd_microstep: 427.01 | bwd_inner_microstep: 426.86 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.06
[2024-12-05 14:21:14,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.78 | bwd_microstep: 488.87 | bwd_inner_microstep: 488.73 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:21:15,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.57 | bwd_microstep: 444.40 | bwd_inner_microstep: 444.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:17,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 341.57 | bwd_microstep: 690.74 | bwd_inner_microstep: 690.57 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 14:21:17,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.81 | bwd_microstep: 433.48 | bwd_inner_microstep: 433.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:19,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 373.45 | bwd_microstep: 780.14 | bwd_inner_microstep: 780.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:21:20,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.44 | bwd_microstep: 450.30 | bwd_inner_microstep: 450.16 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:21:21,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.07 | bwd_microstep: 456.08 | bwd_inner_microstep: 455.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:21:23,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.78 | bwd_microstep: 453.94 | bwd_inner_microstep: 453.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:21:24,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.71 | bwd_microstep: 481.94 | bwd_inner_microstep: 481.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:21:25,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.36 | bwd_microstep: 369.02 | bwd_inner_microstep: 368.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:26,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.04 | bwd_microstep: 409.67 | bwd_inner_microstep: 409.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:21:27,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.27 | bwd_microstep: 463.91 | bwd_inner_microstep: 463.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:28,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.72 | bwd_microstep: 450.44 | bwd_inner_microstep: 450.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:29,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.01 | bwd_microstep: 439.56 | bwd_inner_microstep: 439.43 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:21:31,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.78 | optimizer_gradients: 14.29 | optimizer_step: 23.32
[2024-12-05 14:21:31,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.02 | bwd_microstep: 1660.68 | bwd_inner_microstep: 1648.21 | bwd_allreduce_microstep: 12.35 | step_microstep: 115.04
[2024-12-05 14:21:31,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4508.27 | bwd: 8900.20 | bwd_inner: 8886.01 | bwd_allreduce: 13.03 | step: 116.26

 35%|███▌      | 136/384 [38:27<1:16:57, 18.62s/it]
                                                   
{'loss': 0.1214, 'grad_norm': 1.4602234363555908, 'learning_rate': 7.5e-07, 'epoch': 0.71}

 35%|███▌      | 136/384 [38:27<1:16:57, 18.62s/it][2024-12-05 14:21:32,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.58 | bwd_microstep: 371.45 | bwd_inner_microstep: 371.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:34,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 358.17 | bwd_microstep: 741.59 | bwd_inner_microstep: 741.45 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:21:35,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.62 | bwd_microstep: 413.51 | bwd_inner_microstep: 413.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:36,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.79 | bwd_microstep: 404.88 | bwd_inner_microstep: 404.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:37,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.96 | bwd_microstep: 429.47 | bwd_inner_microstep: 429.34 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:21:38,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.99 | bwd_microstep: 532.80 | bwd_inner_microstep: 532.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:21:39,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.73 | bwd_microstep: 624.63 | bwd_inner_microstep: 624.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:40,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.89 | bwd_microstep: 453.11 | bwd_inner_microstep: 452.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:41,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.21 | bwd_microstep: 415.07 | bwd_inner_microstep: 414.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:42,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.54 | bwd_microstep: 431.32 | bwd_inner_microstep: 431.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:21:43,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.51 | bwd_microstep: 413.60 | bwd_inner_microstep: 413.47 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:21:44,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.06 | bwd_microstep: 562.16 | bwd_inner_microstep: 562.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:21:45,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.14 | bwd_microstep: 415.50 | bwd_inner_microstep: 415.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:46,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.83 | bwd_microstep: 450.11 | bwd_inner_microstep: 449.95 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:21:47,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 341.80 | bwd_microstep: 689.59 | bwd_inner_microstep: 689.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:49,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.06 | optimizer_gradients: 14.05 | optimizer_step: 23.35
[2024-12-05 14:21:49,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.84 | bwd_microstep: 902.66 | bwd_inner_microstep: 890.55 | bwd_allreduce_microstep: 12.02 | step_microstep: 112.56
[2024-12-05 14:21:49,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4474.76 | bwd: 8251.43 | bwd_inner: 8237.61 | bwd_allreduce: 12.65 | step: 113.60

 36%|███▌      | 137/384 [38:45<1:15:59, 18.46s/it]
                                                   
{'loss': 0.1406, 'grad_norm': 1.725661277770996, 'learning_rate': 7.463342749120013e-07, 'epoch': 0.71}

 36%|███▌      | 137/384 [38:45<1:15:59, 18.46s/it][2024-12-05 14:21:50,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.81 | bwd_microstep: 585.62 | bwd_inner_microstep: 585.41 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.14
[2024-12-05 14:21:52,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.70 | bwd_microstep: 618.63 | bwd_inner_microstep: 618.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:21:53,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.07 | bwd_microstep: 453.71 | bwd_inner_microstep: 453.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:21:54,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.52 | bwd_microstep: 434.39 | bwd_inner_microstep: 434.25 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 14:21:55,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.14 | bwd_microstep: 571.09 | bwd_inner_microstep: 570.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:56,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.40 | bwd_microstep: 356.48 | bwd_inner_microstep: 356.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:57,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.11 | bwd_microstep: 371.96 | bwd_inner_microstep: 371.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:21:57,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.57 | bwd_microstep: 421.64 | bwd_inner_microstep: 421.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:21:58,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.06 | bwd_microstep: 482.93 | bwd_inner_microstep: 482.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:21:59,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.74 | bwd_microstep: 516.13 | bwd_inner_microstep: 516.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:22:01,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.26 | bwd_microstep: 560.06 | bwd_inner_microstep: 559.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:22:02,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.14 | bwd_microstep: 446.19 | bwd_inner_microstep: 446.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:22:02,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.86 | bwd_microstep: 350.87 | bwd_inner_microstep: 350.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:22:03,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.90 | bwd_microstep: 436.12 | bwd_inner_microstep: 435.96 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:22:05,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 341.67 | bwd_microstep: 685.16 | bwd_inner_microstep: 685.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:22:08,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.81 | optimizer_gradients: 14.05 | optimizer_step: 23.08
[2024-12-05 14:22:08,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.50 | bwd_microstep: 2671.71 | bwd_inner_microstep: 2659.50 | bwd_allreduce_microstep: 12.10 | step_microstep: 112.52
[2024-12-05 14:22:08,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4454.52 | bwd: 9962.69 | bwd_inner: 9948.78 | bwd_allreduce: 12.79 | step: 113.69

 36%|███▌      | 138/384 [39:04<1:16:00, 18.54s/it]
                                                   
{'loss': 0.1492, 'grad_norm': 1.6164398193359375, 'learning_rate': 7.426509812655405e-07, 'epoch': 0.72}

 36%|███▌      | 138/384 [39:04<1:16:00, 18.54s/it][2024-12-05 14:22:09,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.31 | bwd_microstep: 377.04 | bwd_inner_microstep: 376.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:22:10,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.76 | bwd_microstep: 476.06 | bwd_inner_microstep: 475.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:22:11,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.47 | bwd_microstep: 401.51 | bwd_inner_microstep: 401.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:22:12,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.61 | bwd_microstep: 592.24 | bwd_inner_microstep: 592.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:22:13,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 244.63 | bwd_microstep: 378.97 | bwd_inner_microstep: 378.81 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:22:14,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.01 | bwd_microstep: 431.61 | bwd_inner_microstep: 431.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:22:15,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.67 | bwd_microstep: 431.04 | bwd_inner_microstep: 430.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:22:17,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 359.13 | bwd_microstep: 740.69 | bwd_inner_microstep: 740.55 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:22:18,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.18 | bwd_microstep: 514.55 | bwd_inner_microstep: 514.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:22:19,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.08 | bwd_microstep: 425.52 | bwd_inner_microstep: 425.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:22:20,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.58 | bwd_microstep: 409.03 | bwd_inner_microstep: 408.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:22:21,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.76 | bwd_microstep: 402.60 | bwd_inner_microstep: 402.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:22:21,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.87 | bwd_microstep: 405.93 | bwd_inner_microstep: 405.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:22:23,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 420.41 | bwd_microstep: 929.94 | bwd_inner_microstep: 929.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:22:24,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.87 | bwd_microstep: 620.41 | bwd_inner_microstep: 620.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:22:26,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.03 | optimizer_gradients: 14.09 | optimizer_step: 23.46
[2024-12-05 14:22:26,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.30 | bwd_microstep: 821.96 | bwd_inner_microstep: 809.75 | bwd_allreduce_microstep: 12.10 | step_microstep: 113.25
[2024-12-05 14:22:26,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4516.72 | bwd: 8359.08 | bwd_inner: 8345.25 | bwd_allreduce: 12.75 | step: 114.32

 36%|███▌      | 139/384 [39:22<1:14:37, 18.28s/it]
                                                   
{'loss': 0.1669, 'grad_norm': 1.609312891960144, 'learning_rate': 7.389503817530905e-07, 'epoch': 0.72}

 36%|███▌      | 139/384 [39:22<1:14:37, 18.28s/it][2024-12-05 14:22:27,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.89 | bwd_microstep: 386.05 | bwd_inner_microstep: 385.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:22:28,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.06 | bwd_microstep: 585.33 | bwd_inner_microstep: 585.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:22:29,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.12 | bwd_microstep: 404.54 | bwd_inner_microstep: 404.37 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 14:22:30,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.12 | bwd_microstep: 456.52 | bwd_inner_microstep: 456.36 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:22:31,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.13 | bwd_microstep: 448.84 | bwd_inner_microstep: 448.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:22:32,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.82 | bwd_microstep: 435.66 | bwd_inner_microstep: 435.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:22:33,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.40 | bwd_microstep: 447.50 | bwd_inner_microstep: 447.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:22:34,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.23 | bwd_microstep: 616.48 | bwd_inner_microstep: 616.33 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:22:35,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.49 | bwd_microstep: 412.68 | bwd_inner_microstep: 412.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:22:36,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.42 | bwd_microstep: 589.29 | bwd_inner_microstep: 589.15 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:22:37,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.91 | bwd_microstep: 480.09 | bwd_inner_microstep: 479.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:22:38,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.16 | bwd_microstep: 460.07 | bwd_inner_microstep: 459.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:22:40,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.45 | bwd_microstep: 434.55 | bwd_inner_microstep: 434.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:22:40,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.06 | bwd_microstep: 435.29 | bwd_inner_microstep: 435.16 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:22:41,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.45 | bwd_microstep: 507.52 | bwd_inner_microstep: 507.40 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:22:44,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.77 | optimizer_gradients: 14.03 | optimizer_step: 23.37
[2024-12-05 14:22:44,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.64 | bwd_microstep: 1745.43 | bwd_inner_microstep: 1733.47 | bwd_allreduce_microstep: 11.87 | step_microstep: 112.62
[2024-12-05 14:22:44,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4424.45 | bwd: 8845.84 | bwd_inner: 8832.14 | bwd_allreduce: 12.55 | step: 113.74

 36%|███▋      | 140/384 [39:40<1:13:53, 18.17s/it]
                                                   
{'loss': 0.1318, 'grad_norm': 1.512629508972168, 'learning_rate': 7.352327403013779e-07, 'epoch': 0.73}

 36%|███▋      | 140/384 [39:40<1:13:53, 18.17s/it][2024-12-05 14:22:44,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.50 | bwd_microstep: 391.87 | bwd_inner_microstep: 391.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:22:45,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.42 | bwd_microstep: 487.65 | bwd_inner_microstep: 487.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:22:46,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.14 | bwd_microstep: 447.72 | bwd_inner_microstep: 447.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:22:47,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.86 | bwd_microstep: 462.90 | bwd_inner_microstep: 462.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:22:48,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.31 | bwd_microstep: 437.62 | bwd_inner_microstep: 437.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:22:50,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 370.32 | bwd_microstep: 790.35 | bwd_inner_microstep: 790.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:22:51,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.38 | bwd_microstep: 562.73 | bwd_inner_microstep: 562.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:22:52,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.95 | bwd_microstep: 676.89 | bwd_inner_microstep: 676.76 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:22:53,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.99 | bwd_microstep: 461.50 | bwd_inner_microstep: 461.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:22:54,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.72 | bwd_microstep: 336.37 | bwd_inner_microstep: 336.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:22:55,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.03 | bwd_microstep: 552.07 | bwd_inner_microstep: 551.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:22:56,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.75 | bwd_microstep: 483.26 | bwd_inner_microstep: 483.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:22:58,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.11 | bwd_microstep: 584.36 | bwd_inner_microstep: 584.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:22:59,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.59 | bwd_microstep: 591.64 | bwd_inner_microstep: 591.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:23:00,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.71 | bwd_microstep: 418.24 | bwd_inner_microstep: 418.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:23:03,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.39 | optimizer_gradients: 14.05 | optimizer_step: 23.13
[2024-12-05 14:23:03,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 350.67 | bwd_microstep: 2611.91 | bwd_inner_microstep: 2599.94 | bwd_allreduce_microstep: 11.88 | step_microstep: 112.14
[2024-12-05 14:23:03,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4676.51 | bwd: 10297.07 | bwd_inner: 10283.51 | bwd_allreduce: 12.49 | step: 113.23

 37%|███▋      | 141/384 [40:00<1:15:33, 18.66s/it]
                                                   
{'loss': 0.1762, 'grad_norm': 1.7129087448120117, 'learning_rate': 7.314983220525604e-07, 'epoch': 0.73}

 37%|███▋      | 141/384 [40:00<1:15:33, 18.66s/it][2024-12-05 14:23:04,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.97 | bwd_microstep: 404.09 | bwd_inner_microstep: 403.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:23:05,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.02 | bwd_microstep: 447.44 | bwd_inner_microstep: 447.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:23:06,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.41 | bwd_microstep: 425.03 | bwd_inner_microstep: 424.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:23:07,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.90 | bwd_microstep: 411.38 | bwd_inner_microstep: 411.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:23:08,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.39 | bwd_microstep: 403.75 | bwd_inner_microstep: 403.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:23:09,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.22 | bwd_microstep: 399.76 | bwd_inner_microstep: 399.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:23:10,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.55 | bwd_microstep: 587.49 | bwd_inner_microstep: 587.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:23:11,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.76 | bwd_microstep: 568.33 | bwd_inner_microstep: 568.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:23:12,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.31 | bwd_microstep: 559.54 | bwd_inner_microstep: 559.37 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:23:13,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.39 | bwd_microstep: 434.81 | bwd_inner_microstep: 434.66 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:23:14,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.29 | bwd_microstep: 563.83 | bwd_inner_microstep: 563.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:23:15,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.59 | bwd_microstep: 381.43 | bwd_inner_microstep: 381.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:23:16,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 395.12 | bwd_microstep: 880.87 | bwd_inner_microstep: 880.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:23:17,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.56 | bwd_microstep: 583.23 | bwd_inner_microstep: 583.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:23:19,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 354.39 | bwd_microstep: 733.60 | bwd_inner_microstep: 733.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:23:21,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.69 | optimizer_gradients: 14.04 | optimizer_step: 23.24
[2024-12-05 14:23:21,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.87 | bwd_microstep: 1522.83 | bwd_inner_microstep: 1510.81 | bwd_allreduce_microstep: 11.93 | step_microstep: 112.44
[2024-12-05 14:23:21,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4628.80 | bwd: 9307.43 | bwd_inner: 9293.69 | bwd_allreduce: 12.56 | step: 113.52

 37%|███▋      | 142/384 [40:17<1:13:36, 18.25s/it]
                                                   
{'loss': 0.1738, 'grad_norm': 1.6480506658554077, 'learning_rate': 7.277473933453169e-07, 'epoch': 0.74}

 37%|███▋      | 142/384 [40:17<1:13:36, 18.25s/it][2024-12-05 14:23:22,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.86 | bwd_microstep: 380.62 | bwd_inner_microstep: 380.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:23:23,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.56 | bwd_microstep: 524.31 | bwd_inner_microstep: 524.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:23:24,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.26 | bwd_microstep: 346.63 | bwd_inner_microstep: 346.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:23:25,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 349.67 | bwd_microstep: 703.74 | bwd_inner_microstep: 703.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:23:26,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.73 | bwd_microstep: 401.98 | bwd_inner_microstep: 401.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:23:27,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.17 | bwd_microstep: 525.86 | bwd_inner_microstep: 525.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:23:28,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.20 | bwd_microstep: 488.71 | bwd_inner_microstep: 488.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:23:29,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.21 | bwd_microstep: 401.54 | bwd_inner_microstep: 401.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:23:30,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.30 | bwd_microstep: 357.15 | bwd_inner_microstep: 357.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:23:31,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 362.03 | bwd_microstep: 746.35 | bwd_inner_microstep: 746.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:23:33,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 373.23 | bwd_microstep: 786.54 | bwd_inner_microstep: 786.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:23:34,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.78 | bwd_microstep: 452.16 | bwd_inner_microstep: 452.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:23:35,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.94 | bwd_microstep: 427.19 | bwd_inner_microstep: 427.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:23:36,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.98 | bwd_microstep: 414.54 | bwd_inner_microstep: 414.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:23:36,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.51 | bwd_microstep: 473.60 | bwd_inner_microstep: 473.48 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:23:39,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.27 | optimizer_gradients: 14.07 | optimizer_step: 23.14
[2024-12-05 14:23:39,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.53 | bwd_microstep: 1618.67 | bwd_inner_microstep: 1606.68 | bwd_allreduce_microstep: 11.89 | step_microstep: 112.08
[2024-12-05 14:23:39,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4539.04 | bwd: 9049.58 | bwd_inner: 9036.04 | bwd_allreduce: 12.49 | step: 113.03

 37%|███▋      | 143/384 [40:35<1:13:20, 18.26s/it]
                                                   
{'loss': 0.1558, 'grad_norm': 1.5904552936553955, 'learning_rate': 7.239802216958522e-07, 'epoch': 0.74}

 37%|███▋      | 143/384 [40:35<1:13:20, 18.26s/it][2024-12-05 14:23:40,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.71 | bwd_microstep: 502.12 | bwd_inner_microstep: 501.99 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:23:41,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.14 | bwd_microstep: 563.21 | bwd_inner_microstep: 563.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:23:42,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.74 | bwd_microstep: 448.78 | bwd_inner_microstep: 448.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:23:44,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 393.43 | bwd_microstep: 867.38 | bwd_inner_microstep: 867.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:23:45,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.45 | bwd_microstep: 465.68 | bwd_inner_microstep: 465.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:23:46,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.21 | bwd_microstep: 494.55 | bwd_inner_microstep: 494.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:23:47,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.37 | bwd_microstep: 551.54 | bwd_inner_microstep: 551.40 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:23:48,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.15 | bwd_microstep: 402.51 | bwd_inner_microstep: 402.32 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 14:23:49,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.62 | bwd_microstep: 410.96 | bwd_inner_microstep: 410.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:23:50,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.23 | bwd_microstep: 439.99 | bwd_inner_microstep: 439.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:23:51,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.01 | bwd_microstep: 449.16 | bwd_inner_microstep: 449.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:23:52,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.40 | bwd_microstep: 451.15 | bwd_inner_microstep: 451.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:23:53,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.50 | bwd_microstep: 559.49 | bwd_inner_microstep: 559.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:23:54,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.21 | bwd_microstep: 435.81 | bwd_inner_microstep: 435.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:23:55,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.51 | bwd_microstep: 482.23 | bwd_inner_microstep: 482.10 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:23:57,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.49 | optimizer_gradients: 14.13 | optimizer_step: 23.28
[2024-12-05 14:23:57,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.91 | bwd_microstep: 1866.05 | bwd_inner_microstep: 1853.68 | bwd_allreduce_microstep: 12.25 | step_microstep: 112.54
[2024-12-05 14:23:57,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4602.67 | bwd: 9390.59 | bwd_inner: 9376.53 | bwd_allreduce: 12.92 | step: 113.70

 38%|███▊      | 144/384 [40:54<1:13:27, 18.36s/it]
                                                   
{'loss': 0.1661, 'grad_norm': 1.6988767385482788, 'learning_rate': 7.201970757788171e-07, 'epoch': 0.75}

 38%|███▊      | 144/384 [40:54<1:13:27, 18.36s/it][2024-12-05 14:23:58,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.79 | bwd_microstep: 503.25 | bwd_inner_microstep: 503.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:24:00,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.62 | bwd_microstep: 448.87 | bwd_inner_microstep: 448.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:24:00,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.90 | bwd_microstep: 462.93 | bwd_inner_microstep: 462.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:24:02,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.55 | bwd_microstep: 509.35 | bwd_inner_microstep: 509.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:24:03,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.95 | bwd_microstep: 612.76 | bwd_inner_microstep: 612.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:24:04,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.75 | bwd_microstep: 448.03 | bwd_inner_microstep: 447.88 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:24:05,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.56 | bwd_microstep: 521.52 | bwd_inner_microstep: 521.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:24:06,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.18 | bwd_microstep: 447.28 | bwd_inner_microstep: 447.14 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 14:24:07,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.38 | bwd_microstep: 441.29 | bwd_inner_microstep: 441.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:24:08,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.26 | bwd_microstep: 453.85 | bwd_inner_microstep: 453.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:24:09,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.98 | bwd_microstep: 592.27 | bwd_inner_microstep: 592.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:24:10,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.77 | bwd_microstep: 435.95 | bwd_inner_microstep: 435.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:24:11,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.33 | bwd_microstep: 602.93 | bwd_inner_microstep: 602.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:24:12,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.78 | bwd_microstep: 441.36 | bwd_inner_microstep: 441.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:24:13,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.47 | bwd_microstep: 433.59 | bwd_inner_microstep: 433.39 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08
[2024-12-05 14:24:15,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.42 | optimizer_gradients: 14.16 | optimizer_step: 23.64
[2024-12-05 14:24:15,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 229.07 | bwd_microstep: 1122.56 | bwd_inner_microstep: 1110.27 | bwd_allreduce_microstep: 12.18 | step_microstep: 114.63
[2024-12-05 14:24:15,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4455.41 | bwd: 8477.78 | bwd_inner: 8463.72 | bwd_allreduce: 12.86 | step: 115.83

 38%|███▊      | 145/384 [41:11<1:12:19, 18.16s/it]
                                                   
{'loss': 0.1338, 'grad_norm': 1.5458639860153198, 'learning_rate': 7.163982254081474e-07, 'epoch': 0.75}

 38%|███▊      | 145/384 [41:11<1:12:19, 18.16s/it][2024-12-05 14:24:16,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.80 | bwd_microstep: 416.04 | bwd_inner_microstep: 415.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:24:17,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 384.47 | bwd_microstep: 840.10 | bwd_inner_microstep: 839.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:24:18,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.05 | bwd_microstep: 426.32 | bwd_inner_microstep: 426.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:24:19,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.57 | bwd_microstep: 353.73 | bwd_inner_microstep: 353.59 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:24:20,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.84 | bwd_microstep: 440.32 | bwd_inner_microstep: 440.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:24:21,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.63 | bwd_microstep: 403.77 | bwd_inner_microstep: 403.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:24:22,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.57 | bwd_microstep: 402.67 | bwd_inner_microstep: 402.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:24:23,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.55 | bwd_microstep: 425.39 | bwd_inner_microstep: 425.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:24:24,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.89 | bwd_microstep: 411.19 | bwd_inner_microstep: 411.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:24:25,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.46 | bwd_microstep: 414.15 | bwd_inner_microstep: 414.03 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:24:26,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.82 | bwd_microstep: 456.68 | bwd_inner_microstep: 456.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:24:27,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.14 | bwd_microstep: 455.66 | bwd_inner_microstep: 455.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:24:28,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.48 | bwd_microstep: 543.20 | bwd_inner_microstep: 543.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:24:29,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.33 | bwd_microstep: 427.94 | bwd_inner_microstep: 427.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:24:30,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.10 | bwd_microstep: 486.89 | bwd_inner_microstep: 486.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:24:33,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.37 | optimizer_gradients: 14.07 | optimizer_step: 23.28
[2024-12-05 14:24:33,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.44 | bwd_microstep: 2535.09 | bwd_inner_microstep: 2522.93 | bwd_allreduce_microstep: 12.06 | step_microstep: 113.58
[2024-12-05 14:24:33,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4376.20 | bwd: 9439.12 | bwd_inner: 9425.31 | bwd_allreduce: 12.70 | step: 114.65

 38%|███▊      | 146/384 [41:29<1:11:45, 18.09s/it]
                                                   
{'loss': 0.1405, 'grad_norm': 1.5240118503570557, 'learning_rate': 7.125839415178203e-07, 'epoch': 0.76}

 38%|███▊      | 146/384 [41:29<1:11:45, 18.09s/it][2024-12-05 14:24:34,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.71 | bwd_microstep: 592.90 | bwd_inner_microstep: 592.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:24:35,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.85 | bwd_microstep: 382.18 | bwd_inner_microstep: 382.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:24:36,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.47 | bwd_microstep: 689.44 | bwd_inner_microstep: 689.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:24:37,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.84 | bwd_microstep: 525.73 | bwd_inner_microstep: 525.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:24:38,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.59 | bwd_microstep: 357.27 | bwd_inner_microstep: 357.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:24:39,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.84 | bwd_microstep: 588.05 | bwd_inner_microstep: 587.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:24:41,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.69 | bwd_microstep: 542.75 | bwd_inner_microstep: 542.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:24:41,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.14 | bwd_microstep: 412.10 | bwd_inner_microstep: 411.92 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:24:43,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.97 | bwd_microstep: 586.01 | bwd_inner_microstep: 585.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:24:44,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.87 | bwd_microstep: 426.29 | bwd_inner_microstep: 426.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:24:45,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.76 | bwd_microstep: 616.29 | bwd_inner_microstep: 616.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:24:46,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.49 | bwd_microstep: 405.96 | bwd_inner_microstep: 405.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:24:47,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.27 | bwd_microstep: 567.26 | bwd_inner_microstep: 567.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:24:48,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.85 | bwd_microstep: 456.34 | bwd_inner_microstep: 456.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:24:49,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.91 | bwd_microstep: 407.85 | bwd_inner_microstep: 407.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:24:50,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.00 | optimizer_gradients: 14.07 | optimizer_step: 23.13
[2024-12-05 14:24:50,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 349.60 | bwd_microstep: 944.58 | bwd_inner_microstep: 932.36 | bwd_allreduce_microstep: 12.12 | step_microstep: 114.76
[2024-12-05 14:24:50,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4621.96 | bwd: 8500.97 | bwd_inner: 8487.04 | bwd_allreduce: 12.76 | step: 115.86

 38%|███▊      | 147/384 [41:47<1:10:31, 17.85s/it]
                                                   
{'loss': 0.1682, 'grad_norm': 1.7464412450790405, 'learning_rate': 7.087544961425316e-07, 'epoch': 0.76}

 38%|███▊      | 147/384 [41:47<1:10:31, 17.85s/it][2024-12-05 14:24:52,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.14 | bwd_microstep: 424.14 | bwd_inner_microstep: 424.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:24:53,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.06 | bwd_microstep: 593.85 | bwd_inner_microstep: 593.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:24:54,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 356.31 | bwd_microstep: 747.46 | bwd_inner_microstep: 747.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:24:55,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.46 | bwd_microstep: 400.89 | bwd_inner_microstep: 400.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:24:56,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.75 | bwd_microstep: 435.56 | bwd_inner_microstep: 435.44 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:24:57,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.49 | bwd_microstep: 455.92 | bwd_inner_microstep: 455.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:24:58,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.34 | bwd_microstep: 384.02 | bwd_inner_microstep: 383.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:24:59,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.17 | bwd_microstep: 442.86 | bwd_inner_microstep: 442.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:25:00,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.37 | bwd_microstep: 406.32 | bwd_inner_microstep: 406.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:25:01,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.73 | bwd_microstep: 479.72 | bwd_inner_microstep: 479.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:25:02,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.21 | bwd_microstep: 480.19 | bwd_inner_microstep: 480.05 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:25:04,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 402.97 | bwd_microstep: 908.22 | bwd_inner_microstep: 908.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:25:05,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 345.36 | bwd_microstep: 715.54 | bwd_inner_microstep: 715.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:25:06,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.45 | bwd_microstep: 563.12 | bwd_inner_microstep: 562.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:25:08,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.86 | bwd_microstep: 462.03 | bwd_inner_microstep: 461.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:25:09,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.53 | optimizer_gradients: 14.00 | optimizer_step: 23.22
[2024-12-05 14:25:09,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.67 | bwd_microstep: 798.33 | bwd_inner_microstep: 786.40 | bwd_allreduce_microstep: 11.84 | step_microstep: 112.23
[2024-12-05 14:25:09,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4656.41 | bwd: 8698.19 | bwd_inner: 8684.59 | bwd_allreduce: 12.50 | step: 113.49

 39%|███▊      | 148/384 [42:05<1:11:07, 18.08s/it]
                                                   
{'loss': 0.1839, 'grad_norm': 1.7425211668014526, 'learning_rate': 7.049101623982937e-07, 'epoch': 0.77}

 39%|███▊      | 148/384 [42:05<1:11:07, 18.08s/it][2024-12-05 14:25:10,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.40 | bwd_microstep: 442.45 | bwd_inner_microstep: 442.32 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:25:11,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.48 | bwd_microstep: 448.34 | bwd_inner_microstep: 448.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:25:12,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 348.85 | bwd_microstep: 708.07 | bwd_inner_microstep: 707.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:25:14,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.78 | bwd_microstep: 685.81 | bwd_inner_microstep: 685.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:25:15,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.50 | bwd_microstep: 402.40 | bwd_inner_microstep: 402.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:25:16,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 242.78 | bwd_microstep: 371.40 | bwd_inner_microstep: 371.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:25:16,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.01 | bwd_microstep: 413.29 | bwd_inner_microstep: 413.14 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:25:17,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.06 | bwd_microstep: 535.68 | bwd_inner_microstep: 535.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:25:18,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.41 | bwd_microstep: 451.54 | bwd_inner_microstep: 451.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:25:19,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.26 | bwd_microstep: 451.75 | bwd_inner_microstep: 451.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:25:20,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.46 | bwd_microstep: 483.25 | bwd_inner_microstep: 483.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:25:21,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.12 | bwd_microstep: 540.50 | bwd_inner_microstep: 540.37 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:25:22,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.95 | bwd_microstep: 522.39 | bwd_inner_microstep: 522.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:25:23,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 360.17 | bwd_microstep: 741.93 | bwd_inner_microstep: 741.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:25:24,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.88 | bwd_microstep: 352.06 | bwd_inner_microstep: 351.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:25:26,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.62 | optimizer_gradients: 14.12 | optimizer_step: 23.18
[2024-12-05 14:25:26,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.51 | bwd_microstep: 1259.53 | bwd_inner_microstep: 1247.43 | bwd_allreduce_microstep: 12.01 | step_microstep: 112.77
[2024-12-05 14:25:26,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4586.69 | bwd: 8810.38 | bwd_inner: 8796.63 | bwd_allreduce: 12.65 | step: 113.95

 39%|███▉      | 149/384 [42:22<1:09:34, 17.77s/it]
                                                   
{'loss': 0.1124, 'grad_norm': 1.4918522834777832, 'learning_rate': 7.010512144629579e-07, 'epoch': 0.77}

 39%|███▉      | 149/384 [42:22<1:09:34, 17.77s/it][2024-12-05 14:25:27,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.33 | bwd_microstep: 403.16 | bwd_inner_microstep: 403.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:25:28,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.79 | bwd_microstep: 518.18 | bwd_inner_microstep: 518.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:25:29,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.02 | bwd_microstep: 478.26 | bwd_inner_microstep: 478.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:25:30,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.63 | bwd_microstep: 413.90 | bwd_inner_microstep: 413.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:25:31,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.97 | bwd_microstep: 453.35 | bwd_inner_microstep: 453.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:25:32,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.18 | bwd_microstep: 418.01 | bwd_inner_microstep: 417.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:25:33,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.86 | bwd_microstep: 447.31 | bwd_inner_microstep: 447.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:25:34,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.43 | bwd_microstep: 409.69 | bwd_inner_microstep: 409.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:25:34,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.08 | bwd_microstep: 456.70 | bwd_inner_microstep: 456.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:25:35,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.02 | bwd_microstep: 526.13 | bwd_inner_microstep: 526.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:25:36,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.07 | bwd_microstep: 480.87 | bwd_inner_microstep: 480.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:25:37,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.60 | bwd_microstep: 482.50 | bwd_inner_microstep: 482.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:25:38,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.90 | bwd_microstep: 450.75 | bwd_inner_microstep: 450.60 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:25:39,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.04 | bwd_microstep: 483.67 | bwd_inner_microstep: 483.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:25:40,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.84 | bwd_microstep: 428.62 | bwd_inner_microstep: 428.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:25:43,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.51 | optimizer_gradients: 14.08 | optimizer_step: 23.22
[2024-12-05 14:25:43,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.84 | bwd_microstep: 2093.37 | bwd_inner_microstep: 2081.33 | bwd_allreduce_microstep: 11.94 | step_microstep: 112.36
[2024-12-05 14:25:43,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4331.67 | bwd: 8944.46 | bwd_inner: 8930.80 | bwd_allreduce: 12.57 | step: 113.53

 39%|███▉      | 150/384 [42:39<1:08:23, 17.54s/it]
                                                   
{'loss': 0.1546, 'grad_norm': 1.6756539344787598, 'learning_rate': 6.971779275566593e-07, 'epoch': 0.78}

 39%|███▉      | 150/384 [42:39<1:08:23, 17.54s/it][2024-12-05 14:25:44,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.48 | bwd_microstep: 399.32 | bwd_inner_microstep: 399.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:25:45,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.34 | bwd_microstep: 415.82 | bwd_inner_microstep: 415.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:25:46,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.52 | bwd_microstep: 401.18 | bwd_inner_microstep: 401.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:25:47,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.20 | bwd_microstep: 414.92 | bwd_inner_microstep: 414.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:25:48,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 341.95 | bwd_microstep: 722.83 | bwd_inner_microstep: 722.69 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:25:49,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.42 | bwd_microstep: 522.63 | bwd_inner_microstep: 522.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:25:50,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.45 | bwd_microstep: 442.08 | bwd_inner_microstep: 441.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:25:51,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.47 | bwd_microstep: 398.74 | bwd_inner_microstep: 398.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:25:52,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 361.73 | bwd_microstep: 741.39 | bwd_inner_microstep: 741.23 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:25:54,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 346.85 | bwd_microstep: 704.02 | bwd_inner_microstep: 703.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:25:54,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.95 | bwd_microstep: 480.17 | bwd_inner_microstep: 480.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:25:55,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.19 | bwd_microstep: 450.11 | bwd_inner_microstep: 449.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:25:56,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.10 | bwd_microstep: 534.67 | bwd_inner_microstep: 534.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:25:58,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 346.28 | bwd_microstep: 705.37 | bwd_inner_microstep: 705.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:25:59,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.61 | bwd_microstep: 380.13 | bwd_inner_microstep: 380.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:26:01,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.12 | optimizer_gradients: 14.04 | optimizer_step: 23.14
[2024-12-05 14:26:01,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.88 | bwd_microstep: 2456.51 | bwd_inner_microstep: 2444.31 | bwd_allreduce_microstep: 12.10 | step_microstep: 111.73
[2024-12-05 14:26:01,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4570.51 | bwd: 10169.90 | bwd_inner: 10156.09 | bwd_allreduce: 12.73 | step: 112.85

 39%|███▉      | 151/384 [42:58<1:09:02, 17.78s/it]
                                                   
{'loss': 0.1313, 'grad_norm': 1.5222876071929932, 'learning_rate': 6.93290577922188e-07, 'epoch': 0.78}

 39%|███▉      | 151/384 [42:58<1:09:02, 17.78s/it][2024-12-05 14:26:02,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.89 | bwd_microstep: 325.22 | bwd_inner_microstep: 325.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:26:03,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.37 | bwd_microstep: 537.89 | bwd_inner_microstep: 537.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:26:05,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.81 | bwd_microstep: 598.94 | bwd_inner_microstep: 598.79 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08
[2024-12-05 14:26:06,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.15 | bwd_microstep: 436.85 | bwd_inner_microstep: 436.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:26:07,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.36 | bwd_microstep: 344.44 | bwd_inner_microstep: 344.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:26:07,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.95 | bwd_microstep: 463.40 | bwd_inner_microstep: 463.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:26:08,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.56 | bwd_microstep: 407.00 | bwd_inner_microstep: 406.85 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:26:09,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.50 | bwd_microstep: 459.95 | bwd_inner_microstep: 459.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:26:10,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.50 | bwd_microstep: 412.08 | bwd_inner_microstep: 411.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:26:11,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.79 | bwd_microstep: 500.14 | bwd_inner_microstep: 500.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:26:12,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.63 | bwd_microstep: 448.40 | bwd_inner_microstep: 448.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:26:13,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.13 | bwd_microstep: 431.77 | bwd_inner_microstep: 431.63 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:26:14,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.47 | bwd_microstep: 403.06 | bwd_inner_microstep: 402.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:26:15,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.30 | bwd_microstep: 495.20 | bwd_inner_microstep: 495.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:26:17,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 342.96 | bwd_microstep: 689.58 | bwd_inner_microstep: 689.43 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 14:26:20,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.38 | optimizer_gradients: 14.14 | optimizer_step: 23.19
[2024-12-05 14:26:20,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.61 | bwd_microstep: 2715.60 | bwd_inner_microstep: 2703.24 | bwd_allreduce_microstep: 12.25 | step_microstep: 112.53
[2024-12-05 14:26:20,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4366.07 | bwd: 9669.51 | bwd_inner: 9655.46 | bwd_allreduce: 12.94 | step: 113.60

 40%|███▉      | 152/384 [43:16<1:09:38, 18.01s/it]
                                                   
{'loss': 0.1349, 'grad_norm': 1.548453688621521, 'learning_rate': 6.89389442805288e-07, 'epoch': 0.79}

 40%|███▉      | 152/384 [43:16<1:09:38, 18.01s/it][2024-12-05 14:26:21,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.01 | bwd_microstep: 573.84 | bwd_inner_microstep: 573.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:26:22,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.44 | bwd_microstep: 533.02 | bwd_inner_microstep: 532.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:26:23,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.91 | bwd_microstep: 429.90 | bwd_inner_microstep: 429.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:26:25,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.30 | bwd_microstep: 684.07 | bwd_inner_microstep: 683.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:26:25,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.80 | bwd_microstep: 403.99 | bwd_inner_microstep: 403.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:26:26,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.40 | bwd_microstep: 494.58 | bwd_inner_microstep: 494.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:26:27,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.40 | bwd_microstep: 348.62 | bwd_inner_microstep: 348.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:26:28,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.34 | bwd_microstep: 482.74 | bwd_inner_microstep: 482.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:26:29,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.35 | bwd_microstep: 439.72 | bwd_inner_microstep: 439.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:26:30,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.38 | bwd_microstep: 459.32 | bwd_inner_microstep: 459.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:26:32,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 361.87 | bwd_microstep: 748.91 | bwd_inner_microstep: 748.76 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:26:33,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.20 | bwd_microstep: 474.61 | bwd_inner_microstep: 474.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:26:34,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.05 | bwd_microstep: 352.46 | bwd_inner_microstep: 352.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:26:35,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.42 | bwd_microstep: 398.47 | bwd_inner_microstep: 398.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:26:35,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.67 | bwd_microstep: 430.62 | bwd_inner_microstep: 430.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:26:37,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.69 | optimizer_gradients: 13.99 | optimizer_step: 23.15
[2024-12-05 14:26:37,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.21 | bwd_microstep: 1276.56 | bwd_inner_microstep: 1264.63 | bwd_allreduce_microstep: 11.85 | step_microstep: 114.07
[2024-12-05 14:26:37,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4440.80 | bwd: 8531.44 | bwd_inner: 8517.85 | bwd_allreduce: 12.48 | step: 115.16

 40%|███▉      | 153/384 [43:34<1:08:45, 17.86s/it]
                                                   
{'loss': 0.1568, 'grad_norm': 1.6898419857025146, 'learning_rate': 6.85474800434884e-07, 'epoch': 0.8}

 40%|███▉      | 153/384 [43:34<1:08:45, 17.86s/it][2024-12-05 14:26:39,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.70 | bwd_microstep: 523.72 | bwd_inner_microstep: 523.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:26:40,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.48 | bwd_microstep: 563.41 | bwd_inner_microstep: 563.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:26:40,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.70 | bwd_microstep: 406.61 | bwd_inner_microstep: 406.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:26:41,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.05 | bwd_microstep: 491.32 | bwd_inner_microstep: 491.18 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:26:42,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.19 | bwd_microstep: 553.53 | bwd_inner_microstep: 553.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:26:43,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.83 | bwd_microstep: 447.05 | bwd_inner_microstep: 446.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:26:44,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.87 | bwd_microstep: 455.37 | bwd_inner_microstep: 455.21 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:26:45,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.40 | bwd_microstep: 457.11 | bwd_inner_microstep: 456.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:26:46,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.47 | bwd_microstep: 462.79 | bwd_inner_microstep: 462.65 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:26:47,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.02 | bwd_microstep: 521.33 | bwd_inner_microstep: 521.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:26:48,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.09 | bwd_microstep: 438.74 | bwd_inner_microstep: 438.61 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:26:49,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.77 | bwd_microstep: 447.75 | bwd_inner_microstep: 447.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:26:50,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.31 | bwd_microstep: 497.28 | bwd_inner_microstep: 497.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:26:51,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.54 | bwd_microstep: 626.32 | bwd_inner_microstep: 626.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:26:52,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.89 | bwd_microstep: 413.20 | bwd_inner_microstep: 413.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:26:54,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.00 | optimizer_gradients: 14.12 | optimizer_step: 23.17
[2024-12-05 14:26:54,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.83 | bwd_microstep: 1315.90 | bwd_inner_microstep: 1303.67 | bwd_allreduce_microstep: 12.13 | step_microstep: 114.65
[2024-12-05 14:26:54,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4484.18 | bwd: 8621.43 | bwd_inner: 8607.53 | bwd_allreduce: 12.77 | step: 115.79

 40%|████      | 154/384 [43:51<1:07:19, 17.56s/it]
                                                   
{'loss': 0.161, 'grad_norm': 1.8794304132461548, 'learning_rate': 6.815469300032373e-07, 'epoch': 0.8}

 40%|████      | 154/384 [43:51<1:07:19, 17.56s/it][2024-12-05 14:26:55,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.80 | bwd_microstep: 556.30 | bwd_inner_microstep: 556.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:26:57,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.20 | bwd_microstep: 533.09 | bwd_inner_microstep: 532.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:26:58,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.45 | bwd_microstep: 426.26 | bwd_inner_microstep: 426.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:26:59,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 388.52 | bwd_microstep: 851.16 | bwd_inner_microstep: 851.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:27:00,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 356.40 | bwd_microstep: 741.49 | bwd_inner_microstep: 741.36 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:27:02,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.34 | bwd_microstep: 523.42 | bwd_inner_microstep: 523.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:27:02,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.24 | bwd_microstep: 398.50 | bwd_inner_microstep: 398.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:27:03,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.50 | bwd_microstep: 443.47 | bwd_inner_microstep: 443.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:27:04,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.45 | bwd_microstep: 402.68 | bwd_inner_microstep: 402.52 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:27:05,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.42 | bwd_microstep: 539.03 | bwd_inner_microstep: 538.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:27:06,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.73 | bwd_microstep: 429.08 | bwd_inner_microstep: 428.91 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 14:27:07,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.88 | bwd_microstep: 410.64 | bwd_inner_microstep: 410.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:27:08,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.14 | bwd_microstep: 417.63 | bwd_inner_microstep: 417.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:27:09,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.19 | bwd_microstep: 450.31 | bwd_inner_microstep: 450.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:27:10,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.62 | bwd_microstep: 409.93 | bwd_inner_microstep: 409.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:27:13,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.89 | optimizer_gradients: 14.24 | optimizer_step: 23.30
[2024-12-05 14:27:13,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.61 | bwd_microstep: 1989.09 | bwd_inner_microstep: 1976.53 | bwd_allreduce_microstep: 12.45 | step_microstep: 115.09
[2024-12-05 14:27:13,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4630.53 | bwd: 9522.06 | bwd_inner: 9507.82 | bwd_allreduce: 13.10 | step: 116.28

 40%|████      | 155/384 [44:09<1:07:47, 17.76s/it]
                                                   
{'loss': 0.1615, 'grad_norm': 1.6816418170928955, 'learning_rate': 6.776061116460352e-07, 'epoch': 0.81}

 40%|████      | 155/384 [44:09<1:07:47, 17.76s/it][2024-12-05 14:27:13,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.75 | bwd_microstep: 382.86 | bwd_inner_microstep: 382.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:27:14,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.67 | bwd_microstep: 436.39 | bwd_inner_microstep: 436.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:27:15,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.96 | bwd_microstep: 486.38 | bwd_inner_microstep: 486.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:27:16,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.69 | bwd_microstep: 523.39 | bwd_inner_microstep: 523.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:27:17,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.33 | bwd_microstep: 611.04 | bwd_inner_microstep: 610.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:27:19,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.25 | bwd_microstep: 458.39 | bwd_inner_microstep: 458.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:27:20,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.09 | bwd_microstep: 459.00 | bwd_inner_microstep: 458.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:27:21,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.69 | bwd_microstep: 477.77 | bwd_inner_microstep: 477.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:27:22,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.64 | bwd_microstep: 437.78 | bwd_inner_microstep: 437.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:27:23,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.89 | bwd_microstep: 522.94 | bwd_inner_microstep: 522.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:27:24,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.68 | bwd_microstep: 482.27 | bwd_inner_microstep: 482.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:27:25,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.35 | bwd_microstep: 413.41 | bwd_inner_microstep: 413.28 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:27:26,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.92 | bwd_microstep: 439.03 | bwd_inner_microstep: 438.89 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:27:27,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.83 | bwd_microstep: 484.86 | bwd_inner_microstep: 484.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:27:28,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.08 | bwd_microstep: 682.31 | bwd_inner_microstep: 682.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:27:31,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.51 | optimizer_gradients: 14.02 | optimizer_step: 23.07
[2024-12-05 14:27:31,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.79 | bwd_microstep: 1703.06 | bwd_inner_microstep: 1691.12 | bwd_allreduce_microstep: 11.84 | step_microstep: 112.18
[2024-12-05 14:27:31,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4484.68 | bwd: 9000.86 | bwd_inner: 8987.28 | bwd_allreduce: 12.48 | step: 113.42

 41%|████      | 156/384 [44:27<1:07:51, 17.86s/it]
                                                   
{'loss': 0.1948, 'grad_norm': 1.960444688796997, 'learning_rate': 6.7365262642241e-07, 'epoch': 0.81}

 41%|████      | 156/384 [44:27<1:07:51, 17.86s/it][2024-12-05 14:27:32,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.22 | bwd_microstep: 423.31 | bwd_inner_microstep: 423.14 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:27:32,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.19 | bwd_microstep: 535.27 | bwd_inner_microstep: 535.14 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.06
[2024-12-05 14:27:34,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.92 | bwd_microstep: 524.45 | bwd_inner_microstep: 524.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:27:34,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.18 | bwd_microstep: 454.98 | bwd_inner_microstep: 454.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:27:35,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.57 | bwd_microstep: 442.77 | bwd_inner_microstep: 442.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:27:36,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.08 | bwd_microstep: 457.49 | bwd_inner_microstep: 457.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:27:37,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.68 | bwd_microstep: 585.64 | bwd_inner_microstep: 585.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:27:38,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.31 | bwd_microstep: 483.92 | bwd_inner_microstep: 483.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:27:39,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.61 | bwd_microstep: 478.60 | bwd_inner_microstep: 478.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:27:40,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.29 | bwd_microstep: 484.31 | bwd_inner_microstep: 484.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:27:41,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.43 | bwd_microstep: 442.52 | bwd_inner_microstep: 442.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:27:42,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.88 | bwd_microstep: 458.30 | bwd_inner_microstep: 458.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:27:44,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.81 | bwd_microstep: 584.80 | bwd_inner_microstep: 584.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:27:45,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.92 | bwd_microstep: 461.08 | bwd_inner_microstep: 460.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:27:46,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.12 | bwd_microstep: 414.42 | bwd_inner_microstep: 414.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:27:48,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.81 | optimizer_gradients: 14.06 | optimizer_step: 23.05
[2024-12-05 14:27:48,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.76 | bwd_microstep: 1990.78 | bwd_inner_microstep: 1978.63 | bwd_allreduce_microstep: 12.05 | step_microstep: 112.42
[2024-12-05 14:27:48,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4398.01 | bwd: 9222.63 | bwd_inner: 9208.82 | bwd_allreduce: 12.67 | step: 113.55

 41%|████      | 157/384 [44:45<1:07:24, 17.82s/it]
                                                   
{'loss': 0.161, 'grad_norm': 1.6450730562210083, 'learning_rate': 6.696867562948962e-07, 'epoch': 0.82}

 41%|████      | 157/384 [44:45<1:07:24, 17.82s/it][2024-12-05 14:27:49,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.01 | bwd_microstep: 404.21 | bwd_inner_microstep: 404.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:27:51,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.52 | bwd_microstep: 437.30 | bwd_inner_microstep: 437.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:27:52,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 348.12 | bwd_microstep: 676.71 | bwd_inner_microstep: 676.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:27:53,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.99 | bwd_microstep: 431.39 | bwd_inner_microstep: 431.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:27:54,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 332.57 | bwd_microstep: 668.74 | bwd_inner_microstep: 668.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:27:55,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.31 | bwd_microstep: 422.77 | bwd_inner_microstep: 422.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:27:56,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.97 | bwd_microstep: 586.99 | bwd_inner_microstep: 586.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:27:57,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.30 | bwd_microstep: 444.96 | bwd_inner_microstep: 444.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:27:58,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.08 | bwd_microstep: 621.74 | bwd_inner_microstep: 621.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:27:59,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.81 | bwd_microstep: 613.82 | bwd_inner_microstep: 613.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:28:00,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.75 | bwd_microstep: 603.75 | bwd_inner_microstep: 603.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:28:01,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.93 | bwd_microstep: 430.35 | bwd_inner_microstep: 430.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:28:03,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 385.66 | bwd_microstep: 839.44 | bwd_inner_microstep: 839.33 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:28:04,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.53 | bwd_microstep: 408.65 | bwd_inner_microstep: 408.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:28:05,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.56 | bwd_microstep: 587.14 | bwd_inner_microstep: 586.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:28:06,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.64 | optimizer_gradients: 14.08 | optimizer_step: 23.28
[2024-12-05 14:28:06,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.52 | bwd_microstep: 986.11 | bwd_inner_microstep: 973.93 | bwd_allreduce_microstep: 12.07 | step_microstep: 112.73
[2024-12-05 14:28:06,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4831.70 | bwd: 9164.07 | bwd_inner: 9150.23 | bwd_allreduce: 12.70 | step: 113.86

 41%|████      | 158/384 [45:03<1:07:20, 17.88s/it]
                                                   
{'loss': 0.1476, 'grad_norm': 1.4993230104446411, 'learning_rate': 6.657087841093179e-07, 'epoch': 0.82}

 41%|████      | 158/384 [45:03<1:07:20, 17.88s/it][2024-12-05 14:28:07,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.11 | bwd_microstep: 406.31 | bwd_inner_microstep: 406.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:28:09,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.42 | bwd_microstep: 588.71 | bwd_inner_microstep: 588.55 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09
[2024-12-05 14:28:10,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.77 | bwd_microstep: 614.60 | bwd_inner_microstep: 614.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:28:11,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.00 | bwd_microstep: 688.28 | bwd_inner_microstep: 688.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:28:12,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.79 | bwd_microstep: 503.94 | bwd_inner_microstep: 503.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:28:14,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.30 | bwd_microstep: 593.87 | bwd_inner_microstep: 593.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:28:15,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.46 | bwd_microstep: 456.78 | bwd_inner_microstep: 456.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:28:16,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.09 | bwd_microstep: 676.77 | bwd_inner_microstep: 676.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:28:17,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.44 | bwd_microstep: 393.27 | bwd_inner_microstep: 393.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:28:18,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.37 | bwd_microstep: 487.65 | bwd_inner_microstep: 487.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:28:19,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.23 | bwd_microstep: 446.37 | bwd_inner_microstep: 446.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:28:20,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.43 | bwd_microstep: 411.43 | bwd_inner_microstep: 411.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:28:21,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.17 | bwd_microstep: 460.47 | bwd_inner_microstep: 460.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:28:22,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.05 | bwd_microstep: 497.85 | bwd_inner_microstep: 497.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:28:23,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.42 | bwd_microstep: 570.94 | bwd_inner_microstep: 570.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:28:25,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.00 | optimizer_gradients: 14.02 | optimizer_step: 23.34
[2024-12-05 14:28:25,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.96 | bwd_microstep: 791.10 | bwd_inner_microstep: 779.13 | bwd_allreduce_microstep: 11.87 | step_microstep: 111.65
[2024-12-05 14:28:25,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4599.10 | bwd: 8588.32 | bwd_inner: 8574.71 | bwd_allreduce: 12.52 | step: 112.85

 41%|████▏     | 159/384 [45:21<1:07:39, 18.04s/it]
                                                   
{'loss': 0.1599, 'grad_norm': 1.856066346168518, 'learning_rate': 6.61718993574619e-07, 'epoch': 0.83}

 41%|████▏     | 159/384 [45:21<1:07:39, 18.04s/it][2024-12-05 14:28:26,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.36 | bwd_microstep: 342.11 | bwd_inner_microstep: 341.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:28:27,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.87 | bwd_microstep: 565.97 | bwd_inner_microstep: 565.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:28:28,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.74 | bwd_microstep: 527.32 | bwd_inner_microstep: 527.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:28:29,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.44 | bwd_microstep: 524.43 | bwd_inner_microstep: 524.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:28:30,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.61 | bwd_microstep: 438.98 | bwd_inner_microstep: 438.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:28:31,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.89 | bwd_microstep: 432.72 | bwd_inner_microstep: 432.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:28:32,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.69 | bwd_microstep: 454.37 | bwd_inner_microstep: 454.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:28:33,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.64 | bwd_microstep: 450.83 | bwd_inner_microstep: 450.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:28:34,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.31 | bwd_microstep: 675.21 | bwd_inner_microstep: 675.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:28:35,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.90 | bwd_microstep: 356.68 | bwd_inner_microstep: 356.54 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:28:36,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.26 | bwd_microstep: 422.89 | bwd_inner_microstep: 422.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:28:37,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.32 | bwd_microstep: 409.30 | bwd_inner_microstep: 409.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:28:39,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.68 | bwd_microstep: 564.32 | bwd_inner_microstep: 564.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:28:40,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.70 | bwd_microstep: 350.80 | bwd_inner_microstep: 350.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:28:41,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.89 | bwd_microstep: 481.96 | bwd_inner_microstep: 481.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:28:43,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.28 | optimizer_gradients: 14.08 | optimizer_step: 23.17
[2024-12-05 14:28:43,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.01 | bwd_microstep: 1676.13 | bwd_inner_microstep: 1663.98 | bwd_allreduce_microstep: 12.05 | step_microstep: 112.02
[2024-12-05 14:28:43,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4366.41 | bwd: 8674.02 | bwd_inner: 8660.21 | bwd_allreduce: 12.68 | step: 113.14

 42%|████▏     | 160/384 [45:39<1:07:39, 18.12s/it]
                                                   
{'loss': 0.1548, 'grad_norm': 1.6243445873260498, 'learning_rate': 6.577176692426278e-07, 'epoch': 0.83}

 42%|████▏     | 160/384 [45:39<1:07:39, 18.12s/it][2024-12-05 14:28:44,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.72 | bwd_microstep: 369.72 | bwd_inner_microstep: 369.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:28:45,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.11 | bwd_microstep: 446.31 | bwd_inner_microstep: 446.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:28:46,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.74 | bwd_microstep: 486.50 | bwd_inner_microstep: 486.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:28:47,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.43 | bwd_microstep: 431.61 | bwd_inner_microstep: 431.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:28:48,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.02 | bwd_microstep: 414.52 | bwd_inner_microstep: 414.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:28:49,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.50 | bwd_microstep: 561.18 | bwd_inner_microstep: 561.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:28:50,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.69 | bwd_microstep: 422.67 | bwd_inner_microstep: 422.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:28:51,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.93 | bwd_microstep: 669.03 | bwd_inner_microstep: 668.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:28:52,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.11 | bwd_microstep: 426.74 | bwd_inner_microstep: 426.57 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.11
[2024-12-05 14:28:53,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.18 | bwd_microstep: 437.44 | bwd_inner_microstep: 437.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:28:54,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.20 | bwd_microstep: 444.93 | bwd_inner_microstep: 444.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.05
[2024-12-05 14:28:55,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.18 | bwd_microstep: 587.44 | bwd_inner_microstep: 587.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:28:56,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.48 | bwd_microstep: 680.40 | bwd_inner_microstep: 680.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:28:57,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.37 | bwd_microstep: 614.19 | bwd_inner_microstep: 614.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:28:59,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 379.90 | bwd_microstep: 843.99 | bwd_inner_microstep: 843.75 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.16
[2024-12-05 14:29:01,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.73 | optimizer_gradients: 14.07 | optimizer_step: 23.30
[2024-12-05 14:29:01,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.84 | bwd_microstep: 1455.02 | bwd_inner_microstep: 1442.73 | bwd_allreduce_microstep: 12.20 | step_microstep: 113.07
[2024-12-05 14:29:01,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4668.48 | bwd: 9291.69 | bwd_inner: 9277.75 | bwd_allreduce: 12.88 | step: 114.35

 42%|████▏     | 161/384 [45:57<1:07:08, 18.07s/it]
                                                   
{'loss': 0.1428, 'grad_norm': 1.538219928741455, 'learning_rate': 6.537050964877625e-07, 'epoch': 0.84}

 42%|████▏     | 161/384 [45:57<1:07:08, 18.07s/it][2024-12-05 14:29:02,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.43 | bwd_microstep: 389.42 | bwd_inner_microstep: 389.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:29:03,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.71 | bwd_microstep: 425.55 | bwd_inner_microstep: 425.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:29:04,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 229.97 | bwd_microstep: 345.27 | bwd_inner_microstep: 345.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:29:05,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.74 | bwd_microstep: 548.97 | bwd_inner_microstep: 548.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:29:06,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.00 | bwd_microstep: 573.85 | bwd_inner_microstep: 573.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:29:07,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.83 | bwd_microstep: 434.67 | bwd_inner_microstep: 434.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:29:08,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.54 | bwd_microstep: 405.92 | bwd_inner_microstep: 405.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:29:09,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 330.03 | bwd_microstep: 621.30 | bwd_inner_microstep: 621.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:29:10,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.34 | bwd_microstep: 459.08 | bwd_inner_microstep: 458.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:29:11,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.99 | bwd_microstep: 420.85 | bwd_inner_microstep: 420.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:29:12,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.87 | bwd_microstep: 455.77 | bwd_inner_microstep: 455.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:29:13,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 408.06 | bwd_microstep: 924.76 | bwd_inner_microstep: 924.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:29:14,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.72 | bwd_microstep: 410.57 | bwd_inner_microstep: 410.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:29:15,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.95 | bwd_microstep: 541.20 | bwd_inner_microstep: 541.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:29:16,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.80 | bwd_microstep: 445.08 | bwd_inner_microstep: 444.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:29:18,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.18 | optimizer_gradients: 14.00 | optimizer_step: 23.34
[2024-12-05 14:29:18,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.49 | bwd_microstep: 1525.77 | bwd_inner_microstep: 1513.77 | bwd_allreduce_microstep: 11.92 | step_microstep: 111.92
[2024-12-05 14:29:18,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4490.54 | bwd: 8928.02 | bwd_inner: 8914.38 | bwd_allreduce: 12.54 | step: 113.04

 42%|████▏     | 162/384 [46:15<1:05:58, 17.83s/it]
                                                   
{'loss': 0.1205, 'grad_norm': 1.429417371749878, 'learning_rate': 6.496815614866791e-07, 'epoch': 0.84}

 42%|████▏     | 162/384 [46:15<1:05:58, 17.83s/it][2024-12-05 14:29:20,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.79 | bwd_microstep: 584.27 | bwd_inner_microstep: 584.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:29:21,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.59 | bwd_microstep: 455.19 | bwd_inner_microstep: 455.05 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:29:22,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.41 | bwd_microstep: 676.83 | bwd_inner_microstep: 676.68 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:29:23,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 238.15 | bwd_microstep: 358.03 | bwd_inner_microstep: 357.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:29:24,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.33 | bwd_microstep: 409.21 | bwd_inner_microstep: 409.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:29:25,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.67 | bwd_microstep: 461.85 | bwd_inner_microstep: 461.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:29:26,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.92 | bwd_microstep: 452.78 | bwd_inner_microstep: 452.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:29:27,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.44 | bwd_microstep: 415.65 | bwd_inner_microstep: 415.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:29:28,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.44 | bwd_microstep: 440.51 | bwd_inner_microstep: 440.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:29:29,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.35 | bwd_microstep: 557.36 | bwd_inner_microstep: 557.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:29:29,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 229.15 | bwd_microstep: 334.23 | bwd_inner_microstep: 334.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:29:30,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.65 | bwd_microstep: 560.10 | bwd_inner_microstep: 559.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:29:31,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.47 | bwd_microstep: 597.15 | bwd_inner_microstep: 597.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:29:32,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.74 | bwd_microstep: 459.79 | bwd_inner_microstep: 459.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:29:33,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.21 | bwd_microstep: 594.42 | bwd_inner_microstep: 594.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:29:36,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.53 | optimizer_gradients: 13.98 | optimizer_step: 23.29
[2024-12-05 14:29:36,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.00 | bwd_microstep: 1511.27 | bwd_inner_microstep: 1499.32 | bwd_allreduce_microstep: 11.87 | step_microstep: 112.23
[2024-12-05 14:29:36,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4500.38 | bwd: 8868.62 | bwd_inner: 8855.03 | bwd_allreduce: 12.51 | step: 113.38

 42%|████▏     | 163/384 [46:32<1:05:05, 17.67s/it]
                                                   
{'loss': 0.1903, 'grad_norm': 1.7732073068618774, 'learning_rate': 6.456473511978606e-07, 'epoch': 0.85}

 42%|████▏     | 163/384 [46:32<1:05:05, 17.67s/it][2024-12-05 14:29:37,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.68 | bwd_microstep: 564.08 | bwd_inner_microstep: 563.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:29:38,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.62 | bwd_microstep: 472.47 | bwd_inner_microstep: 472.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:29:39,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.68 | bwd_microstep: 576.82 | bwd_inner_microstep: 576.65 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09
[2024-12-05 14:29:40,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.30 | bwd_microstep: 428.34 | bwd_inner_microstep: 428.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:29:40,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.24 | bwd_microstep: 352.06 | bwd_inner_microstep: 351.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:29:42,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.09 | bwd_microstep: 636.70 | bwd_inner_microstep: 636.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:29:43,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.68 | bwd_microstep: 444.88 | bwd_inner_microstep: 444.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:29:44,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 422.37 | bwd_microstep: 916.50 | bwd_inner_microstep: 916.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:29:45,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.14 | bwd_microstep: 517.53 | bwd_inner_microstep: 517.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:29:46,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.21 | bwd_microstep: 532.09 | bwd_inner_microstep: 531.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:29:47,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.21 | bwd_microstep: 440.44 | bwd_inner_microstep: 440.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:29:48,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.90 | bwd_microstep: 409.07 | bwd_inner_microstep: 408.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:29:49,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.65 | bwd_microstep: 426.56 | bwd_inner_microstep: 426.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:29:50,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.69 | bwd_microstep: 475.58 | bwd_inner_microstep: 475.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:29:51,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.75 | bwd_microstep: 684.09 | bwd_inner_microstep: 683.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:29:53,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.47 | optimizer_gradients: 14.09 | optimizer_step: 23.16
[2024-12-05 14:29:53,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.80 | bwd_microstep: 1688.55 | bwd_inner_microstep: 1676.24 | bwd_allreduce_microstep: 12.17 | step_microstep: 112.73
[2024-12-05 14:29:53,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4763.08 | bwd: 9565.75 | bwd_inner: 9551.83 | bwd_allreduce: 12.83 | step: 113.86

 43%|████▎     | 164/384 [46:49<1:04:43, 17.65s/it]
                                                   
{'loss': 0.1571, 'grad_norm': 1.575061321258545, 'learning_rate': 6.416027533411519e-07, 'epoch': 0.85}

 43%|████▎     | 164/384 [46:49<1:04:43, 17.65s/it][2024-12-05 14:29:54,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.88 | bwd_microstep: 419.56 | bwd_inner_microstep: 419.42 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:29:55,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 325.14 | bwd_microstep: 619.06 | bwd_inner_microstep: 618.89 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:29:56,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.62 | bwd_microstep: 450.09 | bwd_inner_microstep: 449.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:29:57,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.69 | bwd_microstep: 423.74 | bwd_inner_microstep: 423.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:29:58,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.10 | bwd_microstep: 431.79 | bwd_inner_microstep: 431.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:29:59,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.74 | bwd_microstep: 432.27 | bwd_inner_microstep: 432.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:30:00,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.15 | bwd_microstep: 449.02 | bwd_inner_microstep: 448.84 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.12
[2024-12-05 14:30:01,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.58 | bwd_microstep: 485.95 | bwd_inner_microstep: 485.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:30:02,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.80 | bwd_microstep: 448.72 | bwd_inner_microstep: 448.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:30:03,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.59 | bwd_microstep: 624.93 | bwd_inner_microstep: 624.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:30:04,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.27 | bwd_microstep: 474.15 | bwd_inner_microstep: 473.96 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09
[2024-12-05 14:30:05,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.09 | bwd_microstep: 425.20 | bwd_inner_microstep: 425.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:30:05,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.02 | bwd_microstep: 389.17 | bwd_inner_microstep: 389.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:30:07,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.09 | bwd_microstep: 518.53 | bwd_inner_microstep: 518.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:30:08,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.97 | bwd_microstep: 422.83 | bwd_inner_microstep: 422.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:30:10,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.36 | optimizer_gradients: 14.02 | optimizer_step: 23.13
[2024-12-05 14:30:10,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.58 | bwd_microstep: 1753.44 | bwd_inner_microstep: 1741.24 | bwd_allreduce_microstep: 12.10 | step_microstep: 113.64
[2024-12-05 14:30:10,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4426.39 | bwd: 8768.48 | bwd_inner: 8754.44 | bwd_allreduce: 12.80 | step: 115.02

 43%|████▎     | 165/384 [47:06<1:03:38, 17.44s/it]
                                                   
{'loss': 0.1484, 'grad_norm': 1.6352499723434448, 'learning_rate': 6.375480563772389e-07, 'epoch': 0.86}

 43%|████▎     | 165/384 [47:06<1:03:38, 17.44s/it][2024-12-05 14:30:11,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.74 | bwd_microstep: 418.08 | bwd_inner_microstep: 417.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:30:12,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.77 | bwd_microstep: 421.22 | bwd_inner_microstep: 421.07 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:30:13,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.48 | bwd_microstep: 422.06 | bwd_inner_microstep: 421.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:30:14,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.01 | bwd_microstep: 454.82 | bwd_inner_microstep: 454.69 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:30:15,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.19 | bwd_microstep: 429.55 | bwd_inner_microstep: 429.41 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:30:16,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.97 | bwd_microstep: 428.86 | bwd_inner_microstep: 428.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:30:17,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.60 | bwd_microstep: 433.43 | bwd_inner_microstep: 433.26 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:30:18,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 356.67 | bwd_microstep: 731.11 | bwd_inner_microstep: 730.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:30:19,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.79 | bwd_microstep: 401.26 | bwd_inner_microstep: 401.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:30:20,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.41 | bwd_microstep: 444.24 | bwd_inner_microstep: 444.11 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:30:21,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.06 | bwd_microstep: 476.89 | bwd_inner_microstep: 476.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:30:22,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.59 | bwd_microstep: 443.17 | bwd_inner_microstep: 443.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:30:23,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.11 | bwd_microstep: 426.69 | bwd_inner_microstep: 426.50 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.12
[2024-12-05 14:30:24,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.37 | bwd_microstep: 409.76 | bwd_inner_microstep: 409.59 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 14:30:25,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.60 | bwd_microstep: 594.17 | bwd_inner_microstep: 594.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:30:28,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.28 | optimizer_gradients: 14.06 | optimizer_step: 23.34
[2024-12-05 14:30:28,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.80 | bwd_microstep: 1922.97 | bwd_inner_microstep: 1910.81 | bwd_allreduce_microstep: 12.06 | step_microstep: 112.36
[2024-12-05 14:30:28,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4459.23 | bwd: 8858.28 | bwd_inner: 8844.32 | bwd_allreduce: 12.78 | step: 113.72

 43%|████▎     | 166/384 [47:24<1:03:31, 17.48s/it]
                                                   
{'loss': 0.1735, 'grad_norm': 1.7170888185501099, 'learning_rate': 6.334835494870758e-07, 'epoch': 0.86}

 43%|████▎     | 166/384 [47:24<1:03:31, 17.48s/it][2024-12-05 14:30:29,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.44 | bwd_microstep: 374.88 | bwd_inner_microstep: 374.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:30:30,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.95 | bwd_microstep: 445.65 | bwd_inner_microstep: 445.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:30:31,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 369.27 | bwd_microstep: 764.15 | bwd_inner_microstep: 764.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:30:32,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.04 | bwd_microstep: 455.18 | bwd_inner_microstep: 455.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:30:33,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.01 | bwd_microstep: 442.88 | bwd_inner_microstep: 442.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:30:34,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.40 | bwd_microstep: 429.05 | bwd_inner_microstep: 428.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:30:35,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.52 | bwd_microstep: 490.46 | bwd_inner_microstep: 490.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:30:36,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.21 | bwd_microstep: 598.44 | bwd_inner_microstep: 598.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:30:37,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.24 | bwd_microstep: 484.88 | bwd_inner_microstep: 484.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:30:38,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.94 | bwd_microstep: 502.51 | bwd_inner_microstep: 502.35 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:30:39,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.37 | bwd_microstep: 401.88 | bwd_inner_microstep: 401.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:30:40,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.59 | bwd_microstep: 430.08 | bwd_inner_microstep: 429.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:30:41,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.15 | bwd_microstep: 436.46 | bwd_inner_microstep: 436.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:30:42,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 342.73 | bwd_microstep: 681.89 | bwd_inner_microstep: 681.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:30:43,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.74 | bwd_microstep: 491.11 | bwd_inner_microstep: 490.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:30:46,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.88 | optimizer_gradients: 14.22 | optimizer_step: 23.29
[2024-12-05 14:30:46,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.59 | bwd_microstep: 2829.11 | bwd_inner_microstep: 2816.97 | bwd_allreduce_microstep: 12.04 | step_microstep: 112.10
[2024-12-05 14:30:46,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4492.27 | bwd: 10258.58 | bwd_inner: 10244.73 | bwd_allreduce: 12.70 | step: 113.29

 43%|████▎     | 167/384 [47:43<1:04:31, 17.84s/it]
                                                   
{'loss': 0.1339, 'grad_norm': 1.5450496673583984, 'learning_rate': 6.294095225512604e-07, 'epoch': 0.87}

 43%|████▎     | 167/384 [47:43<1:04:31, 17.84s/it][2024-12-05 14:30:47,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.93 | bwd_microstep: 387.05 | bwd_inner_microstep: 386.91 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:30:48,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.63 | bwd_microstep: 405.91 | bwd_inner_microstep: 405.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:30:49,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.28 | bwd_microstep: 401.50 | bwd_inner_microstep: 401.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:30:50,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.44 | bwd_microstep: 475.24 | bwd_inner_microstep: 475.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:30:51,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.67 | bwd_microstep: 453.11 | bwd_inner_microstep: 453.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:30:52,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.39 | bwd_microstep: 423.73 | bwd_inner_microstep: 423.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:30:53,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.72 | bwd_microstep: 445.11 | bwd_inner_microstep: 444.99 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:30:54,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.10 | bwd_microstep: 427.05 | bwd_inner_microstep: 426.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:30:55,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.36 | bwd_microstep: 485.62 | bwd_inner_microstep: 485.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:30:56,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 342.87 | bwd_microstep: 657.11 | bwd_inner_microstep: 656.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:30:58,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.66 | bwd_microstep: 609.23 | bwd_inner_microstep: 609.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:30:59,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.16 | bwd_microstep: 454.57 | bwd_inner_microstep: 454.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:30:59,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.75 | bwd_microstep: 372.83 | bwd_inner_microstep: 372.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:31:00,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.95 | bwd_microstep: 463.31 | bwd_inner_microstep: 463.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:31:02,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.80 | bwd_microstep: 625.58 | bwd_inner_microstep: 625.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:31:05,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.08 | optimizer_gradients: 14.05 | optimizer_step: 24.10
[2024-12-05 14:31:05,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.59 | bwd_microstep: 2597.13 | bwd_inner_microstep: 2584.98 | bwd_allreduce_microstep: 12.08 | step_microstep: 113.89
[2024-12-05 14:31:05,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4402.33 | bwd: 9684.07 | bwd_inner: 9670.28 | bwd_allreduce: 12.69 | step: 114.90

 44%|████▍     | 168/384 [48:01<1:05:12, 18.12s/it]
                                                   
{'loss': 0.1393, 'grad_norm': 1.5989458560943604, 'learning_rate': 6.253262661293602e-07, 'epoch': 0.87}

 44%|████▍     | 168/384 [48:01<1:05:12, 18.12s/it][2024-12-05 14:31:06,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.08 | bwd_microstep: 365.63 | bwd_inner_microstep: 365.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:31:07,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.12 | bwd_microstep: 545.86 | bwd_inner_microstep: 545.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:31:08,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.34 | bwd_microstep: 489.69 | bwd_inner_microstep: 489.53 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08
[2024-12-05 14:31:10,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.34 | bwd_microstep: 450.19 | bwd_inner_microstep: 450.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:31:11,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.92 | bwd_microstep: 564.34 | bwd_inner_microstep: 564.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:31:12,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.71 | bwd_microstep: 453.09 | bwd_inner_microstep: 452.95 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:31:13,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.52 | bwd_microstep: 436.10 | bwd_inner_microstep: 435.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:31:14,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.29 | bwd_microstep: 427.26 | bwd_inner_microstep: 427.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:31:15,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.02 | bwd_microstep: 504.55 | bwd_inner_microstep: 504.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:31:16,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.82 | bwd_microstep: 526.68 | bwd_inner_microstep: 526.55 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:31:16,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.16 | bwd_microstep: 461.30 | bwd_inner_microstep: 461.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:31:18,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.12 | bwd_microstep: 683.71 | bwd_inner_microstep: 683.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:31:19,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.73 | bwd_microstep: 401.13 | bwd_inner_microstep: 401.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:31:20,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.23 | bwd_microstep: 454.07 | bwd_inner_microstep: 453.82 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.16
[2024-12-05 14:31:21,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.50 | bwd_microstep: 443.06 | bwd_inner_microstep: 442.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:31:23,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.26 | optimizer_gradients: 14.04 | optimizer_step: 23.12
[2024-12-05 14:31:23,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.30 | bwd_microstep: 1462.66 | bwd_inner_microstep: 1450.54 | bwd_allreduce_microstep: 12.03 | step_microstep: 112.03
[2024-12-05 14:31:23,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4435.24 | bwd: 8669.35 | bwd_inner: 8655.43 | bwd_allreduce: 12.76 | step: 113.28

 44%|████▍     | 169/384 [48:19<1:04:32, 18.01s/it]
                                                   
{'loss': 0.1421, 'grad_norm': 1.464534878730774, 'learning_rate': 6.2123407143919e-07, 'epoch': 0.88}

 44%|████▍     | 169/384 [48:19<1:04:32, 18.01s/it][2024-12-05 14:31:24,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.66 | bwd_microstep: 417.25 | bwd_inner_microstep: 417.13 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:31:25,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.55 | bwd_microstep: 395.93 | bwd_inner_microstep: 395.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:31:26,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.68 | bwd_microstep: 436.18 | bwd_inner_microstep: 436.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:31:27,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.36 | bwd_microstep: 408.39 | bwd_inner_microstep: 408.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:31:28,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.44 | bwd_microstep: 475.44 | bwd_inner_microstep: 475.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:31:28,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.48 | bwd_microstep: 477.73 | bwd_inner_microstep: 477.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:31:30,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.11 | bwd_microstep: 522.02 | bwd_inner_microstep: 521.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:31:31,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.00 | bwd_microstep: 401.48 | bwd_inner_microstep: 401.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:31:32,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.38 | bwd_microstep: 431.24 | bwd_inner_microstep: 431.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:31:33,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.47 | bwd_microstep: 686.22 | bwd_inner_microstep: 686.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:31:34,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.53 | bwd_microstep: 488.58 | bwd_inner_microstep: 488.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:31:35,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 375.26 | bwd_microstep: 781.46 | bwd_inner_microstep: 781.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:31:36,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.14 | bwd_microstep: 575.88 | bwd_inner_microstep: 575.74 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.06
[2024-12-05 14:31:37,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.38 | bwd_microstep: 582.40 | bwd_inner_microstep: 582.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:31:38,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.33 | bwd_microstep: 485.80 | bwd_inner_microstep: 485.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:31:40,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.72 | optimizer_gradients: 14.05 | optimizer_step: 23.14
[2024-12-05 14:31:40,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.24 | bwd_microstep: 1868.47 | bwd_inner_microstep: 1856.29 | bwd_allreduce_microstep: 12.09 | step_microstep: 112.30
[2024-12-05 14:31:40,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4578.07 | bwd: 9434.47 | bwd_inner: 9420.61 | bwd_allreduce: 12.74 | step: 113.34

 44%|████▍     | 170/384 [48:37<1:03:42, 17.86s/it]
                                                   
{'loss': 0.154, 'grad_norm': 1.6165345907211304, 'learning_rate': 6.17133230336041e-07, 'epoch': 0.88}

 44%|████▍     | 170/384 [48:37<1:03:42, 17.86s/it][2024-12-05 14:31:41,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.21 | bwd_microstep: 385.90 | bwd_inner_microstep: 385.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:31:42,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.56 | bwd_microstep: 435.83 | bwd_inner_microstep: 435.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:31:43,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.16 | bwd_microstep: 458.27 | bwd_inner_microstep: 458.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:31:44,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.79 | bwd_microstep: 438.89 | bwd_inner_microstep: 438.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:31:45,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.82 | bwd_microstep: 406.09 | bwd_inner_microstep: 405.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:31:46,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.15 | bwd_microstep: 464.65 | bwd_inner_microstep: 464.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:31:47,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 325.68 | bwd_microstep: 607.59 | bwd_inner_microstep: 607.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:31:49,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.14 | bwd_microstep: 541.35 | bwd_inner_microstep: 541.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:31:50,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.51 | bwd_microstep: 488.50 | bwd_inner_microstep: 488.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:31:51,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.22 | bwd_microstep: 542.49 | bwd_inner_microstep: 542.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:31:52,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.64 | bwd_microstep: 424.62 | bwd_inner_microstep: 424.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:31:53,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.83 | bwd_microstep: 432.67 | bwd_inner_microstep: 432.52 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:31:53,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.28 | bwd_microstep: 427.15 | bwd_inner_microstep: 427.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:31:55,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.56 | bwd_microstep: 444.65 | bwd_inner_microstep: 444.49 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09
[2024-12-05 14:31:55,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.53 | bwd_microstep: 412.00 | bwd_inner_microstep: 411.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:31:59,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.28 | optimizer_gradients: 14.08 | optimizer_step: 23.09
[2024-12-05 14:31:59,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.95 | bwd_microstep: 2611.54 | bwd_inner_microstep: 2599.33 | bwd_allreduce_microstep: 12.11 | step_microstep: 111.88
[2024-12-05 14:31:59,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4327.07 | bwd: 9522.17 | bwd_inner: 9508.29 | bwd_allreduce: 12.78 | step: 113.04

 45%|████▍     | 171/384 [48:55<1:03:53, 18.00s/it]
                                                   
{'loss': 0.1614, 'grad_norm': 1.5105384588241577, 'learning_rate': 6.130240352918674e-07, 'epoch': 0.89}

 45%|████▍     | 171/384 [48:55<1:03:53, 18.00s/it][2024-12-05 14:32:00,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.19 | bwd_microstep: 674.53 | bwd_inner_microstep: 674.39 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:32:01,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.96 | bwd_microstep: 435.92 | bwd_inner_microstep: 435.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:32:02,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.57 | bwd_microstep: 532.95 | bwd_inner_microstep: 532.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:32:03,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.56 | bwd_microstep: 427.24 | bwd_inner_microstep: 427.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:32:05,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.69 | bwd_microstep: 452.86 | bwd_inner_microstep: 452.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:32:06,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.34 | bwd_microstep: 439.21 | bwd_inner_microstep: 439.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:32:07,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.78 | bwd_microstep: 452.04 | bwd_inner_microstep: 451.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:32:08,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.72 | bwd_microstep: 402.62 | bwd_inner_microstep: 402.47 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.06
[2024-12-05 14:32:09,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.16 | bwd_microstep: 485.65 | bwd_inner_microstep: 485.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:32:10,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.66 | bwd_microstep: 414.80 | bwd_inner_microstep: 414.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:32:11,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.18 | bwd_microstep: 450.36 | bwd_inner_microstep: 450.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:32:12,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.48 | bwd_microstep: 500.72 | bwd_inner_microstep: 500.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:32:13,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.30 | bwd_microstep: 456.51 | bwd_inner_microstep: 456.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:32:14,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.18 | bwd_microstep: 426.88 | bwd_inner_microstep: 426.73 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:32:15,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.27 | bwd_microstep: 537.26 | bwd_inner_microstep: 537.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:32:18,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.19 | optimizer_gradients: 14.08 | optimizer_step: 23.17
[2024-12-05 14:32:18,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.17 | bwd_microstep: 2387.64 | bwd_inner_microstep: 2375.58 | bwd_allreduce_microstep: 11.96 | step_microstep: 112.06
[2024-12-05 14:32:18,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4399.31 | bwd: 9477.18 | bwd_inner: 9463.44 | bwd_allreduce: 12.64 | step: 113.19

 45%|████▍     | 172/384 [49:14<1:05:02, 18.41s/it]
                                                   
{'loss': 0.1213, 'grad_norm': 1.6538264751434326, 'learning_rate': 6.089067793744257e-07, 'epoch': 0.89}

 45%|████▍     | 172/384 [49:14<1:05:02, 18.41s/it][2024-12-05 14:32:20,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.90 | bwd_microstep: 590.28 | bwd_inner_microstep: 590.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:32:21,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 383.03 | bwd_microstep: 837.12 | bwd_inner_microstep: 836.97 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:32:22,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.45 | bwd_microstep: 476.99 | bwd_inner_microstep: 476.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:32:23,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.54 | bwd_microstep: 353.44 | bwd_inner_microstep: 353.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:32:24,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.36 | bwd_microstep: 414.91 | bwd_inner_microstep: 414.78 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:32:25,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 356.82 | bwd_microstep: 747.43 | bwd_inner_microstep: 747.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:32:26,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.73 | bwd_microstep: 485.64 | bwd_inner_microstep: 485.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 14:32:27,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.88 | bwd_microstep: 431.45 | bwd_inner_microstep: 431.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:32:28,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.39 | bwd_microstep: 618.87 | bwd_inner_microstep: 618.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:32:28,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 243.91 | bwd_microstep: 380.78 | bwd_inner_microstep: 380.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:32:29,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.99 | bwd_microstep: 401.62 | bwd_inner_microstep: 401.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:32:30,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.57 | bwd_microstep: 425.91 | bwd_inner_microstep: 425.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:32:31,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.44 | bwd_microstep: 429.14 | bwd_inner_microstep: 428.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:32:32,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.90 | bwd_microstep: 487.88 | bwd_inner_microstep: 487.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:32:33,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.84 | bwd_microstep: 447.70 | bwd_inner_microstep: 447.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:32:37,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.17 | optimizer_gradients: 14.11 | optimizer_step: 23.10
[2024-12-05 14:32:37,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.12 | bwd_microstep: 2940.52 | bwd_inner_microstep: 2928.43 | bwd_allreduce_microstep: 11.99 | step_microstep: 112.20
[2024-12-05 14:32:37,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4554.92 | bwd: 10469.66 | bwd_inner: 10455.93 | bwd_allreduce: 12.64 | step: 113.38

 45%|████▌     | 173/384 [49:33<1:05:04, 18.51s/it]
                                                   
{'loss': 0.1349, 'grad_norm': 1.5659717321395874, 'learning_rate': 6.047817562263743e-07, 'epoch': 0.9}

 45%|████▌     | 173/384 [49:33<1:05:04, 18.51s/it][2024-12-05 14:32:38,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.74 | bwd_microstep: 489.20 | bwd_inner_microstep: 489.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:32:39,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.87 | bwd_microstep: 334.45 | bwd_inner_microstep: 334.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:32:40,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.78 | bwd_microstep: 369.78 | bwd_inner_microstep: 369.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:32:41,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.07 | bwd_microstep: 326.51 | bwd_inner_microstep: 326.36 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:32:42,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.15 | bwd_microstep: 555.21 | bwd_inner_microstep: 555.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:32:43,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.50 | bwd_microstep: 408.41 | bwd_inner_microstep: 408.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:32:44,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.40 | bwd_microstep: 407.13 | bwd_inner_microstep: 407.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:32:45,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.97 | bwd_microstep: 615.64 | bwd_inner_microstep: 615.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:32:46,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.24 | bwd_microstep: 443.96 | bwd_inner_microstep: 443.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:32:47,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.83 | bwd_microstep: 442.32 | bwd_inner_microstep: 442.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:32:48,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.94 | bwd_microstep: 445.07 | bwd_inner_microstep: 444.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:32:49,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.48 | bwd_microstep: 445.71 | bwd_inner_microstep: 445.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:32:50,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.58 | bwd_microstep: 451.91 | bwd_inner_microstep: 451.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:32:51,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.06 | bwd_microstep: 447.31 | bwd_inner_microstep: 447.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:32:51,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.44 | bwd_microstep: 411.90 | bwd_inner_microstep: 411.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:32:55,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.36 | optimizer_gradients: 14.05 | optimizer_step: 23.20
[2024-12-05 14:32:55,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.85 | bwd_microstep: 2667.44 | bwd_inner_microstep: 2655.26 | bwd_allreduce_microstep: 12.07 | step_microstep: 112.11
[2024-12-05 14:32:55,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4268.96 | bwd: 9261.93 | bwd_inner: 9248.22 | bwd_allreduce: 12.67 | step: 113.11

 45%|████▌     | 174/384 [49:51<1:03:55, 18.27s/it]
                                                   
{'loss': 0.1572, 'grad_norm': 1.6108099222183228, 'learning_rate': 6.0064926004433e-07, 'epoch': 0.9}

 45%|████▌     | 174/384 [49:51<1:03:55, 18.27s/it][2024-12-05 14:32:56,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.25 | bwd_microstep: 454.15 | bwd_inner_microstep: 454.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:32:57,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.38 | bwd_microstep: 687.29 | bwd_inner_microstep: 687.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:32:58,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.98 | bwd_microstep: 685.76 | bwd_inner_microstep: 685.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:32:59,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.93 | bwd_microstep: 464.95 | bwd_inner_microstep: 464.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:33:00,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.61 | bwd_microstep: 414.07 | bwd_inner_microstep: 413.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:33:01,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.58 | bwd_microstep: 557.89 | bwd_inner_microstep: 557.66 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.14
[2024-12-05 14:33:02,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.64 | bwd_microstep: 461.77 | bwd_inner_microstep: 461.62 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:33:04,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.08 | bwd_microstep: 720.60 | bwd_inner_microstep: 720.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 14:33:04,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.34 | bwd_microstep: 447.31 | bwd_inner_microstep: 447.18 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:33:05,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.96 | bwd_microstep: 460.92 | bwd_inner_microstep: 460.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:33:06,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.71 | bwd_microstep: 474.63 | bwd_inner_microstep: 474.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:33:07,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.15 | bwd_microstep: 396.38 | bwd_inner_microstep: 396.25 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:33:08,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.78 | bwd_microstep: 440.33 | bwd_inner_microstep: 440.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:33:09,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.27 | bwd_microstep: 608.59 | bwd_inner_microstep: 608.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:33:10,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.77 | bwd_microstep: 351.53 | bwd_inner_microstep: 351.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:33:12,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.19 | optimizer_gradients: 14.00 | optimizer_step: 23.35
[2024-12-05 14:33:12,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.34 | bwd_microstep: 1313.56 | bwd_inner_microstep: 1301.62 | bwd_allreduce_microstep: 11.84 | step_microstep: 112.15
[2024-12-05 14:33:12,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4517.86 | bwd: 8939.73 | bwd_inner: 8926.11 | bwd_allreduce: 12.52 | step: 113.37

 46%|████▌     | 175/384 [50:08<1:02:46, 18.02s/it]
                                                   
{'loss': 0.1376, 'grad_norm': 1.7230148315429688, 'learning_rate': 5.965095855578868e-07, 'epoch': 0.91}

 46%|████▌     | 175/384 [50:08<1:02:46, 18.02s/it][2024-12-05 14:33:13,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.14 | bwd_microstep: 450.79 | bwd_inner_microstep: 450.63 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:33:14,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.07 | bwd_microstep: 399.51 | bwd_inner_microstep: 399.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:33:15,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.05 | bwd_microstep: 435.03 | bwd_inner_microstep: 434.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:33:16,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 356.89 | bwd_microstep: 744.22 | bwd_inner_microstep: 744.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 14:33:17,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.41 | bwd_microstep: 411.27 | bwd_inner_microstep: 411.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:33:18,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.18 | bwd_microstep: 539.43 | bwd_inner_microstep: 539.28 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:33:19,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.37 | bwd_microstep: 413.33 | bwd_inner_microstep: 413.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:33:20,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.65 | bwd_microstep: 494.87 | bwd_inner_microstep: 494.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:33:21,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.98 | bwd_microstep: 448.98 | bwd_inner_microstep: 448.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:33:22,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.29 | bwd_microstep: 686.36 | bwd_inner_microstep: 686.22 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:33:23,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.59 | bwd_microstep: 591.99 | bwd_inner_microstep: 591.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:33:24,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.59 | bwd_microstep: 567.38 | bwd_inner_microstep: 567.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:33:26,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.29 | bwd_microstep: 430.36 | bwd_inner_microstep: 430.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:33:27,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.51 | bwd_microstep: 595.22 | bwd_inner_microstep: 595.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:33:28,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.78 | bwd_microstep: 445.29 | bwd_inner_microstep: 445.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:33:29,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.41 | optimizer_gradients: 14.04 | optimizer_step: 23.24
[2024-12-05 14:33:29,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.46 | bwd_microstep: 872.12 | bwd_inner_microstep: 859.48 | bwd_allreduce_microstep: 12.53 | step_microstep: 112.00
[2024-12-05 14:33:29,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4585.31 | bwd: 8526.14 | bwd_inner: 8511.83 | bwd_allreduce: 13.16 | step: 113.20

 46%|████▌     | 176/384 [50:26<1:01:45, 17.82s/it]
                                                   
{'loss': 0.1572, 'grad_norm': 1.7344483137130737, 'learning_rate': 5.923630280085947e-07, 'epoch': 0.91}

 46%|████▌     | 176/384 [50:26<1:01:45, 17.82s/it][2024-12-05 14:33:30,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.37 | bwd_microstep: 322.53 | bwd_inner_microstep: 322.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:33:31,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.54 | bwd_microstep: 462.65 | bwd_inner_microstep: 462.46 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.12
[2024-12-05 14:33:32,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.19 | bwd_microstep: 528.27 | bwd_inner_microstep: 528.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:33:33,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.57 | bwd_microstep: 457.70 | bwd_inner_microstep: 457.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:33:34,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.87 | bwd_microstep: 496.09 | bwd_inner_microstep: 495.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:33:36,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 361.16 | bwd_microstep: 739.38 | bwd_inner_microstep: 739.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:33:37,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.41 | bwd_microstep: 452.58 | bwd_inner_microstep: 452.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:33:38,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.44 | bwd_microstep: 431.33 | bwd_inner_microstep: 431.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:33:39,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.71 | bwd_microstep: 430.42 | bwd_inner_microstep: 430.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:33:40,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.37 | bwd_microstep: 496.63 | bwd_inner_microstep: 496.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:33:41,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.67 | bwd_microstep: 582.09 | bwd_inner_microstep: 581.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:33:42,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.53 | bwd_microstep: 449.81 | bwd_inner_microstep: 449.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:33:43,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.62 | bwd_microstep: 451.29 | bwd_inner_microstep: 451.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:33:44,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.89 | bwd_microstep: 452.73 | bwd_inner_microstep: 452.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:33:45,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.13 | bwd_microstep: 429.83 | bwd_inner_microstep: 429.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:33:47,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.28 | optimizer_gradients: 14.03 | optimizer_step: 23.20
[2024-12-05 14:33:47,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.13 | bwd_microstep: 1675.02 | bwd_inner_microstep: 1663.01 | bwd_allreduce_microstep: 11.93 | step_microstep: 113.84
[2024-12-05 14:33:47,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4486.68 | bwd: 8858.32 | bwd_inner: 8844.60 | bwd_allreduce: 12.58 | step: 114.96

 46%|████▌     | 177/384 [50:44<1:01:40, 17.87s/it]
                                                   
{'loss': 0.1626, 'grad_norm': 1.6481879949569702, 'learning_rate': 5.882098831289043e-07, 'epoch': 0.92}

 46%|████▌     | 177/384 [50:44<1:01:40, 17.87s/it][2024-12-05 14:33:48,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.49 | bwd_microstep: 421.84 | bwd_inner_microstep: 421.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:33:49,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.53 | bwd_microstep: 446.26 | bwd_inner_microstep: 446.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:33:51,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.65 | bwd_microstep: 424.28 | bwd_inner_microstep: 424.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:33:52,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.75 | bwd_microstep: 458.90 | bwd_inner_microstep: 458.74 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07
[2024-12-05 14:33:53,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.09 | bwd_microstep: 563.76 | bwd_inner_microstep: 563.63 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:33:54,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.29 | bwd_microstep: 483.86 | bwd_inner_microstep: 483.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:33:55,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.67 | bwd_microstep: 435.96 | bwd_inner_microstep: 435.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:33:56,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.52 | bwd_microstep: 422.52 | bwd_inner_microstep: 422.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:33:57,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.18 | bwd_microstep: 558.24 | bwd_inner_microstep: 558.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:33:58,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.23 | bwd_microstep: 568.30 | bwd_inner_microstep: 568.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:33:59,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.86 | bwd_microstep: 457.96 | bwd_inner_microstep: 457.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:34:00,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.36 | bwd_microstep: 447.71 | bwd_inner_microstep: 447.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:34:01,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.84 | bwd_microstep: 433.05 | bwd_inner_microstep: 432.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:34:03,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.87 | bwd_microstep: 533.17 | bwd_inner_microstep: 533.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:34:04,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.28 | bwd_microstep: 545.39 | bwd_inner_microstep: 545.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:34:06,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.91 | optimizer_gradients: 14.10 | optimizer_step: 23.29
[2024-12-05 14:34:06,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 341.39 | bwd_microstep: 1833.46 | bwd_inner_microstep: 1821.45 | bwd_allreduce_microstep: 11.91 | step_microstep: 114.92
[2024-12-05 14:34:06,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4526.03 | bwd: 9034.67 | bwd_inner: 9020.97 | bwd_allreduce: 12.54 | step: 115.99

 46%|████▋     | 178/384 [51:02<1:02:11, 18.12s/it]
                                                   
{'loss': 0.1756, 'grad_norm': 1.7318065166473389, 'learning_rate': 5.840504471210741e-07, 'epoch': 0.93}

 46%|████▋     | 178/384 [51:02<1:02:11, 18.12s/it][2024-12-05 14:34:07,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.86 | bwd_microstep: 453.07 | bwd_inner_microstep: 452.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:34:08,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.09 | bwd_microstep: 464.82 | bwd_inner_microstep: 464.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:34:09,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.34 | bwd_microstep: 478.61 | bwd_inner_microstep: 478.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:34:10,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.31 | bwd_microstep: 353.10 | bwd_inner_microstep: 352.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:34:11,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.28 | bwd_microstep: 523.81 | bwd_inner_microstep: 523.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:34:12,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 370.56 | bwd_microstep: 731.11 | bwd_inner_microstep: 730.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:34:14,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.25 | bwd_microstep: 684.58 | bwd_inner_microstep: 684.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:34:15,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.84 | bwd_microstep: 612.41 | bwd_inner_microstep: 612.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:34:16,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.09 | bwd_microstep: 482.27 | bwd_inner_microstep: 482.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:34:17,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.43 | bwd_microstep: 426.17 | bwd_inner_microstep: 426.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:34:18,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.87 | bwd_microstep: 825.39 | bwd_inner_microstep: 825.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:34:19,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.61 | bwd_microstep: 441.02 | bwd_inner_microstep: 440.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:34:21,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.21 | bwd_microstep: 518.22 | bwd_inner_microstep: 518.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:34:22,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.77 | bwd_microstep: 413.34 | bwd_inner_microstep: 413.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:34:23,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.64 | bwd_microstep: 488.24 | bwd_inner_microstep: 488.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:34:25,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.10 | optimizer_gradients: 14.05 | optimizer_step: 23.13
[2024-12-05 14:34:25,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.79 | bwd_microstep: 1270.44 | bwd_inner_microstep: 1258.24 | bwd_allreduce_microstep: 12.09 | step_microstep: 111.81
[2024-12-05 14:34:25,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4648.02 | bwd: 9166.63 | bwd_inner: 9152.84 | bwd_allreduce: 12.70 | step: 112.92

 47%|████▋     | 179/384 [51:21<1:02:22, 18.25s/it]
                                                   
{'loss': 0.147, 'grad_norm': 1.676165223121643, 'learning_rate': 5.79885016636046e-07, 'epoch': 0.93}

 47%|████▋     | 179/384 [51:21<1:02:22, 18.25s/it][2024-12-05 14:34:26,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.87 | bwd_microstep: 453.14 | bwd_inner_microstep: 452.99 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:34:27,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.51 | bwd_microstep: 445.35 | bwd_inner_microstep: 445.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:34:28,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.00 | bwd_microstep: 417.48 | bwd_inner_microstep: 417.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:34:29,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.49 | bwd_microstep: 480.89 | bwd_inner_microstep: 480.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:34:30,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.83 | bwd_microstep: 441.05 | bwd_inner_microstep: 440.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:34:31,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.23 | bwd_microstep: 460.17 | bwd_inner_microstep: 460.03 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:34:32,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.40 | bwd_microstep: 523.09 | bwd_inner_microstep: 522.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:34:33,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.55 | bwd_microstep: 496.15 | bwd_inner_microstep: 495.97 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 14:34:34,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.11 | bwd_microstep: 522.03 | bwd_inner_microstep: 521.90 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:34:36,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.47 | bwd_microstep: 441.73 | bwd_inner_microstep: 441.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:34:37,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.55 | bwd_microstep: 526.13 | bwd_inner_microstep: 525.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:34:37,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.70 | bwd_microstep: 415.06 | bwd_inner_microstep: 414.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:34:39,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 355.36 | bwd_microstep: 694.73 | bwd_inner_microstep: 694.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:34:39,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.50 | bwd_microstep: 409.13 | bwd_inner_microstep: 409.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:34:40,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.10 | bwd_microstep: 438.34 | bwd_inner_microstep: 438.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:34:42,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.31 | optimizer_gradients: 14.10 | optimizer_step: 23.14
[2024-12-05 14:34:42,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.48 | bwd_microstep: 1157.90 | bwd_inner_microstep: 1145.83 | bwd_allreduce_microstep: 11.97 | step_microstep: 113.42
[2024-12-05 14:34:42,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4377.22 | bwd: 8322.37 | bwd_inner: 8308.60 | bwd_allreduce: 12.63 | step: 114.61

 47%|████▋     | 180/384 [51:38<1:01:21, 18.05s/it]
                                                   
{'loss': 0.1249, 'grad_norm': 1.4620646238327026, 'learning_rate': 5.757138887522883e-07, 'epoch': 0.94}

 47%|████▋     | 180/384 [51:38<1:01:21, 18.05s/it][2024-12-05 14:34:43,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.09 | bwd_microstep: 398.84 | bwd_inner_microstep: 398.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:34:44,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.41 | bwd_microstep: 433.28 | bwd_inner_microstep: 433.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:34:46,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 325.00 | bwd_microstep: 624.63 | bwd_inner_microstep: 624.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:34:47,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.37 | bwd_microstep: 450.52 | bwd_inner_microstep: 450.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:34:48,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.90 | bwd_microstep: 559.54 | bwd_inner_microstep: 559.36 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:34:49,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.29 | bwd_microstep: 410.37 | bwd_inner_microstep: 410.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:34:50,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.91 | bwd_microstep: 444.39 | bwd_inner_microstep: 444.26 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:34:51,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.39 | bwd_microstep: 457.90 | bwd_inner_microstep: 457.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:34:52,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.13 | bwd_microstep: 540.90 | bwd_inner_microstep: 540.75 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:34:53,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.34 | bwd_microstep: 425.30 | bwd_inner_microstep: 425.16 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:34:54,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.29 | bwd_microstep: 480.43 | bwd_inner_microstep: 480.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:34:55,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.07 | bwd_microstep: 597.62 | bwd_inner_microstep: 597.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:34:56,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.24 | bwd_microstep: 482.80 | bwd_inner_microstep: 482.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:34:57,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.51 | bwd_microstep: 451.53 | bwd_inner_microstep: 451.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:34:58,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.77 | bwd_microstep: 491.01 | bwd_inner_microstep: 490.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:35:01,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.85 | optimizer_gradients: 14.08 | optimizer_step: 23.12
[2024-12-05 14:35:01,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 325.16 | bwd_microstep: 1828.05 | bwd_inner_microstep: 1815.95 | bwd_allreduce_microstep: 12.01 | step_microstep: 112.68
[2024-12-05 14:35:01,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4557.93 | bwd: 9077.08 | bwd_inner: 9063.32 | bwd_allreduce: 12.65 | step: 113.77

 47%|████▋     | 181/384 [51:57<1:01:34, 18.20s/it]
                                                   
{'loss': 0.163, 'grad_norm': 1.65927255153656, 'learning_rate': 5.71537360954607e-07, 'epoch': 0.94}

 47%|████▋     | 181/384 [51:57<1:01:34, 18.20s/it][2024-12-05 14:35:02,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.45 | bwd_microstep: 426.84 | bwd_inner_microstep: 426.71 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:35:03,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.38 | bwd_microstep: 438.19 | bwd_inner_microstep: 438.03 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:35:04,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.04 | bwd_microstep: 414.95 | bwd_inner_microstep: 414.76 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 14:35:05,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 351.18 | bwd_microstep: 722.07 | bwd_inner_microstep: 721.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:35:06,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 367.94 | bwd_microstep: 725.11 | bwd_inner_microstep: 724.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:35:07,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.54 | bwd_microstep: 452.18 | bwd_inner_microstep: 452.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:35:08,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.06 | bwd_microstep: 509.94 | bwd_inner_microstep: 509.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:35:10,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.02 | bwd_microstep: 633.31 | bwd_inner_microstep: 633.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:35:11,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.56 | bwd_microstep: 411.29 | bwd_inner_microstep: 411.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:35:11,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.09 | bwd_microstep: 451.48 | bwd_inner_microstep: 451.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:35:12,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.72 | bwd_microstep: 451.66 | bwd_inner_microstep: 451.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:35:14,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.68 | bwd_microstep: 627.15 | bwd_inner_microstep: 627.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:35:15,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.87 | bwd_microstep: 616.90 | bwd_inner_microstep: 616.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:35:16,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.83 | bwd_microstep: 410.54 | bwd_inner_microstep: 410.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:35:17,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.95 | bwd_microstep: 494.13 | bwd_inner_microstep: 494.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:35:19,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.01 | optimizer_gradients: 14.03 | optimizer_step: 23.11
[2024-12-05 14:35:19,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.09 | bwd_microstep: 1349.15 | bwd_inner_microstep: 1336.97 | bwd_allreduce_microstep: 12.08 | step_microstep: 114.12
[2024-12-05 14:35:19,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4689.47 | bwd: 9134.87 | bwd_inner: 9120.95 | bwd_allreduce: 12.74 | step: 115.26

 47%|████▋     | 182/384 [52:16<1:01:40, 18.32s/it]
                                                   
{'loss': 0.1351, 'grad_norm': 1.5536587238311768, 'learning_rate': 5.673557311129306e-07, 'epoch': 0.95}

 47%|████▋     | 182/384 [52:16<1:01:40, 18.32s/it][2024-12-05 14:35:21,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 363.89 | bwd_microstep: 732.12 | bwd_inner_microstep: 731.96 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:35:22,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.35 | bwd_microstep: 431.77 | bwd_inner_microstep: 431.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:35:23,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.24 | bwd_microstep: 428.95 | bwd_inner_microstep: 428.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:35:24,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.10 | bwd_microstep: 408.56 | bwd_inner_microstep: 408.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:35:25,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.88 | bwd_microstep: 412.51 | bwd_inner_microstep: 412.36 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:35:26,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.03 | bwd_microstep: 688.42 | bwd_inner_microstep: 688.27 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:35:27,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.72 | bwd_microstep: 449.99 | bwd_inner_microstep: 449.86 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:35:28,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.05 | bwd_microstep: 463.23 | bwd_inner_microstep: 463.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:35:29,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.78 | bwd_microstep: 462.33 | bwd_inner_microstep: 462.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:35:30,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.58 | bwd_microstep: 402.80 | bwd_inner_microstep: 402.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:35:31,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.68 | bwd_microstep: 435.79 | bwd_inner_microstep: 435.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:35:32,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.59 | bwd_microstep: 625.94 | bwd_inner_microstep: 625.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:35:33,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.39 | bwd_microstep: 439.20 | bwd_inner_microstep: 439.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:35:34,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.66 | bwd_microstep: 429.75 | bwd_inner_microstep: 429.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:35:35,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.18 | bwd_microstep: 433.80 | bwd_inner_microstep: 433.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:35:38,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.40 | optimizer_gradients: 14.04 | optimizer_step: 23.14
[2024-12-05 14:35:38,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.31 | bwd_microstep: 2481.86 | bwd_inner_microstep: 2469.72 | bwd_allreduce_microstep: 12.03 | step_microstep: 114.28
[2024-12-05 14:35:38,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4444.47 | bwd: 9727.01 | bwd_inner: 9713.15 | bwd_allreduce: 12.71 | step: 115.46

 48%|████▊     | 183/384 [52:34<1:01:16, 18.29s/it]
                                                   
{'loss': 0.1771, 'grad_norm': 1.7076836824417114, 'learning_rate': 5.631692974610647e-07, 'epoch': 0.95}

 48%|████▊     | 183/384 [52:34<1:01:16, 18.29s/it][2024-12-05 14:35:38,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.44 | bwd_microstep: 497.79 | bwd_inner_microstep: 497.62 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:35:40,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.55 | bwd_microstep: 437.84 | bwd_inner_microstep: 437.71 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:35:40,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.32 | bwd_microstep: 478.92 | bwd_inner_microstep: 478.79 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:35:41,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.23 | bwd_microstep: 441.12 | bwd_inner_microstep: 440.94 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 14:35:43,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 327.66 | bwd_microstep: 623.35 | bwd_inner_microstep: 623.17 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:35:44,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.03 | bwd_microstep: 384.24 | bwd_inner_microstep: 384.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:35:45,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.22 | bwd_microstep: 485.44 | bwd_inner_microstep: 485.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:35:46,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.14 | bwd_microstep: 460.84 | bwd_inner_microstep: 460.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:35:46,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.83 | bwd_microstep: 404.92 | bwd_inner_microstep: 404.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:35:47,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.37 | bwd_microstep: 405.32 | bwd_inner_microstep: 405.19 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:35:48,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.24 | bwd_microstep: 491.02 | bwd_inner_microstep: 490.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:35:49,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.70 | bwd_microstep: 430.52 | bwd_inner_microstep: 430.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:35:50,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.32 | bwd_microstep: 482.08 | bwd_inner_microstep: 481.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:35:51,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.82 | bwd_microstep: 408.94 | bwd_inner_microstep: 408.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:35:52,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.69 | bwd_microstep: 400.61 | bwd_inner_microstep: 400.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:35:56,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.42 | optimizer_gradients: 14.01 | optimizer_step: 23.36
[2024-12-05 14:35:56,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.43 | bwd_microstep: 2843.55 | bwd_inner_microstep: 2831.54 | bwd_allreduce_microstep: 11.91 | step_microstep: 112.42
[2024-12-05 14:35:56,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4370.07 | bwd: 9676.50 | bwd_inner: 9662.74 | bwd_allreduce: 12.57 | step: 113.59

 48%|████▊     | 184/384 [52:52<1:00:44, 18.22s/it]
                                                   
{'loss': 0.121, 'grad_norm': 1.4979828596115112, 'learning_rate': 5.589783585754231e-07, 'epoch': 0.96}

 48%|████▊     | 184/384 [52:52<1:00:44, 18.22s/it][2024-12-05 14:35:57,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.22 | bwd_microstep: 646.96 | bwd_inner_microstep: 646.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:35:58,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.73 | bwd_microstep: 585.92 | bwd_inner_microstep: 585.74 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10
[2024-12-05 14:35:59,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.27 | bwd_microstep: 398.73 | bwd_inner_microstep: 398.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:36:00,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.18 | bwd_microstep: 694.55 | bwd_inner_microstep: 694.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:36:01,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.70 | bwd_microstep: 430.28 | bwd_inner_microstep: 430.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:36:02,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.49 | bwd_microstep: 449.89 | bwd_inner_microstep: 449.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:36:03,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.24 | bwd_microstep: 438.92 | bwd_inner_microstep: 438.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:36:04,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.99 | bwd_microstep: 400.35 | bwd_inner_microstep: 400.22 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:36:05,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.10 | bwd_microstep: 395.60 | bwd_inner_microstep: 395.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:36:06,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.62 | bwd_microstep: 450.12 | bwd_inner_microstep: 449.99 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:36:07,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.78 | bwd_microstep: 491.33 | bwd_inner_microstep: 491.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:36:08,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.00 | bwd_microstep: 488.90 | bwd_inner_microstep: 488.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:36:09,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.46 | bwd_microstep: 435.25 | bwd_inner_microstep: 435.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:36:10,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.73 | bwd_microstep: 449.75 | bwd_inner_microstep: 449.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:36:11,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.39 | bwd_microstep: 684.58 | bwd_inner_microstep: 684.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:36:13,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.95 | optimizer_gradients: 14.05 | optimizer_step: 23.31
[2024-12-05 14:36:13,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.38 | bwd_microstep: 1017.41 | bwd_inner_microstep: 1005.36 | bwd_allreduce_microstep: 11.96 | step_microstep: 111.73
[2024-12-05 14:36:13,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4553.36 | bwd: 8458.54 | bwd_inner: 8444.81 | bwd_allreduce: 12.60 | step: 112.81

 48%|████▊     | 185/384 [53:09<59:17, 17.88s/it]  
                                                 
{'loss': 0.1458, 'grad_norm': 1.5839756727218628, 'learning_rate': 5.547832133537327e-07, 'epoch': 0.96}

 48%|████▊     | 185/384 [53:09<59:17, 17.88s/it][2024-12-05 14:36:14,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.72 | bwd_microstep: 379.93 | bwd_inner_microstep: 379.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:36:15,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.44 | bwd_microstep: 427.95 | bwd_inner_microstep: 427.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:36:16,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.82 | bwd_microstep: 623.16 | bwd_inner_microstep: 623.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:36:17,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.14 | bwd_microstep: 436.29 | bwd_inner_microstep: 436.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:36:17,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.41 | bwd_microstep: 428.95 | bwd_inner_microstep: 428.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:36:18,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.85 | bwd_microstep: 449.41 | bwd_inner_microstep: 449.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:36:20,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.62 | bwd_microstep: 688.55 | bwd_inner_microstep: 688.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:36:21,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.36 | bwd_microstep: 439.10 | bwd_inner_microstep: 438.96 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:36:22,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.39 | bwd_microstep: 429.02 | bwd_inner_microstep: 428.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:36:23,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.06 | bwd_microstep: 398.11 | bwd_inner_microstep: 397.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:36:24,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.94 | bwd_microstep: 435.90 | bwd_inner_microstep: 435.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:36:25,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.90 | bwd_microstep: 435.97 | bwd_inner_microstep: 435.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:36:26,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.76 | bwd_microstep: 689.91 | bwd_inner_microstep: 689.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:36:27,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.54 | bwd_microstep: 397.49 | bwd_inner_microstep: 397.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:36:28,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 393.75 | bwd_microstep: 879.66 | bwd_inner_microstep: 879.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:36:31,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.15 | optimizer_gradients: 14.01 | optimizer_step: 23.27
[2024-12-05 14:36:31,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.44 | bwd_microstep: 2210.04 | bwd_inner_microstep: 2197.87 | bwd_allreduce_microstep: 12.07 | step_microstep: 113.71
[2024-12-05 14:36:31,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4523.19 | bwd: 9749.43 | bwd_inner: 9735.63 | bwd_allreduce: 12.70 | step: 114.88

 48%|████▊     | 186/384 [53:28<59:40, 18.09s/it]
                                                 
{'loss': 0.1671, 'grad_norm': 1.7546137571334839, 'learning_rate': 5.505841609937161e-07, 'epoch': 0.97}

 48%|████▊     | 186/384 [53:28<59:40, 18.09s/it][2024-12-05 14:36:32,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.83 | bwd_microstep: 383.94 | bwd_inner_microstep: 383.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:36:34,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.37 | bwd_microstep: 493.08 | bwd_inner_microstep: 492.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:36:34,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.11 | bwd_microstep: 393.54 | bwd_inner_microstep: 393.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:36:35,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.25 | bwd_microstep: 404.40 | bwd_inner_microstep: 404.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:36:37,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.63 | bwd_microstep: 461.14 | bwd_inner_microstep: 461.00 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:36:38,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.93 | bwd_microstep: 595.07 | bwd_inner_microstep: 594.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:36:39,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.20 | bwd_microstep: 430.13 | bwd_inner_microstep: 429.98 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 14:36:40,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.29 | bwd_microstep: 403.80 | bwd_inner_microstep: 403.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:36:41,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.45 | bwd_microstep: 433.87 | bwd_inner_microstep: 433.73 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:36:42,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.74 | bwd_microstep: 451.60 | bwd_inner_microstep: 451.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:36:44,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.64 | bwd_microstep: 528.37 | bwd_inner_microstep: 528.17 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 14:36:45,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.90 | bwd_microstep: 540.74 | bwd_inner_microstep: 540.60 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:36:46,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.14 | bwd_microstep: 506.39 | bwd_inner_microstep: 506.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:36:47,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.01 | bwd_microstep: 435.27 | bwd_inner_microstep: 435.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:36:48,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.36 | bwd_microstep: 408.13 | bwd_inner_microstep: 407.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:36:51,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.58 | optimizer_gradients: 14.07 | optimizer_step: 23.25
[2024-12-05 14:36:51,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.35 | bwd_microstep: 2128.06 | bwd_inner_microstep: 2115.91 | bwd_allreduce_microstep: 12.05 | step_microstep: 112.63
[2024-12-05 14:36:51,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4366.28 | bwd: 8997.54 | bwd_inner: 8983.65 | bwd_allreduce: 12.75 | step: 113.85

 49%|████▊     | 187/384 [53:48<1:01:27, 18.72s/it]
                                                   
{'loss': 0.1314, 'grad_norm': 1.7105190753936768, 'learning_rate': 5.463815009717532e-07, 'epoch': 0.97}

 49%|████▊     | 187/384 [53:48<1:01:27, 18.72s/it][2024-12-05 14:36:53,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.26 | bwd_microstep: 390.77 | bwd_inner_microstep: 390.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:36:54,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.79 | bwd_microstep: 415.02 | bwd_inner_microstep: 414.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:36:55,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.03 | bwd_microstep: 480.52 | bwd_inner_microstep: 480.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:36:56,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.71 | bwd_microstep: 558.64 | bwd_inner_microstep: 558.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:36:58,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.57 | bwd_microstep: 486.71 | bwd_inner_microstep: 486.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:36:59,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.75 | bwd_microstep: 412.06 | bwd_inner_microstep: 411.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:37:00,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.53 | bwd_microstep: 527.45 | bwd_inner_microstep: 527.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:37:01,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.00 | bwd_microstep: 417.38 | bwd_inner_microstep: 417.25 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:37:03,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.80 | bwd_microstep: 435.87 | bwd_inner_microstep: 435.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:37:04,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.46 | bwd_microstep: 400.45 | bwd_inner_microstep: 400.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:05,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.79 | bwd_microstep: 358.33 | bwd_inner_microstep: 358.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:37:06,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.69 | bwd_microstep: 490.06 | bwd_inner_microstep: 489.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:08,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.37 | bwd_microstep: 557.12 | bwd_inner_microstep: 557.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:08,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.38 | bwd_microstep: 462.11 | bwd_inner_microstep: 461.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:37:10,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 358.87 | bwd_microstep: 730.66 | bwd_inner_microstep: 730.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:13,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.51 | optimizer_gradients: 14.12 | optimizer_step: 23.24
[2024-12-05 14:37:13,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.21 | bwd_microstep: 1738.41 | bwd_inner_microstep: 1726.30 | bwd_allreduce_microstep: 12.02 | step_microstep: 194.10
[2024-12-05 14:37:13,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4485.28 | bwd: 8861.55 | bwd_inner: 8847.81 | bwd_allreduce: 12.66 | step: 195.25

 49%|████▉     | 188/384 [54:09<1:03:34, 19.46s/it]
                                                   
{'loss': 0.1794, 'grad_norm': 1.8557852506637573, 'learning_rate': 5.421755330215223e-07, 'epoch': 0.98}

 49%|████▉     | 188/384 [54:09<1:03:34, 19.46s/it][2024-12-05 14:37:14,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.63 | bwd_microstep: 370.56 | bwd_inner_microstep: 370.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:37:15,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.30 | bwd_microstep: 453.61 | bwd_inner_microstep: 453.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:37:16,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.86 | bwd_microstep: 535.01 | bwd_inner_microstep: 534.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:37:17,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.35 | bwd_microstep: 411.83 | bwd_inner_microstep: 411.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:37:18,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.71 | bwd_microstep: 458.16 | bwd_inner_microstep: 458.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:37:20,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.33 | bwd_microstep: 431.98 | bwd_inner_microstep: 431.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:37:21,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.69 | bwd_microstep: 431.85 | bwd_inner_microstep: 431.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:37:22,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.20 | bwd_microstep: 439.31 | bwd_inner_microstep: 439.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:37:23,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.03 | bwd_microstep: 493.34 | bwd_inner_microstep: 493.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:24,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.83 | bwd_microstep: 397.57 | bwd_inner_microstep: 397.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:25,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.64 | bwd_microstep: 559.87 | bwd_inner_microstep: 559.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:27,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 357.85 | bwd_microstep: 755.05 | bwd_inner_microstep: 754.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:28,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.47 | bwd_microstep: 431.00 | bwd_inner_microstep: 430.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:29,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.16 | bwd_microstep: 451.84 | bwd_inner_microstep: 451.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:37:30,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.37 | bwd_microstep: 595.83 | bwd_inner_microstep: 595.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:35,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.40 | optimizer_gradients: 14.01 | optimizer_step: 23.20
[2024-12-05 14:37:35,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.48 | bwd_microstep: 4121.78 | bwd_inner_microstep: 4109.66 | bwd_allreduce_microstep: 12.03 | step_microstep: 112.03
[2024-12-05 14:37:35,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4426.95 | bwd: 11338.57 | bwd_inner: 11324.82 | bwd_allreduce: 12.65 | step: 113.11

 49%|████▉     | 189/384 [54:31<1:05:39, 20.20s/it]
                                                   
{'loss': 0.1307, 'grad_norm': 1.569214105606079, 'learning_rate': 5.379665571126231e-07, 'epoch': 0.98}

 49%|████▉     | 189/384 [54:31<1:05:39, 20.20s/it][2024-12-05 14:37:36,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.83 | bwd_microstep: 403.52 | bwd_inner_microstep: 403.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:37:37,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.40 | bwd_microstep: 351.47 | bwd_inner_microstep: 351.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:37:38,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.37 | bwd_microstep: 412.47 | bwd_inner_microstep: 412.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:37:39,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.34 | bwd_microstep: 412.29 | bwd_inner_microstep: 412.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:40,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.16 | bwd_microstep: 409.83 | bwd_inner_microstep: 409.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:41,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.26 | bwd_microstep: 442.99 | bwd_inner_microstep: 442.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:43,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.23 | bwd_microstep: 413.78 | bwd_inner_microstep: 413.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:44,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.34 | bwd_microstep: 359.27 | bwd_inner_microstep: 359.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:37:45,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.59 | bwd_microstep: 410.01 | bwd_inner_microstep: 409.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:46,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.90 | bwd_microstep: 625.57 | bwd_inner_microstep: 625.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:47,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.78 | bwd_microstep: 446.01 | bwd_inner_microstep: 445.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:48,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.89 | bwd_microstep: 627.48 | bwd_inner_microstep: 627.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:50,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 350.45 | bwd_microstep: 676.06 | bwd_inner_microstep: 675.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:51,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.41 | bwd_microstep: 437.74 | bwd_inner_microstep: 437.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:37:52,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.95 | bwd_microstep: 425.97 | bwd_inner_microstep: 425.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:37:55,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.38 | optimizer_gradients: 14.16 | optimizer_step: 23.11
[2024-12-05 14:37:55,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.75 | bwd_microstep: 1730.37 | bwd_inner_microstep: 1718.15 | bwd_allreduce_microstep: 12.12 | step_microstep: 112.18
[2024-12-05 14:37:55,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4353.72 | bwd: 8584.82 | bwd_inner: 8570.94 | bwd_allreduce: 12.75 | step: 113.21

 49%|████▉     | 190/384 [54:51<1:05:13, 20.17s/it]
                                                   
{'loss': 0.1412, 'grad_norm': 1.6137492656707764, 'learning_rate': 5.337548734291826e-07, 'epoch': 0.99}

 49%|████▉     | 190/384 [54:51<1:05:13, 20.17s/it][2024-12-05 14:37:56,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.90 | bwd_microstep: 379.75 | bwd_inner_microstep: 379.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:37:57,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.76 | bwd_microstep: 449.46 | bwd_inner_microstep: 449.32 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 14:37:58,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.92 | bwd_microstep: 403.77 | bwd_inner_microstep: 403.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:37:59,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.40 | bwd_microstep: 338.98 | bwd_inner_microstep: 338.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:38:00,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.11 | bwd_microstep: 450.05 | bwd_inner_microstep: 449.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:38:01,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 364.74 | bwd_microstep: 707.06 | bwd_inner_microstep: 706.94 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:38:03,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.94 | bwd_microstep: 463.72 | bwd_inner_microstep: 463.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:38:04,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.27 | bwd_microstep: 434.65 | bwd_inner_microstep: 434.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:38:05,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.25 | bwd_microstep: 404.51 | bwd_inner_microstep: 404.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:38:06,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 350.53 | bwd_microstep: 721.05 | bwd_inner_microstep: 720.83 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:38:08,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.05 | bwd_microstep: 678.26 | bwd_inner_microstep: 678.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:38:09,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.15 | bwd_microstep: 492.80 | bwd_inner_microstep: 492.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:38:10,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.52 | bwd_microstep: 415.40 | bwd_inner_microstep: 415.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:38:11,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.21 | bwd_microstep: 444.45 | bwd_inner_microstep: 444.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:38:13,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.81 | bwd_microstep: 572.39 | bwd_inner_microstep: 572.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:38:14,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.69 | optimizer_gradients: 14.06 | optimizer_step: 23.27
[2024-12-05 14:38:14,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.82 | bwd_microstep: 800.07 | bwd_inner_microstep: 787.90 | bwd_allreduce_microstep: 12.07 | step_microstep: 112.45
[2024-12-05 14:38:14,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4484.43 | bwd: 8156.39 | bwd_inner: 8142.51 | bwd_allreduce: 12.72 | step: 113.56

 50%|████▉     | 191/384 [55:10<1:04:07, 19.94s/it]
                                                   
{'loss': 0.1627, 'grad_norm': 1.6707996129989624, 'learning_rate': 5.295407823484467e-07, 'epoch': 0.99}

 50%|████▉     | 191/384 [55:10<1:04:07, 19.94s/it][2024-12-05 14:38:15,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.72 | bwd_microstep: 396.29 | bwd_inner_microstep: 396.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:38:17,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.52 | bwd_microstep: 458.28 | bwd_inner_microstep: 458.12 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08
[2024-12-05 14:38:18,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.56 | bwd_microstep: 535.90 | bwd_inner_microstep: 535.70 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:38:19,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 349.01 | bwd_microstep: 717.70 | bwd_inner_microstep: 717.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:38:20,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.00 | bwd_microstep: 483.18 | bwd_inner_microstep: 483.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:38:22,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.21 | bwd_microstep: 360.14 | bwd_inner_microstep: 360.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:38:22,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.31 | bwd_microstep: 451.18 | bwd_inner_microstep: 451.05 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:38:24,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.06 | bwd_microstep: 462.19 | bwd_inner_microstep: 462.04 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07
[2024-12-05 14:38:25,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 343.76 | bwd_microstep: 717.69 | bwd_inner_microstep: 717.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:38:27,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.20 | bwd_microstep: 454.69 | bwd_inner_microstep: 454.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:38:28,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.90 | bwd_microstep: 440.20 | bwd_inner_microstep: 440.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:38:29,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.35 | bwd_microstep: 439.50 | bwd_inner_microstep: 439.36 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:38:31,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.28 | bwd_microstep: 488.22 | bwd_inner_microstep: 488.07 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:38:32,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.02 | bwd_microstep: 689.67 | bwd_inner_microstep: 689.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:38:33,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.72 | bwd_microstep: 434.55 | bwd_inner_microstep: 434.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:38:36,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.93 | optimizer_gradients: 14.13 | optimizer_step: 23.15
[2024-12-05 14:38:36,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.68 | bwd_microstep: 1695.22 | bwd_inner_microstep: 1683.00 | bwd_allreduce_microstep: 12.13 | step_microstep: 111.69
[2024-12-05 14:38:36,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4468.35 | bwd: 9224.59 | bwd_inner: 9210.62 | bwd_allreduce: 12.81 | step: 112.81

 50%|█████     | 192/384 [55:32<1:05:42, 20.53s/it]
                                                   
{'loss': 0.1872, 'grad_norm': 2.3229496479034424, 'learning_rate': 5.253245844193564e-07, 'epoch': 1.0}

 50%|█████     | 192/384 [55:32<1:05:42, 20.53s/it][2024-12-05 14:38:38,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.78 | bwd_microstep: 526.14 | bwd_inner_microstep: 526.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:38:39,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.05 | bwd_microstep: 479.51 | bwd_inner_microstep: 479.38 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.06
[2024-12-05 14:38:40,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.99 | bwd_microstep: 459.66 | bwd_inner_microstep: 459.53 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:38:41,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.83 | bwd_microstep: 687.15 | bwd_inner_microstep: 687.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:38:42,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.04 | bwd_microstep: 437.23 | bwd_inner_microstep: 437.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:38:43,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.83 | bwd_microstep: 505.08 | bwd_inner_microstep: 504.95 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:38:45,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.44 | bwd_microstep: 435.02 | bwd_inner_microstep: 434.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:38:46,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.03 | bwd_microstep: 482.25 | bwd_inner_microstep: 482.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:38:47,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.55 | bwd_microstep: 450.20 | bwd_inner_microstep: 450.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:38:48,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.69 | bwd_microstep: 506.24 | bwd_inner_microstep: 506.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:38:49,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.31 | bwd_microstep: 589.40 | bwd_inner_microstep: 589.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:38:50,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.62 | bwd_microstep: 421.57 | bwd_inner_microstep: 421.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:38:51,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.85 | bwd_microstep: 413.92 | bwd_inner_microstep: 413.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:38:52,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.69 | bwd_microstep: 488.83 | bwd_inner_microstep: 488.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:38:54,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.08 | bwd_microstep: 462.37 | bwd_inner_microstep: 462.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:38:56,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.51 | optimizer_gradients: 14.10 | optimizer_step: 23.30
[2024-12-05 14:38:56,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.03 | bwd_microstep: 1303.57 | bwd_inner_microstep: 1291.60 | bwd_allreduce_microstep: 11.88 | step_microstep: 112.70
[2024-12-05 14:38:56,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4507.87 | bwd: 8648.13 | bwd_inner: 8634.52 | bwd_allreduce: 12.52 | step: 113.73

 50%|█████     | 193/384 [55:52<1:04:41, 20.32s/it]
                                                   
{'loss': 0.1118, 'grad_norm': 1.4374881982803345, 'learning_rate': 5.211065803411134e-07, 'epoch': 1.0}

 50%|█████     | 193/384 [55:52<1:04:41, 20.32s/it][2024-12-05 14:38:57,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.92 | bwd_microstep: 426.55 | bwd_inner_microstep: 426.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:38:58,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.02 | bwd_microstep: 520.82 | bwd_inner_microstep: 520.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:38:59,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.74 | bwd_microstep: 489.85 | bwd_inner_microstep: 489.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:39:01,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.50 | bwd_microstep: 580.03 | bwd_inner_microstep: 579.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:39:02,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.36 | bwd_microstep: 409.45 | bwd_inner_microstep: 409.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:39:02,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.02 | bwd_microstep: 411.68 | bwd_inner_microstep: 411.49 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 14:39:04,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.52 | bwd_microstep: 592.33 | bwd_inner_microstep: 592.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:39:04,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.90 | bwd_microstep: 424.90 | bwd_inner_microstep: 424.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:39:06,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.95 | bwd_microstep: 449.29 | bwd_inner_microstep: 449.15 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:39:07,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.98 | bwd_microstep: 581.22 | bwd_inner_microstep: 581.08 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:39:08,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.08 | bwd_microstep: 355.70 | bwd_inner_microstep: 355.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:39:09,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.09 | bwd_microstep: 427.03 | bwd_inner_microstep: 426.88 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:39:10,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.84 | bwd_microstep: 352.71 | bwd_inner_microstep: 352.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:39:11,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.92 | bwd_microstep: 438.43 | bwd_inner_microstep: 438.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:39:12,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.39 | bwd_microstep: 443.46 | bwd_inner_microstep: 443.30 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:39:16,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.53 | optimizer_gradients: 14.02 | optimizer_step: 23.15
[2024-12-05 14:39:16,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.71 | bwd_microstep: 2492.63 | bwd_inner_microstep: 2480.66 | bwd_allreduce_microstep: 11.87 | step_microstep: 112.40
[2024-12-05 14:39:16,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4305.99 | bwd: 9396.09 | bwd_inner: 9382.37 | bwd_allreduce: 12.56 | step: 113.56

 51%|█████     | 194/384 [56:12<1:03:52, 20.17s/it]
                                                   
{'loss': 0.1603, 'grad_norm': 1.7097264528274536, 'learning_rate': 5.168870709417341e-07, 'epoch': 1.01}

 51%|█████     | 194/384 [56:12<1:03:52, 20.17s/it][2024-12-05 14:39:17,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.82 | bwd_microstep: 417.72 | bwd_inner_microstep: 417.56 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:39:18,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.25 | bwd_microstep: 458.40 | bwd_inner_microstep: 458.25 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:39:19,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.11 | bwd_microstep: 484.89 | bwd_inner_microstep: 484.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:39:20,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.09 | bwd_microstep: 405.22 | bwd_inner_microstep: 405.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:39:22,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.91 | bwd_microstep: 626.26 | bwd_inner_microstep: 626.11 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:39:23,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.53 | bwd_microstep: 583.28 | bwd_inner_microstep: 583.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:39:24,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.11 | bwd_microstep: 406.54 | bwd_inner_microstep: 406.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:39:25,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.74 | bwd_microstep: 409.79 | bwd_inner_microstep: 409.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:39:26,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.62 | bwd_microstep: 678.11 | bwd_inner_microstep: 677.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:39:27,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.70 | bwd_microstep: 432.08 | bwd_inner_microstep: 431.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:39:29,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.56 | bwd_microstep: 523.28 | bwd_inner_microstep: 523.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:39:30,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.90 | bwd_microstep: 489.49 | bwd_inner_microstep: 489.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:39:31,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.64 | bwd_microstep: 615.84 | bwd_inner_microstep: 615.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:39:32,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.38 | bwd_microstep: 446.52 | bwd_inner_microstep: 446.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:39:33,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.24 | bwd_microstep: 453.67 | bwd_inner_microstep: 453.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:39:35,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.49 | optimizer_gradients: 14.06 | optimizer_step: 23.12
[2024-12-05 14:39:35,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.13 | bwd_microstep: 1773.18 | bwd_inner_microstep: 1761.00 | bwd_allreduce_microstep: 12.08 | step_microstep: 112.10
[2024-12-05 14:39:35,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4484.78 | bwd: 9204.25 | bwd_inner: 9190.40 | bwd_allreduce: 12.74 | step: 113.26

 51%|█████     | 195/384 [56:32<1:03:00, 20.01s/it]
                                                   
{'loss': 0.128, 'grad_norm': 1.5904935598373413, 'learning_rate': 5.126663571565939e-07, 'epoch': 1.01}

 51%|█████     | 195/384 [56:32<1:03:00, 20.01s/it][2024-12-05 14:39:36,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.21 | bwd_microstep: 422.36 | bwd_inner_microstep: 422.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:39:37,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.14 | bwd_microstep: 485.75 | bwd_inner_microstep: 485.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:39:39,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.84 | bwd_microstep: 450.70 | bwd_inner_microstep: 450.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:39:39,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.25 | bwd_microstep: 411.54 | bwd_inner_microstep: 411.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:39:41,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 362.24 | bwd_microstep: 741.67 | bwd_inner_microstep: 741.54 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:39:42,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.83 | bwd_microstep: 610.31 | bwd_inner_microstep: 610.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:39:43,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.41 | bwd_microstep: 432.74 | bwd_inner_microstep: 432.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:39:44,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.99 | bwd_microstep: 545.77 | bwd_inner_microstep: 545.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:39:45,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.49 | bwd_microstep: 437.44 | bwd_inner_microstep: 437.27 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:39:46,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.83 | bwd_microstep: 482.06 | bwd_inner_microstep: 481.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:39:47,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.00 | bwd_microstep: 435.76 | bwd_inner_microstep: 435.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:39:49,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.51 | bwd_microstep: 678.42 | bwd_inner_microstep: 678.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:39:49,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.20 | bwd_microstep: 359.14 | bwd_inner_microstep: 359.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:39:50,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 242.02 | bwd_microstep: 369.87 | bwd_inner_microstep: 369.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:39:51,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.83 | bwd_microstep: 480.17 | bwd_inner_microstep: 480.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:39:54,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.73 | optimizer_gradients: 14.03 | optimizer_step: 23.15
[2024-12-05 14:39:54,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.65 | bwd_microstep: 1998.41 | bwd_inner_microstep: 1986.33 | bwd_allreduce_microstep: 11.99 | step_microstep: 112.36
[2024-12-05 14:39:54,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4516.53 | bwd: 9342.12 | bwd_inner: 9328.30 | bwd_allreduce: 12.65 | step: 113.43

 51%|█████     | 196/384 [56:50<1:01:20, 19.58s/it]
                                                   
{'loss': 0.1192, 'grad_norm': 1.5433835983276367, 'learning_rate': 5.084447400069654e-07, 'epoch': 1.02}

 51%|█████     | 196/384 [56:50<1:01:20, 19.58s/it][2024-12-05 14:39:55,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.22 | bwd_microstep: 557.87 | bwd_inner_microstep: 557.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:39:56,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.36 | bwd_microstep: 562.20 | bwd_inner_microstep: 562.04 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:39:57,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.90 | bwd_microstep: 349.89 | bwd_inner_microstep: 349.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:39:58,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.36 | bwd_microstep: 351.90 | bwd_inner_microstep: 351.75 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:39:59,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.43 | bwd_microstep: 520.29 | bwd_inner_microstep: 520.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:40:01,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.75 | bwd_microstep: 586.67 | bwd_inner_microstep: 586.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:40:01,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.87 | bwd_microstep: 326.23 | bwd_inner_microstep: 326.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:40:03,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.30 | bwd_microstep: 483.11 | bwd_inner_microstep: 482.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:40:04,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.76 | bwd_microstep: 564.53 | bwd_inner_microstep: 564.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:40:05,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.19 | bwd_microstep: 428.73 | bwd_inner_microstep: 428.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:40:06,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 396.28 | bwd_microstep: 878.88 | bwd_inner_microstep: 878.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:40:08,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.63 | bwd_microstep: 411.32 | bwd_inner_microstep: 411.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:40:09,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.46 | bwd_microstep: 435.73 | bwd_inner_microstep: 435.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:40:10,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.09 | bwd_microstep: 485.15 | bwd_inner_microstep: 485.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:40:11,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.74 | bwd_microstep: 419.54 | bwd_inner_microstep: 419.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:40:13,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.85 | optimizer_gradients: 14.06 | optimizer_step: 23.32
[2024-12-05 14:40:13,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.57 | bwd_microstep: 1795.14 | bwd_inner_microstep: 1783.13 | bwd_allreduce_microstep: 11.89 | step_microstep: 113.94
[2024-12-05 14:40:13,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4490.99 | bwd: 9157.17 | bwd_inner: 9143.52 | bwd_allreduce: 12.56 | step: 115.03

 51%|█████▏    | 197/384 [57:10<1:00:55, 19.55s/it]
                                                   
{'loss': 0.1188, 'grad_norm': 1.40073561668396, 'learning_rate': 5.042225205785492e-07, 'epoch': 1.02}

 51%|█████▏    | 197/384 [57:10<1:00:55, 19.55s/it][2024-12-05 14:40:14,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.01 | bwd_microstep: 380.27 | bwd_inner_microstep: 380.09 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:40:16,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 327.77 | bwd_microstep: 563.26 | bwd_inner_microstep: 563.12 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:40:17,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.08 | bwd_microstep: 542.49 | bwd_inner_microstep: 542.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:40:19,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.99 | bwd_microstep: 432.41 | bwd_inner_microstep: 432.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:40:20,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.86 | bwd_microstep: 588.61 | bwd_inner_microstep: 588.48 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:40:21,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.25 | bwd_microstep: 674.15 | bwd_inner_microstep: 674.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:40:22,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.81 | bwd_microstep: 411.90 | bwd_inner_microstep: 411.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:40:23,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.75 | bwd_microstep: 451.69 | bwd_inner_microstep: 451.55 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:40:24,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.40 | bwd_microstep: 491.49 | bwd_inner_microstep: 491.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:40:25,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.59 | bwd_microstep: 619.23 | bwd_inner_microstep: 619.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:40:27,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.59 | bwd_microstep: 493.46 | bwd_inner_microstep: 493.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.21
[2024-12-05 14:40:28,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.37 | bwd_microstep: 595.55 | bwd_inner_microstep: 595.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:40:29,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.04 | bwd_microstep: 554.49 | bwd_inner_microstep: 554.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:40:30,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.92 | bwd_microstep: 482.65 | bwd_inner_microstep: 482.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:40:32,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.28 | bwd_microstep: 620.71 | bwd_inner_microstep: 620.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:40:33,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.34 | optimizer_gradients: 14.02 | optimizer_step: 23.25
[2024-12-05 14:40:33,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.57 | bwd_microstep: 861.70 | bwd_inner_microstep: 849.57 | bwd_allreduce_microstep: 12.03 | step_microstep: 114.17
[2024-12-05 14:40:33,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4653.36 | bwd: 8764.03 | bwd_inner: 8750.19 | bwd_allreduce: 12.67 | step: 115.36

 52%|█████▏    | 198/384 [57:29<1:00:53, 19.65s/it]
                                                   
{'loss': 0.1486, 'grad_norm': 1.6374619007110596, 'learning_rate': 5e-07, 'epoch': 1.03}

 52%|█████▏    | 198/384 [57:29<1:00:53, 19.65s/it][2024-12-05 14:40:35,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.70 | bwd_microstep: 425.84 | bwd_inner_microstep: 425.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:40:36,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.72 | bwd_microstep: 564.50 | bwd_inner_microstep: 564.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:40:37,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.41 | bwd_microstep: 621.51 | bwd_inner_microstep: 621.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:40:38,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.30 | bwd_microstep: 457.35 | bwd_inner_microstep: 457.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:40:39,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.99 | bwd_microstep: 572.04 | bwd_inner_microstep: 571.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:40:41,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.63 | bwd_microstep: 430.84 | bwd_inner_microstep: 430.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:40:42,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.52 | bwd_microstep: 519.91 | bwd_inner_microstep: 519.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:40:43,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.12 | bwd_microstep: 621.70 | bwd_inner_microstep: 621.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:40:44,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.48 | bwd_microstep: 467.41 | bwd_inner_microstep: 467.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:40:45,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.19 | bwd_microstep: 485.26 | bwd_inner_microstep: 485.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:40:46,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.51 | bwd_microstep: 446.57 | bwd_inner_microstep: 446.43 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:40:47,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.58 | bwd_microstep: 459.30 | bwd_inner_microstep: 459.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:40:48,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.51 | bwd_microstep: 682.78 | bwd_inner_microstep: 682.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:40:50,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.78 | bwd_microstep: 636.74 | bwd_inner_microstep: 636.60 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:40:51,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.85 | bwd_microstep: 443.97 | bwd_inner_microstep: 443.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:40:53,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.13 | optimizer_gradients: 14.03 | optimizer_step: 23.39
[2024-12-05 14:40:53,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.25 | bwd_microstep: 1492.96 | bwd_inner_microstep: 1480.98 | bwd_allreduce_microstep: 11.89 | step_microstep: 111.75
[2024-12-05 14:40:53,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4620.60 | bwd: 9328.69 | bwd_inner: 9315.08 | bwd_allreduce: 12.51 | step: 112.78

 52%|█████▏    | 199/384 [57:49<1:00:30, 19.63s/it]
                                                   
{'loss': 0.1297, 'grad_norm': 1.4800790548324585, 'learning_rate': 4.957774794214508e-07, 'epoch': 1.03}

 52%|█████▏    | 199/384 [57:49<1:00:30, 19.63s/it][2024-12-05 14:40:55,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 391.38 | bwd_microstep: 853.62 | bwd_inner_microstep: 853.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:40:55,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.36 | bwd_microstep: 403.39 | bwd_inner_microstep: 403.26 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:40:56,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.50 | bwd_microstep: 438.60 | bwd_inner_microstep: 438.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:40:57,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.32 | bwd_microstep: 491.42 | bwd_inner_microstep: 491.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:40:58,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.17 | bwd_microstep: 491.34 | bwd_inner_microstep: 491.20 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:41:00,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.36 | bwd_microstep: 529.01 | bwd_inner_microstep: 528.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:00,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.85 | bwd_microstep: 427.62 | bwd_inner_microstep: 427.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:41:02,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.71 | bwd_microstep: 427.54 | bwd_inner_microstep: 427.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:02,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.24 | bwd_microstep: 455.47 | bwd_inner_microstep: 455.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:03,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.50 | bwd_microstep: 428.48 | bwd_inner_microstep: 428.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:04,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.67 | bwd_microstep: 452.06 | bwd_inner_microstep: 451.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:41:05,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.97 | bwd_microstep: 525.54 | bwd_inner_microstep: 525.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:07,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 348.77 | bwd_microstep: 689.40 | bwd_inner_microstep: 689.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:41:08,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.02 | bwd_microstep: 413.93 | bwd_inner_microstep: 413.75 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:41:09,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.22 | bwd_microstep: 435.56 | bwd_inner_microstep: 435.39 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:41:12,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.99 | optimizer_gradients: 14.00 | optimizer_step: 23.07
[2024-12-05 14:41:12,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.16 | bwd_microstep: 2465.48 | bwd_inner_microstep: 2453.61 | bwd_allreduce_microstep: 11.78 | step_microstep: 112.54
[2024-12-05 14:41:12,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4500.28 | bwd: 9928.44 | bwd_inner: 9914.78 | bwd_allreduce: 12.46 | step: 113.63

 52%|█████▏    | 200/384 [58:08<59:37, 19.44s/it]  
                                                 
{'loss': 0.1466, 'grad_norm': 1.5941686630249023, 'learning_rate': 4.915552599930345e-07, 'epoch': 1.04}

 52%|█████▏    | 200/384 [58:08<59:37, 19.44s/it][2024-12-05 14:41:13,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.38 | bwd_microstep: 458.98 | bwd_inner_microstep: 458.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:41:14,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.91 | bwd_microstep: 474.73 | bwd_inner_microstep: 474.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.05
[2024-12-05 14:41:15,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.68 | bwd_microstep: 540.53 | bwd_inner_microstep: 540.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:16,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.89 | bwd_microstep: 404.50 | bwd_inner_microstep: 404.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:41:17,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 348.48 | bwd_microstep: 707.13 | bwd_inner_microstep: 707.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:18,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 243.21 | bwd_microstep: 373.21 | bwd_inner_microstep: 373.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:19,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.79 | bwd_microstep: 392.67 | bwd_inner_microstep: 392.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:41:20,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.33 | bwd_microstep: 618.31 | bwd_inner_microstep: 618.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:21,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 361.23 | bwd_microstep: 691.33 | bwd_inner_microstep: 691.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:41:23,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.11 | bwd_microstep: 448.77 | bwd_inner_microstep: 448.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:24,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.54 | bwd_microstep: 480.57 | bwd_inner_microstep: 480.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:41:24,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.41 | bwd_microstep: 409.08 | bwd_inner_microstep: 408.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:25,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.96 | bwd_microstep: 531.70 | bwd_inner_microstep: 531.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:41:26,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.86 | bwd_microstep: 401.83 | bwd_inner_microstep: 401.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:41:27,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.87 | bwd_microstep: 587.57 | bwd_inner_microstep: 587.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:30,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.39 | optimizer_gradients: 14.02 | optimizer_step: 23.20
[2024-12-05 14:41:30,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.81 | bwd_microstep: 2046.03 | bwd_inner_microstep: 2034.10 | bwd_allreduce_microstep: 11.85 | step_microstep: 112.25
[2024-12-05 14:41:30,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4592.54 | bwd: 9566.90 | bwd_inner: 9553.34 | bwd_allreduce: 12.46 | step: 113.27

 52%|█████▏    | 201/384 [58:26<58:04, 19.04s/it]
                                                 
{'loss': 0.1264, 'grad_norm': 1.5258111953735352, 'learning_rate': 4.873336428434061e-07, 'epoch': 1.04}

 52%|█████▏    | 201/384 [58:26<58:04, 19.04s/it][2024-12-05 14:41:31,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.58 | bwd_microstep: 314.14 | bwd_inner_microstep: 314.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:41:32,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.25 | bwd_microstep: 681.34 | bwd_inner_microstep: 681.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:41:33,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.91 | bwd_microstep: 477.08 | bwd_inner_microstep: 476.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:41:34,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.56 | bwd_microstep: 426.72 | bwd_inner_microstep: 426.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:35,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.57 | bwd_microstep: 436.95 | bwd_inner_microstep: 436.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:41:36,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.85 | bwd_microstep: 593.51 | bwd_inner_microstep: 593.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:37,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.01 | bwd_microstep: 589.30 | bwd_inner_microstep: 589.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:41:38,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.27 | bwd_microstep: 435.41 | bwd_inner_microstep: 435.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:41:39,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.35 | bwd_microstep: 625.38 | bwd_inner_microstep: 625.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:41:40,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.70 | bwd_microstep: 373.76 | bwd_inner_microstep: 373.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:41,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.87 | bwd_microstep: 447.10 | bwd_inner_microstep: 446.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:41:42,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.08 | bwd_microstep: 489.02 | bwd_inner_microstep: 488.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:41:43,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.90 | bwd_microstep: 437.69 | bwd_inner_microstep: 437.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:41:44,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.95 | bwd_microstep: 431.76 | bwd_inner_microstep: 431.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:45,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.40 | bwd_microstep: 609.27 | bwd_inner_microstep: 609.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:47,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.04 | optimizer_gradients: 13.99 | optimizer_step: 23.35
[2024-12-05 14:41:47,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.22 | bwd_microstep: 1416.76 | bwd_inner_microstep: 1404.75 | bwd_allreduce_microstep: 11.91 | step_microstep: 111.71
[2024-12-05 14:41:47,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4457.53 | bwd: 8785.18 | bwd_inner: 8771.55 | bwd_allreduce: 12.52 | step: 112.72

 53%|█████▎    | 202/384 [58:43<56:05, 18.49s/it]
                                                 
{'loss': 0.1334, 'grad_norm': 1.725998878479004, 'learning_rate': 4.831129290582659e-07, 'epoch': 1.05}

 53%|█████▎    | 202/384 [58:43<56:05, 18.49s/it][2024-12-05 14:41:48,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.94 | bwd_microstep: 401.05 | bwd_inner_microstep: 400.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:41:49,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.51 | bwd_microstep: 437.09 | bwd_inner_microstep: 436.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:50,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.12 | bwd_microstep: 408.07 | bwd_inner_microstep: 407.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:51,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.55 | bwd_microstep: 451.67 | bwd_inner_microstep: 451.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:52,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.30 | bwd_microstep: 405.31 | bwd_inner_microstep: 405.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:41:53,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.39 | bwd_microstep: 474.70 | bwd_inner_microstep: 474.52 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:41:54,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.45 | bwd_microstep: 443.24 | bwd_inner_microstep: 443.06 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:41:55,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.14 | bwd_microstep: 612.82 | bwd_inner_microstep: 612.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:41:56,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.96 | bwd_microstep: 541.94 | bwd_inner_microstep: 541.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:41:57,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.75 | bwd_microstep: 402.65 | bwd_inner_microstep: 402.50 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:41:58,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 382.94 | bwd_microstep: 835.43 | bwd_inner_microstep: 835.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:41:59,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.40 | bwd_microstep: 439.26 | bwd_inner_microstep: 439.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:42:00,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.03 | bwd_microstep: 460.66 | bwd_inner_microstep: 460.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:42:01,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.84 | bwd_microstep: 471.80 | bwd_inner_microstep: 471.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:02,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.19 | bwd_microstep: 563.22 | bwd_inner_microstep: 563.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:05,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.77 | optimizer_gradients: 14.02 | optimizer_step: 23.11
[2024-12-05 14:42:05,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.58 | bwd_microstep: 1613.14 | bwd_inner_microstep: 1601.21 | bwd_allreduce_microstep: 11.84 | step_microstep: 112.38
[2024-12-05 14:42:05,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4535.16 | bwd: 8962.04 | bwd_inner: 8948.42 | bwd_allreduce: 12.49 | step: 113.44

 53%|█████▎    | 203/384 [59:01<55:04, 18.26s/it]
                                                 
{'loss': 0.1503, 'grad_norm': 1.8626656532287598, 'learning_rate': 4.788934196588865e-07, 'epoch': 1.06}

 53%|█████▎    | 203/384 [59:01<55:04, 18.26s/it][2024-12-05 14:42:06,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.95 | bwd_microstep: 374.59 | bwd_inner_microstep: 374.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:07,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 383.39 | bwd_microstep: 832.50 | bwd_inner_microstep: 832.35 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:42:08,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.34 | bwd_microstep: 346.16 | bwd_inner_microstep: 346.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:09,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.06 | bwd_microstep: 565.32 | bwd_inner_microstep: 565.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:10,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.05 | bwd_microstep: 494.81 | bwd_inner_microstep: 494.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:42:12,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 396.54 | bwd_microstep: 883.63 | bwd_inner_microstep: 883.49 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:42:13,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 360.39 | bwd_microstep: 752.49 | bwd_inner_microstep: 752.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:14,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.95 | bwd_microstep: 485.50 | bwd_inner_microstep: 485.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:15,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.72 | bwd_microstep: 435.27 | bwd_inner_microstep: 435.09 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:42:16,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.40 | bwd_microstep: 547.38 | bwd_inner_microstep: 547.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:42:17,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.29 | bwd_microstep: 503.19 | bwd_inner_microstep: 503.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:42:18,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.43 | bwd_microstep: 416.56 | bwd_inner_microstep: 416.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:19,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.12 | bwd_microstep: 633.63 | bwd_inner_microstep: 633.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:20,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.81 | bwd_microstep: 522.38 | bwd_inner_microstep: 522.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:21,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.48 | bwd_microstep: 478.96 | bwd_inner_microstep: 478.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:42:23,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.27 | optimizer_gradients: 14.03 | optimizer_step: 23.12
[2024-12-05 14:42:23,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.46 | bwd_microstep: 1211.43 | bwd_inner_microstep: 1199.27 | bwd_allreduce_microstep: 12.07 | step_microstep: 111.84
[2024-12-05 14:42:23,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4793.44 | bwd: 9483.83 | bwd_inner: 9469.98 | bwd_allreduce: 12.71 | step: 112.87

 53%|█████▎    | 204/384 [59:19<54:19, 18.11s/it]
                                                 
{'loss': 0.1225, 'grad_norm': 1.4593480825424194, 'learning_rate': 4.746754155806437e-07, 'epoch': 1.06}

 53%|█████▎    | 204/384 [59:19<54:19, 18.11s/it][2024-12-05 14:42:24,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.58 | bwd_microstep: 419.42 | bwd_inner_microstep: 419.27 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:42:24,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.63 | bwd_microstep: 443.21 | bwd_inner_microstep: 443.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:42:25,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.26 | bwd_microstep: 452.27 | bwd_inner_microstep: 452.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:26,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.37 | bwd_microstep: 430.54 | bwd_inner_microstep: 430.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:27,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.48 | bwd_microstep: 617.93 | bwd_inner_microstep: 617.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:28,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.10 | bwd_microstep: 463.75 | bwd_inner_microstep: 463.60 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:42:29,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.79 | bwd_microstep: 593.39 | bwd_inner_microstep: 593.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:30,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.14 | bwd_microstep: 337.29 | bwd_inner_microstep: 337.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:42:31,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.81 | bwd_microstep: 449.57 | bwd_inner_microstep: 449.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:42:32,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.79 | bwd_microstep: 415.69 | bwd_inner_microstep: 415.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:33,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.80 | bwd_microstep: 520.31 | bwd_inner_microstep: 520.18 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:42:34,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.83 | bwd_microstep: 424.83 | bwd_inner_microstep: 424.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:35,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 356.75 | bwd_microstep: 704.80 | bwd_inner_microstep: 704.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:36,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 328.85 | bwd_microstep: 621.46 | bwd_inner_microstep: 621.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:37,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.24 | bwd_microstep: 432.80 | bwd_inner_microstep: 432.66 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:42:40,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.28 | optimizer_gradients: 14.01 | optimizer_step: 23.25
[2024-12-05 14:42:40,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.71 | bwd_microstep: 2286.19 | bwd_inner_microstep: 2274.23 | bwd_allreduce_microstep: 11.85 | step_microstep: 112.17
[2024-12-05 14:42:40,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4552.22 | bwd: 9613.42 | bwd_inner: 9599.79 | bwd_allreduce: 12.51 | step: 113.20

 53%|█████▎    | 205/384 [59:36<53:23, 17.90s/it]
                                                 
{'loss': 0.1397, 'grad_norm': 1.6284171342849731, 'learning_rate': 4.7045921765155337e-07, 'epoch': 1.07}

 53%|█████▎    | 205/384 [59:36<53:23, 17.90s/it][2024-12-05 14:42:41,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.72 | bwd_microstep: 647.39 | bwd_inner_microstep: 647.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:43,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.64 | bwd_microstep: 677.75 | bwd_inner_microstep: 677.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:44,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.01 | bwd_microstep: 621.57 | bwd_inner_microstep: 621.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:42:45,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.44 | bwd_microstep: 446.28 | bwd_inner_microstep: 446.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:42:46,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.11 | bwd_microstep: 416.21 | bwd_inner_microstep: 416.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:42:47,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 357.37 | bwd_microstep: 741.53 | bwd_inner_microstep: 741.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:42:48,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.70 | bwd_microstep: 357.67 | bwd_inner_microstep: 357.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:42:49,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.50 | bwd_microstep: 392.55 | bwd_inner_microstep: 392.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:50,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.25 | bwd_microstep: 592.67 | bwd_inner_microstep: 592.10 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.25
[2024-12-05 14:42:52,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.62 | bwd_microstep: 708.94 | bwd_inner_microstep: 708.80 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:42:53,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.07 | bwd_microstep: 432.67 | bwd_inner_microstep: 432.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:53,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.84 | bwd_microstep: 431.48 | bwd_inner_microstep: 431.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:54,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.47 | bwd_microstep: 409.49 | bwd_inner_microstep: 409.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:42:55,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 349.77 | bwd_microstep: 708.09 | bwd_inner_microstep: 707.93 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.06
[2024-12-05 14:42:56,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.62 | bwd_microstep: 450.61 | bwd_inner_microstep: 450.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:42:58,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.63 | optimizer_gradients: 14.08 | optimizer_step: 23.15
[2024-12-05 14:42:58,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.14 | bwd_microstep: 1167.80 | bwd_inner_microstep: 1155.78 | bwd_allreduce_microstep: 11.92 | step_microstep: 114.72
[2024-12-05 14:42:58,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4697.32 | bwd: 9202.72 | bwd_inner: 9188.63 | bwd_allreduce: 12.77 | step: 115.93

 54%|█████▎    | 206/384 [59:54<53:11, 17.93s/it]
                                                 
{'loss': 0.1082, 'grad_norm': 1.4469428062438965, 'learning_rate': 4.662451265708174e-07, 'epoch': 1.07}

 54%|█████▎    | 206/384 [59:54<53:11, 17.93s/it][2024-12-05 14:42:59,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.30 | bwd_microstep: 602.69 | bwd_inner_microstep: 602.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:43:00,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.03 | bwd_microstep: 461.92 | bwd_inner_microstep: 461.78 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:43:01,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.68 | bwd_microstep: 480.38 | bwd_inner_microstep: 480.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:43:02,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.66 | bwd_microstep: 380.57 | bwd_inner_microstep: 380.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:43:02,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.94 | bwd_microstep: 401.78 | bwd_inner_microstep: 401.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:43:03,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.03 | bwd_microstep: 413.29 | bwd_inner_microstep: 413.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:43:04,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.87 | bwd_microstep: 532.48 | bwd_inner_microstep: 532.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:43:05,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.19 | bwd_microstep: 622.63 | bwd_inner_microstep: 622.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:43:06,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.60 | bwd_microstep: 347.42 | bwd_inner_microstep: 347.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:43:07,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 351.66 | bwd_microstep: 716.24 | bwd_inner_microstep: 716.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:43:08,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.62 | bwd_microstep: 434.95 | bwd_inner_microstep: 434.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:43:09,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.02 | bwd_microstep: 432.31 | bwd_inner_microstep: 432.16 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:43:10,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.37 | bwd_microstep: 595.53 | bwd_inner_microstep: 595.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:43:11,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.59 | bwd_microstep: 449.15 | bwd_inner_microstep: 449.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:43:12,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.94 | bwd_microstep: 568.40 | bwd_inner_microstep: 568.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:43:15,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.23 | optimizer_gradients: 14.01 | optimizer_step: 23.17
[2024-12-05 14:43:15,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 344.62 | bwd_microstep: 2570.54 | bwd_inner_microstep: 2558.40 | bwd_allreduce_microstep: 12.04 | step_microstep: 113.78
[2024-12-05 14:43:15,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4613.19 | bwd: 10010.28 | bwd_inner: 9996.45 | bwd_allreduce: 12.67 | step: 114.78

 54%|█████▍    | 207/384 [1:00:11<52:16, 17.72s/it]
                                                   
{'loss': 0.1025, 'grad_norm': 1.3986436128616333, 'learning_rate': 4.620334428873769e-07, 'epoch': 1.08}

 54%|█████▍    | 207/384 [1:00:11<52:16, 17.72s/it][2024-12-05 14:43:16,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.52 | bwd_microstep: 431.05 | bwd_inner_microstep: 430.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:43:17,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.65 | bwd_microstep: 403.64 | bwd_inner_microstep: 403.50 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:43:18,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 379.92 | bwd_microstep: 828.74 | bwd_inner_microstep: 828.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:43:19,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.47 | bwd_microstep: 458.37 | bwd_inner_microstep: 458.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:43:20,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.21 | bwd_microstep: 439.70 | bwd_inner_microstep: 439.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:43:21,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.26 | bwd_microstep: 688.82 | bwd_inner_microstep: 688.68 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:43:22,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.71 | bwd_microstep: 412.66 | bwd_inner_microstep: 412.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:43:23,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.97 | bwd_microstep: 679.56 | bwd_inner_microstep: 679.41 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:43:24,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.21 | bwd_microstep: 589.52 | bwd_inner_microstep: 589.35 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 14:43:25,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.48 | bwd_microstep: 429.35 | bwd_inner_microstep: 429.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:43:26,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.61 | bwd_microstep: 482.32 | bwd_inner_microstep: 482.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:43:27,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.78 | bwd_microstep: 452.32 | bwd_inner_microstep: 452.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:43:28,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.22 | bwd_microstep: 538.86 | bwd_inner_microstep: 538.72 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:43:29,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.49 | bwd_microstep: 447.30 | bwd_inner_microstep: 447.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:43:30,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.14 | bwd_microstep: 538.99 | bwd_inner_microstep: 538.84 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:43:32,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.16 | optimizer_gradients: 14.01 | optimizer_step: 23.29
[2024-12-05 14:43:32,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.56 | bwd_microstep: 1520.67 | bwd_inner_microstep: 1508.51 | bwd_allreduce_microstep: 12.05 | step_microstep: 112.06
[2024-12-05 14:43:32,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4564.26 | bwd: 9341.88 | bwd_inner: 9328.03 | bwd_allreduce: 12.74 | step: 113.28

 54%|█████▍    | 208/384 [1:00:28<51:08, 17.43s/it]
                                                   
{'loss': 0.126, 'grad_norm': 1.6322413682937622, 'learning_rate': 4.5782446697847764e-07, 'epoch': 1.08}

 54%|█████▍    | 208/384 [1:00:28<51:08, 17.43s/it][2024-12-05 14:43:33,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.72 | bwd_microstep: 323.86 | bwd_inner_microstep: 323.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:43:34,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.70 | bwd_microstep: 484.18 | bwd_inner_microstep: 484.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:43:35,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.46 | bwd_microstep: 459.86 | bwd_inner_microstep: 459.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:43:36,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.09 | bwd_microstep: 440.64 | bwd_inner_microstep: 440.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:43:37,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.76 | bwd_microstep: 678.41 | bwd_inner_microstep: 678.26 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:43:38,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.35 | bwd_microstep: 478.16 | bwd_inner_microstep: 478.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:43:39,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.67 | bwd_microstep: 682.41 | bwd_inner_microstep: 682.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:43:40,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.64 | bwd_microstep: 597.25 | bwd_inner_microstep: 597.07 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:43:41,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.00 | bwd_microstep: 611.49 | bwd_inner_microstep: 611.35 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:43:43,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.95 | bwd_microstep: 592.80 | bwd_inner_microstep: 592.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:43:43,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.31 | bwd_microstep: 405.72 | bwd_inner_microstep: 405.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:43:44,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.33 | bwd_microstep: 398.66 | bwd_inner_microstep: 398.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:43:45,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.99 | bwd_microstep: 401.06 | bwd_inner_microstep: 400.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:43:46,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.64 | bwd_microstep: 448.99 | bwd_inner_microstep: 448.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:43:47,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.40 | bwd_microstep: 399.13 | bwd_inner_microstep: 399.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:43:49,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.30 | optimizer_gradients: 14.04 | optimizer_step: 23.46
[2024-12-05 14:43:49,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.61 | bwd_microstep: 1578.00 | bwd_inner_microstep: 1566.03 | bwd_allreduce_microstep: 11.89 | step_microstep: 112.64
[2024-12-05 14:43:49,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4487.73 | bwd: 8980.60 | bwd_inner: 8966.93 | bwd_allreduce: 12.55 | step: 113.71

 54%|█████▍    | 209/384 [1:00:46<50:44, 17.40s/it]
                                                   
{'loss': 0.0932, 'grad_norm': 1.4022878408432007, 'learning_rate': 4.536184990282467e-07, 'epoch': 1.09}

 54%|█████▍    | 209/384 [1:00:46<50:44, 17.40s/it][2024-12-05 14:43:50,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.31 | bwd_microstep: 348.62 | bwd_inner_microstep: 348.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:43:51,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.64 | bwd_microstep: 529.48 | bwd_inner_microstep: 529.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:43:52,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.97 | bwd_microstep: 486.59 | bwd_inner_microstep: 486.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:43:53,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 348.50 | bwd_microstep: 709.52 | bwd_inner_microstep: 709.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:43:54,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.46 | bwd_microstep: 493.49 | bwd_inner_microstep: 493.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:43:56,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.96 | bwd_microstep: 675.79 | bwd_inner_microstep: 675.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:43:57,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.29 | bwd_microstep: 399.79 | bwd_inner_microstep: 399.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:43:58,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.57 | bwd_microstep: 445.82 | bwd_inner_microstep: 445.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:43:59,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.11 | bwd_microstep: 431.47 | bwd_inner_microstep: 431.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:44:00,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.37 | bwd_microstep: 597.22 | bwd_inner_microstep: 597.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:44:01,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 325.55 | bwd_microstep: 589.19 | bwd_inner_microstep: 589.05 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:44:02,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.12 | bwd_microstep: 382.45 | bwd_inner_microstep: 382.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:44:02,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.58 | bwd_microstep: 426.19 | bwd_inner_microstep: 426.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:44:03,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.99 | bwd_microstep: 396.13 | bwd_inner_microstep: 396.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:44:04,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.70 | bwd_microstep: 474.75 | bwd_inner_microstep: 474.61 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:44:07,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.09 | optimizer_gradients: 14.06 | optimizer_step: 23.37
[2024-12-05 14:44:07,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.81 | bwd_microstep: 2161.57 | bwd_inner_microstep: 2149.56 | bwd_allreduce_microstep: 11.92 | step_microstep: 112.87
[2024-12-05 14:44:07,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4509.97 | bwd: 9548.05 | bwd_inner: 9534.44 | bwd_allreduce: 12.54 | step: 113.85

 55%|█████▍    | 210/384 [1:01:03<50:29, 17.41s/it]
                                                   
{'loss': 0.1662, 'grad_norm': 1.8021215200424194, 'learning_rate': 4.4941583900628393e-07, 'epoch': 1.09}

 55%|█████▍    | 210/384 [1:01:03<50:29, 17.41s/it][2024-12-05 14:44:08,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.17 | bwd_microstep: 417.08 | bwd_inner_microstep: 416.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:44:09,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.08 | bwd_microstep: 429.55 | bwd_inner_microstep: 429.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:44:09,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.66 | bwd_microstep: 411.62 | bwd_inner_microstep: 411.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:44:10,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.78 | bwd_microstep: 404.03 | bwd_inner_microstep: 403.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:44:12,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.18 | bwd_microstep: 567.26 | bwd_inner_microstep: 567.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:44:12,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.26 | bwd_microstep: 403.46 | bwd_inner_microstep: 403.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:44:13,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.05 | bwd_microstep: 487.21 | bwd_inner_microstep: 487.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:44:14,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.70 | bwd_microstep: 406.98 | bwd_inner_microstep: 406.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:44:15,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.78 | bwd_microstep: 428.70 | bwd_inner_microstep: 428.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:44:16,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.98 | bwd_microstep: 432.84 | bwd_inner_microstep: 432.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:44:17,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.03 | bwd_microstep: 408.03 | bwd_inner_microstep: 407.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:44:18,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.20 | bwd_microstep: 360.08 | bwd_inner_microstep: 359.93 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.06
[2024-12-05 14:44:19,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.93 | bwd_microstep: 402.36 | bwd_inner_microstep: 402.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:44:20,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.39 | bwd_microstep: 448.43 | bwd_inner_microstep: 448.30 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:44:21,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.86 | bwd_microstep: 403.67 | bwd_inner_microstep: 403.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:44:25,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.48 | optimizer_gradients: 14.06 | optimizer_step: 23.41
[2024-12-05 14:44:25,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.69 | bwd_microstep: 3760.41 | bwd_inner_microstep: 3748.34 | bwd_allreduce_microstep: 11.97 | step_microstep: 113.15
[2024-12-05 14:44:25,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4188.82 | bwd: 10171.69 | bwd_inner: 10157.96 | bwd_allreduce: 12.62 | step: 114.12

 55%|█████▍    | 211/384 [1:01:21<50:57, 17.68s/it]
                                                   
{'loss': 0.1574, 'grad_norm': 1.806060552597046, 'learning_rate': 4.4521678664626745e-07, 'epoch': 1.1}

 55%|█████▍    | 211/384 [1:01:21<50:57, 17.68s/it][2024-12-05 14:44:26,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.67 | bwd_microstep: 319.79 | bwd_inner_microstep: 319.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:44:27,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.17 | bwd_microstep: 582.97 | bwd_inner_microstep: 582.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:44:28,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 350.41 | bwd_microstep: 713.26 | bwd_inner_microstep: 713.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:44:29,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 228.08 | bwd_microstep: 339.07 | bwd_inner_microstep: 338.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:44:30,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.95 | bwd_microstep: 605.90 | bwd_inner_microstep: 605.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:44:31,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.99 | bwd_microstep: 486.99 | bwd_inner_microstep: 486.86 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:44:32,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.97 | bwd_microstep: 462.60 | bwd_inner_microstep: 462.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:44:33,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.14 | bwd_microstep: 458.50 | bwd_inner_microstep: 458.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:44:34,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.22 | bwd_microstep: 486.28 | bwd_inner_microstep: 486.14 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:44:35,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.52 | bwd_microstep: 436.16 | bwd_inner_microstep: 436.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:44:36,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.09 | bwd_microstep: 475.49 | bwd_inner_microstep: 475.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:44:37,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.43 | bwd_microstep: 425.85 | bwd_inner_microstep: 425.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:44:38,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 227.28 | bwd_microstep: 337.48 | bwd_inner_microstep: 337.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:44:39,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.21 | bwd_microstep: 424.74 | bwd_inner_microstep: 424.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:44:40,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.21 | bwd_microstep: 528.71 | bwd_inner_microstep: 528.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:44:43,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.49 | optimizer_gradients: 13.99 | optimizer_step: 23.15
[2024-12-05 14:44:43,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.57 | bwd_microstep: 2539.77 | bwd_inner_microstep: 2527.65 | bwd_allreduce_microstep: 12.03 | step_microstep: 112.05
[2024-12-05 14:44:43,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4372.97 | bwd: 9623.56 | bwd_inner: 9609.79 | bwd_allreduce: 12.65 | step: 113.14

 55%|█████▌    | 212/384 [1:01:39<50:38, 17.66s/it]
                                                   
{'loss': 0.1467, 'grad_norm': 1.7470866441726685, 'learning_rate': 4.4102164142457705e-07, 'epoch': 1.1}

 55%|█████▌    | 212/384 [1:01:39<50:38, 17.66s/it][2024-12-05 14:44:44,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.63 | bwd_microstep: 556.23 | bwd_inner_microstep: 556.08 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:44:45,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.97 | bwd_microstep: 451.29 | bwd_inner_microstep: 451.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:44:46,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.73 | bwd_microstep: 562.20 | bwd_inner_microstep: 562.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:44:48,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.42 | bwd_microstep: 566.38 | bwd_inner_microstep: 566.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:44:49,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.48 | bwd_microstep: 405.85 | bwd_inner_microstep: 405.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:44:50,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.23 | bwd_microstep: 397.83 | bwd_inner_microstep: 397.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:44:50,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.83 | bwd_microstep: 413.83 | bwd_inner_microstep: 413.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:44:51,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.39 | bwd_microstep: 454.12 | bwd_inner_microstep: 453.98 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:44:53,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 343.65 | bwd_microstep: 712.83 | bwd_inner_microstep: 712.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:44:54,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.15 | bwd_microstep: 360.02 | bwd_inner_microstep: 359.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:44:55,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.31 | bwd_microstep: 433.99 | bwd_inner_microstep: 433.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:44:56,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.60 | bwd_microstep: 401.17 | bwd_inner_microstep: 401.02 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:44:57,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.39 | bwd_microstep: 395.52 | bwd_inner_microstep: 395.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:44:58,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.75 | bwd_microstep: 357.30 | bwd_inner_microstep: 357.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:44:59,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.45 | bwd_microstep: 485.70 | bwd_inner_microstep: 485.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:02,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.41 | optimizer_gradients: 14.03 | optimizer_step: 23.50
[2024-12-05 14:45:02,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.96 | bwd_microstep: 1948.83 | bwd_inner_microstep: 1936.76 | bwd_allreduce_microstep: 11.98 | step_microstep: 112.73
[2024-12-05 14:45:02,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4306.98 | bwd: 8903.09 | bwd_inner: 8889.40 | bwd_allreduce: 12.62 | step: 113.90

 55%|█████▌    | 213/384 [1:01:58<51:36, 18.11s/it]
                                                   
{'loss': 0.1084, 'grad_norm': 1.5723119974136353, 'learning_rate': 4.368307025389355e-07, 'epoch': 1.11}

 55%|█████▌    | 213/384 [1:01:58<51:36, 18.11s/it][2024-12-05 14:45:03,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.75 | bwd_microstep: 412.78 | bwd_inner_microstep: 412.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:04,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.44 | bwd_microstep: 445.15 | bwd_inner_microstep: 445.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:06,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.90 | bwd_microstep: 438.52 | bwd_inner_microstep: 438.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:07,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.36 | bwd_microstep: 381.03 | bwd_inner_microstep: 380.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:08,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.64 | bwd_microstep: 488.10 | bwd_inner_microstep: 487.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:09,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.58 | bwd_microstep: 446.22 | bwd_inner_microstep: 446.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:10,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.20 | bwd_microstep: 691.91 | bwd_inner_microstep: 691.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:45:11,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 239.24 | bwd_microstep: 357.57 | bwd_inner_microstep: 357.43 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:45:12,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.01 | bwd_microstep: 616.49 | bwd_inner_microstep: 616.36 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:45:13,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.15 | bwd_microstep: 582.21 | bwd_inner_microstep: 582.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:45:15,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.63 | bwd_microstep: 481.78 | bwd_inner_microstep: 481.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:45:15,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.12 | bwd_microstep: 413.43 | bwd_inner_microstep: 413.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:45:16,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.50 | bwd_microstep: 538.01 | bwd_inner_microstep: 537.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:18,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.02 | bwd_microstep: 594.17 | bwd_inner_microstep: 594.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:19,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.62 | bwd_microstep: 405.47 | bwd_inner_microstep: 405.35 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:45:21,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.38 | optimizer_gradients: 14.03 | optimizer_step: 23.47
[2024-12-05 14:45:21,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.79 | bwd_microstep: 1516.48 | bwd_inner_microstep: 1504.46 | bwd_allreduce_microstep: 11.92 | step_microstep: 112.55
[2024-12-05 14:45:21,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4494.01 | bwd: 8809.29 | bwd_inner: 8795.58 | bwd_allreduce: 12.55 | step: 113.55

 56%|█████▌    | 214/384 [1:02:17<52:05, 18.38s/it]
                                                   
{'loss': 0.116, 'grad_norm': 1.5377353429794312, 'learning_rate': 4.326442688870696e-07, 'epoch': 1.11}

 56%|█████▌    | 214/384 [1:02:17<52:05, 18.38s/it][2024-12-05 14:45:22,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.94 | bwd_microstep: 371.39 | bwd_inner_microstep: 371.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:45:23,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.30 | bwd_microstep: 475.07 | bwd_inner_microstep: 474.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:24,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.65 | bwd_microstep: 496.61 | bwd_inner_microstep: 496.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:25,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.64 | bwd_microstep: 430.28 | bwd_inner_microstep: 430.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:26,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.80 | bwd_microstep: 457.30 | bwd_inner_microstep: 457.13 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:45:27,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.17 | bwd_microstep: 613.55 | bwd_inner_microstep: 613.39 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:45:29,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.90 | bwd_microstep: 518.02 | bwd_inner_microstep: 517.86 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:45:29,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.55 | bwd_microstep: 352.41 | bwd_inner_microstep: 352.29 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:45:31,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.87 | bwd_microstep: 450.83 | bwd_inner_microstep: 450.66 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:45:32,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.62 | bwd_microstep: 455.65 | bwd_inner_microstep: 455.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:33,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.04 | bwd_microstep: 490.38 | bwd_inner_microstep: 490.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:34,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.97 | bwd_microstep: 477.13 | bwd_inner_microstep: 477.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:35,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.87 | bwd_microstep: 488.46 | bwd_inner_microstep: 488.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:36,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.35 | bwd_microstep: 440.14 | bwd_inner_microstep: 440.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:38,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.72 | bwd_microstep: 618.22 | bwd_inner_microstep: 618.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:39,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.91 | optimizer_gradients: 14.04 | optimizer_step: 23.40
[2024-12-05 14:45:39,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.73 | bwd_microstep: 871.71 | bwd_inner_microstep: 859.68 | bwd_allreduce_microstep: 11.93 | step_microstep: 112.05
[2024-12-05 14:45:39,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4402.20 | bwd: 8007.18 | bwd_inner: 7993.41 | bwd_allreduce: 12.60 | step: 113.09

 56%|█████▌    | 215/384 [1:02:36<51:51, 18.41s/it]
                                                   
{'loss': 0.1483, 'grad_norm': 1.7077683210372925, 'learning_rate': 4.2846263904539303e-07, 'epoch': 1.12}

 56%|█████▌    | 215/384 [1:02:36<51:51, 18.41s/it][2024-12-05 14:45:40,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.95 | bwd_microstep: 428.42 | bwd_inner_microstep: 428.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:45:41,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.36 | bwd_microstep: 441.24 | bwd_inner_microstep: 441.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:43,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.18 | bwd_microstep: 482.66 | bwd_inner_microstep: 482.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:44,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.22 | bwd_microstep: 443.40 | bwd_inner_microstep: 443.24 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.06
[2024-12-05 14:45:45,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.72 | bwd_microstep: 625.70 | bwd_inner_microstep: 625.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:45:46,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.02 | bwd_microstep: 355.35 | bwd_inner_microstep: 355.22 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:45:47,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.40 | bwd_microstep: 456.94 | bwd_inner_microstep: 456.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:48,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.00 | bwd_microstep: 521.57 | bwd_inner_microstep: 521.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:49,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.96 | bwd_microstep: 439.05 | bwd_inner_microstep: 438.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:50,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.14 | bwd_microstep: 478.41 | bwd_inner_microstep: 478.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:51,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.06 | bwd_microstep: 383.28 | bwd_inner_microstep: 383.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:52,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.94 | bwd_microstep: 426.17 | bwd_inner_microstep: 426.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:53,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.27 | bwd_microstep: 537.78 | bwd_inner_microstep: 537.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:54,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.24 | bwd_microstep: 451.73 | bwd_inner_microstep: 451.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:45:55,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.70 | bwd_microstep: 478.96 | bwd_inner_microstep: 478.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:45:58,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.56 | optimizer_gradients: 14.05 | optimizer_step: 23.12
[2024-12-05 14:45:58,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.83 | bwd_microstep: 2306.71 | bwd_inner_microstep: 2294.60 | bwd_allreduce_microstep: 12.04 | step_microstep: 112.15
[2024-12-05 14:45:58,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4386.04 | bwd: 9257.34 | bwd_inner: 9243.56 | bwd_allreduce: 12.66 | step: 113.17

 56%|█████▋    | 216/384 [1:02:54<51:56, 18.55s/it]
                                                   
{'loss': 0.1527, 'grad_norm': 1.8333083391189575, 'learning_rate': 4.242861112477118e-07, 'epoch': 1.12}

 56%|█████▋    | 216/384 [1:02:54<51:56, 18.55s/it][2024-12-05 14:45:59,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.23 | bwd_microstep: 390.13 | bwd_inner_microstep: 390.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:00,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.70 | bwd_microstep: 422.71 | bwd_inner_microstep: 422.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:46:01,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.00 | bwd_microstep: 341.64 | bwd_inner_microstep: 341.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:03,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.11 | bwd_microstep: 682.94 | bwd_inner_microstep: 682.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:04,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.59 | bwd_microstep: 682.20 | bwd_inner_microstep: 682.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:46:05,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.84 | bwd_microstep: 544.05 | bwd_inner_microstep: 543.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:46:06,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.71 | bwd_microstep: 429.91 | bwd_inner_microstep: 429.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:46:07,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.15 | bwd_microstep: 558.10 | bwd_inner_microstep: 557.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:46:08,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.82 | bwd_microstep: 459.15 | bwd_inner_microstep: 459.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:09,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.83 | bwd_microstep: 357.77 | bwd_inner_microstep: 357.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:46:10,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.66 | bwd_microstep: 402.04 | bwd_inner_microstep: 401.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:46:11,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.02 | bwd_microstep: 418.06 | bwd_inner_microstep: 417.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:46:12,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.52 | bwd_microstep: 400.84 | bwd_inner_microstep: 400.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:46:13,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.30 | bwd_microstep: 448.37 | bwd_inner_microstep: 448.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:46:15,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.15 | bwd_microstep: 690.09 | bwd_inner_microstep: 689.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:46:16,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.68 | optimizer_gradients: 14.17 | optimizer_step: 23.30
[2024-12-05 14:46:16,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.47 | bwd_microstep: 1161.21 | bwd_inner_microstep: 1148.90 | bwd_allreduce_microstep: 12.19 | step_microstep: 114.46
[2024-12-05 14:46:16,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4454.18 | bwd: 8389.18 | bwd_inner: 8375.22 | bwd_allreduce: 12.81 | step: 115.49

 57%|█████▋    | 217/384 [1:03:13<51:21, 18.45s/it]
                                                   
{'loss': 0.121, 'grad_norm': 1.5769239664077759, 'learning_rate': 4.201149833639539e-07, 'epoch': 1.13}

 57%|█████▋    | 217/384 [1:03:13<51:21, 18.45s/it][2024-12-05 14:46:18,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.82 | bwd_microstep: 413.11 | bwd_inner_microstep: 412.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:46:19,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 393.26 | bwd_microstep: 873.42 | bwd_inner_microstep: 873.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:46:21,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 360.30 | bwd_microstep: 747.68 | bwd_inner_microstep: 747.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:22,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.28 | bwd_microstep: 459.89 | bwd_inner_microstep: 459.72 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:46:23,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.64 | bwd_microstep: 553.28 | bwd_inner_microstep: 553.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:24,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.92 | bwd_microstep: 452.88 | bwd_inner_microstep: 452.73 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:46:25,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.61 | bwd_microstep: 427.47 | bwd_inner_microstep: 427.32 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:46:26,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.63 | bwd_microstep: 693.63 | bwd_inner_microstep: 693.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:27,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.32 | bwd_microstep: 484.53 | bwd_inner_microstep: 484.36 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 14:46:28,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.05 | bwd_microstep: 408.11 | bwd_inner_microstep: 407.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:46:29,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.98 | bwd_microstep: 413.53 | bwd_inner_microstep: 413.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:46:30,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.00 | bwd_microstep: 351.62 | bwd_inner_microstep: 351.48 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:46:31,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.13 | bwd_microstep: 586.40 | bwd_inner_microstep: 586.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:46:32,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.06 | bwd_microstep: 458.22 | bwd_inner_microstep: 458.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:33,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.08 | bwd_microstep: 437.33 | bwd_inner_microstep: 437.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:35,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.31 | optimizer_gradients: 14.05 | optimizer_step: 23.11
[2024-12-05 14:46:35,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.65 | bwd_microstep: 1004.98 | bwd_inner_microstep: 992.79 | bwd_allreduce_microstep: 12.07 | step_microstep: 111.95
[2024-12-05 14:46:35,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4659.79 | bwd: 8766.07 | bwd_inner: 8752.11 | bwd_allreduce: 12.77 | step: 113.03

 57%|█████▋    | 218/384 [1:03:31<50:50, 18.38s/it]
                                                   
{'loss': 0.173, 'grad_norm': 1.919921636581421, 'learning_rate': 4.15949552878926e-07, 'epoch': 1.13}

 57%|█████▋    | 218/384 [1:03:31<50:50, 18.38s/it][2024-12-05 14:46:35,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.46 | bwd_microstep: 368.81 | bwd_inner_microstep: 368.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:36,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.38 | bwd_microstep: 431.33 | bwd_inner_microstep: 431.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:46:37,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.16 | bwd_microstep: 356.58 | bwd_inner_microstep: 356.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:38,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.64 | bwd_microstep: 528.81 | bwd_inner_microstep: 528.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:39,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.53 | bwd_microstep: 437.29 | bwd_inner_microstep: 437.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:46:40,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.75 | bwd_microstep: 436.91 | bwd_inner_microstep: 436.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:41,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.16 | bwd_microstep: 412.47 | bwd_inner_microstep: 412.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:42,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.94 | bwd_microstep: 448.97 | bwd_inner_microstep: 448.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:43,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.69 | bwd_microstep: 456.40 | bwd_inner_microstep: 456.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:44,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.24 | bwd_microstep: 455.25 | bwd_inner_microstep: 455.08 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.06
[2024-12-05 14:46:45,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.39 | bwd_microstep: 435.61 | bwd_inner_microstep: 435.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:46,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.01 | bwd_microstep: 531.80 | bwd_inner_microstep: 531.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:47,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.47 | bwd_microstep: 440.24 | bwd_inner_microstep: 440.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:48,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.72 | bwd_microstep: 555.80 | bwd_inner_microstep: 555.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:46:49,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.43 | bwd_microstep: 411.67 | bwd_inner_microstep: 411.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:46:53,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.46 | optimizer_gradients: 14.04 | optimizer_step: 23.10
[2024-12-05 14:46:53,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.72 | bwd_microstep: 3135.81 | bwd_inner_microstep: 3123.53 | bwd_allreduce_microstep: 12.19 | step_microstep: 112.09
[2024-12-05 14:46:53,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4232.77 | bwd: 9843.75 | bwd_inner: 9829.84 | bwd_allreduce: 12.85 | step: 113.17

 57%|█████▋    | 219/384 [1:03:49<50:15, 18.28s/it]
                                                   
{'loss': 0.1227, 'grad_norm': 1.5571967363357544, 'learning_rate': 4.117901168710959e-07, 'epoch': 1.14}

 57%|█████▋    | 219/384 [1:03:49<50:15, 18.28s/it][2024-12-05 14:46:54,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.58 | bwd_microstep: 373.39 | bwd_inner_microstep: 373.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:54,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.30 | bwd_microstep: 357.77 | bwd_inner_microstep: 357.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:56,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.59 | bwd_microstep: 613.35 | bwd_inner_microstep: 613.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:46:57,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 346.74 | bwd_microstep: 703.41 | bwd_inner_microstep: 703.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:46:58,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.37 | bwd_microstep: 474.62 | bwd_inner_microstep: 474.49 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:46:59,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.27 | bwd_microstep: 434.76 | bwd_inner_microstep: 434.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:47:00,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 344.20 | bwd_microstep: 706.89 | bwd_inner_microstep: 706.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:47:01,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.68 | bwd_microstep: 456.63 | bwd_inner_microstep: 456.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:47:02,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.00 | bwd_microstep: 399.60 | bwd_inner_microstep: 399.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:03,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.65 | bwd_microstep: 426.94 | bwd_inner_microstep: 426.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:47:04,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.32 | bwd_microstep: 436.53 | bwd_inner_microstep: 436.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:05,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.88 | bwd_microstep: 489.44 | bwd_inner_microstep: 489.28 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:47:06,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.10 | bwd_microstep: 412.66 | bwd_inner_microstep: 412.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:47:07,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.83 | bwd_microstep: 432.73 | bwd_inner_microstep: 432.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:47:08,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.27 | bwd_microstep: 435.11 | bwd_inner_microstep: 434.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:47:11,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.22 | optimizer_gradients: 14.03 | optimizer_step: 23.34
[2024-12-05 14:47:11,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.60 | bwd_microstep: 3041.19 | bwd_inner_microstep: 3029.05 | bwd_allreduce_microstep: 12.05 | step_microstep: 112.14
[2024-12-05 14:47:11,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4404.47 | bwd: 10194.99 | bwd_inner: 10181.14 | bwd_allreduce: 12.69 | step: 113.23

 57%|█████▋    | 220/384 [1:04:08<50:19, 18.41s/it]
                                                   
{'loss': 0.1422, 'grad_norm': 1.6683669090270996, 'learning_rate': 4.0763697199140546e-07, 'epoch': 1.14}

 57%|█████▋    | 220/384 [1:04:08<50:19, 18.41s/it][2024-12-05 14:47:13,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.36 | bwd_microstep: 419.75 | bwd_inner_microstep: 419.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:47:14,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.84 | bwd_microstep: 434.81 | bwd_inner_microstep: 434.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:15,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.83 | bwd_microstep: 487.89 | bwd_inner_microstep: 487.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:16,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.71 | bwd_microstep: 524.98 | bwd_inner_microstep: 524.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:17,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.10 | bwd_microstep: 427.81 | bwd_inner_microstep: 427.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:18,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.90 | bwd_microstep: 418.00 | bwd_inner_microstep: 417.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:47:19,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.37 | bwd_microstep: 439.54 | bwd_inner_microstep: 439.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:20,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.47 | bwd_microstep: 433.25 | bwd_inner_microstep: 433.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:47:21,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.02 | bwd_microstep: 378.99 | bwd_inner_microstep: 378.86 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:47:22,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.99 | bwd_microstep: 402.50 | bwd_inner_microstep: 402.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:47:23,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.18 | bwd_microstep: 549.21 | bwd_inner_microstep: 549.06 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08
[2024-12-05 14:47:24,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.74 | bwd_microstep: 560.70 | bwd_inner_microstep: 560.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:47:25,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.77 | bwd_microstep: 346.80 | bwd_inner_microstep: 346.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:47:26,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.42 | bwd_microstep: 666.47 | bwd_inner_microstep: 666.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:47:27,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.23 | bwd_microstep: 371.84 | bwd_inner_microstep: 371.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:47:30,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.18 | optimizer_gradients: 14.21 | optimizer_step: 23.65
[2024-12-05 14:47:30,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.34 | bwd_microstep: 2751.45 | bwd_inner_microstep: 2738.90 | bwd_allreduce_microstep: 12.42 | step_microstep: 114.75
[2024-12-05 14:47:30,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4345.33 | bwd: 9613.98 | bwd_inner: 9599.78 | bwd_allreduce: 13.06 | step: 115.86

 58%|█████▊    | 221/384 [1:04:26<50:15, 18.50s/it]
                                                   
{'loss': 0.1256, 'grad_norm': 1.6401057243347168, 'learning_rate': 4.034904144421134e-07, 'epoch': 1.15}

 58%|█████▊    | 221/384 [1:04:26<50:15, 18.50s/it][2024-12-05 14:47:31,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.38 | bwd_microstep: 448.27 | bwd_inner_microstep: 448.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:47:32,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.91 | bwd_microstep: 447.54 | bwd_inner_microstep: 447.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.05
[2024-12-05 14:47:33,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.46 | bwd_microstep: 482.28 | bwd_inner_microstep: 482.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:34,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.00 | bwd_microstep: 577.12 | bwd_inner_microstep: 577.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:35,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.36 | bwd_microstep: 401.75 | bwd_inner_microstep: 401.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.05
[2024-12-05 14:47:36,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 348.08 | bwd_microstep: 708.29 | bwd_inner_microstep: 708.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:37,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.41 | bwd_microstep: 443.86 | bwd_inner_microstep: 443.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:39,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.48 | bwd_microstep: 670.73 | bwd_inner_microstep: 670.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:40,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.39 | bwd_microstep: 668.57 | bwd_inner_microstep: 668.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:41,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.06 | bwd_microstep: 403.02 | bwd_inner_microstep: 402.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:42,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.18 | bwd_microstep: 403.99 | bwd_inner_microstep: 403.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:43,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.64 | bwd_microstep: 461.79 | bwd_inner_microstep: 461.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:44,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.11 | bwd_microstep: 644.06 | bwd_inner_microstep: 643.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:47:45,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.48 | bwd_microstep: 426.44 | bwd_inner_microstep: 426.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:47:46,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.51 | bwd_microstep: 427.64 | bwd_inner_microstep: 427.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:47:49,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.46 | optimizer_gradients: 14.28 | optimizer_step: 23.53
[2024-12-05 14:47:49,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.09 | bwd_microstep: 2075.56 | bwd_inner_microstep: 2063.19 | bwd_allreduce_microstep: 12.27 | step_microstep: 116.69
[2024-12-05 14:47:49,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4670.59 | bwd: 9690.90 | bwd_inner: 9676.96 | bwd_allreduce: 12.87 | step: 117.68

 58%|█████▊    | 222/384 [1:04:45<50:08, 18.57s/it]
                                                   
{'loss': 0.1302, 'grad_norm': 1.5419056415557861, 'learning_rate': 3.9935073995566987e-07, 'epoch': 1.15}

 58%|█████▊    | 222/384 [1:04:45<50:08, 18.57s/it][2024-12-05 14:47:50,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.43 | bwd_microstep: 371.62 | bwd_inner_microstep: 371.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:51,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.19 | bwd_microstep: 477.38 | bwd_inner_microstep: 477.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:52,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.42 | bwd_microstep: 489.99 | bwd_inner_microstep: 489.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:53,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.26 | bwd_microstep: 522.26 | bwd_inner_microstep: 522.13 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:47:54,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.42 | bwd_microstep: 399.33 | bwd_inner_microstep: 399.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:55,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 355.72 | bwd_microstep: 737.90 | bwd_inner_microstep: 737.77 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:47:56,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.46 | bwd_microstep: 442.46 | bwd_inner_microstep: 442.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:57,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.87 | bwd_microstep: 423.02 | bwd_inner_microstep: 422.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:47:58,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.37 | bwd_microstep: 401.48 | bwd_inner_microstep: 401.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:47:59,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.18 | bwd_microstep: 447.64 | bwd_inner_microstep: 447.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:48:00,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.93 | bwd_microstep: 580.51 | bwd_inner_microstep: 580.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:01,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 244.65 | bwd_microstep: 378.50 | bwd_inner_microstep: 378.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:02,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.69 | bwd_microstep: 401.31 | bwd_inner_microstep: 401.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:48:03,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.77 | bwd_microstep: 427.22 | bwd_inner_microstep: 427.09 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:48:04,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.37 | bwd_microstep: 490.63 | bwd_inner_microstep: 490.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:48:07,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.32 | optimizer_gradients: 14.10 | optimizer_step: 23.17
[2024-12-05 14:48:07,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.95 | bwd_microstep: 1654.92 | bwd_inner_microstep: 1642.72 | bwd_allreduce_microstep: 12.09 | step_microstep: 112.46
[2024-12-05 14:48:07,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4369.76 | bwd: 8646.16 | bwd_inner: 8632.33 | bwd_allreduce: 12.74 | step: 113.49

 58%|█████▊    | 223/384 [1:05:03<49:09, 18.32s/it]
                                                   
{'loss': 0.1471, 'grad_norm': 1.650795817375183, 'learning_rate': 3.952182437736256e-07, 'epoch': 1.16}

 58%|█████▊    | 223/384 [1:05:03<49:09, 18.32s/it][2024-12-05 14:48:08,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.33 | bwd_microstep: 558.16 | bwd_inner_microstep: 558.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:48:09,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.97 | bwd_microstep: 524.87 | bwd_inner_microstep: 524.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:48:10,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.68 | bwd_microstep: 588.86 | bwd_inner_microstep: 588.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:48:11,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.64 | bwd_microstep: 425.27 | bwd_inner_microstep: 425.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:48:12,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.10 | bwd_microstep: 416.62 | bwd_inner_microstep: 416.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:48:13,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.71 | bwd_microstep: 446.90 | bwd_inner_microstep: 446.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:48:14,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.39 | bwd_microstep: 421.53 | bwd_inner_microstep: 421.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:48:15,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.03 | bwd_microstep: 586.14 | bwd_inner_microstep: 586.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:16,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.58 | bwd_microstep: 451.50 | bwd_inner_microstep: 451.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:48:17,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.45 | bwd_microstep: 438.81 | bwd_inner_microstep: 438.68 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:48:18,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.16 | bwd_microstep: 523.88 | bwd_inner_microstep: 523.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:19,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.10 | bwd_microstep: 426.67 | bwd_inner_microstep: 426.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:20,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.43 | bwd_microstep: 675.91 | bwd_inner_microstep: 675.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:21,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.81 | bwd_microstep: 510.06 | bwd_inner_microstep: 509.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:22,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.28 | bwd_microstep: 584.99 | bwd_inner_microstep: 584.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:24,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.14 | optimizer_gradients: 14.07 | optimizer_step: 23.48
[2024-12-05 14:48:24,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.50 | bwd_microstep: 899.26 | bwd_inner_microstep: 887.15 | bwd_allreduce_microstep: 11.99 | step_microstep: 112.20
[2024-12-05 14:48:24,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4537.20 | bwd: 8479.43 | bwd_inner: 8465.71 | bwd_allreduce: 12.60 | step: 113.27

 58%|█████▊    | 224/384 [1:05:20<48:07, 18.04s/it]
                                                   
{'loss': 0.1298, 'grad_norm': 1.672743797302246, 'learning_rate': 3.910932206255742e-07, 'epoch': 1.16}

 58%|█████▊    | 224/384 [1:05:20<48:07, 18.04s/it][2024-12-05 14:48:25,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.25 | bwd_microstep: 594.10 | bwd_inner_microstep: 593.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:48:26,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.36 | bwd_microstep: 560.76 | bwd_inner_microstep: 560.62 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:48:27,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.63 | bwd_microstep: 359.39 | bwd_inner_microstep: 359.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:48:28,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.99 | bwd_microstep: 498.35 | bwd_inner_microstep: 498.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:48:29,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.09 | bwd_microstep: 546.82 | bwd_inner_microstep: 546.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:30,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.10 | bwd_microstep: 567.21 | bwd_inner_microstep: 567.06 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:48:31,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.46 | bwd_microstep: 413.71 | bwd_inner_microstep: 413.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:32,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.96 | bwd_microstep: 448.43 | bwd_inner_microstep: 448.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:33,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.05 | bwd_microstep: 403.08 | bwd_inner_microstep: 402.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:48:34,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.40 | bwd_microstep: 406.23 | bwd_inner_microstep: 406.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:48:35,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.21 | bwd_microstep: 436.92 | bwd_inner_microstep: 436.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:48:36,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.23 | bwd_microstep: 433.28 | bwd_inner_microstep: 433.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:48:37,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.70 | bwd_microstep: 443.07 | bwd_inner_microstep: 442.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:48:38,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 406.66 | bwd_microstep: 927.25 | bwd_inner_microstep: 927.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:48:39,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.43 | bwd_microstep: 429.64 | bwd_inner_microstep: 429.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:48:41,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.16 | optimizer_gradients: 14.07 | optimizer_step: 23.32
[2024-12-05 14:48:41,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.19 | bwd_microstep: 1323.69 | bwd_inner_microstep: 1311.54 | bwd_allreduce_microstep: 12.05 | step_microstep: 112.13
[2024-12-05 14:48:41,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4488.77 | bwd: 8791.91 | bwd_inner: 8778.10 | bwd_allreduce: 12.69 | step: 113.28

 59%|█████▊    | 225/384 [1:05:37<46:57, 17.72s/it]
                                                   
{'loss': 0.1414, 'grad_norm': 1.6902425289154053, 'learning_rate': 3.869759647081325e-07, 'epoch': 1.17}

 59%|█████▊    | 225/384 [1:05:37<46:57, 17.72s/it][2024-12-05 14:48:42,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.44 | bwd_microstep: 579.12 | bwd_inner_microstep: 578.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:43,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.88 | bwd_microstep: 448.52 | bwd_inner_microstep: 448.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:48:44,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.74 | bwd_microstep: 481.18 | bwd_inner_microstep: 481.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:45,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.00 | bwd_microstep: 533.46 | bwd_inner_microstep: 533.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:46,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.31 | bwd_microstep: 561.17 | bwd_inner_microstep: 561.04 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:48:47,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 327.62 | bwd_microstep: 617.51 | bwd_inner_microstep: 617.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:48,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.77 | bwd_microstep: 492.83 | bwd_inner_microstep: 492.69 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:48:49,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.68 | bwd_microstep: 409.33 | bwd_inner_microstep: 409.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:50,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.83 | bwd_microstep: 528.25 | bwd_inner_microstep: 528.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:51,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.87 | bwd_microstep: 443.79 | bwd_inner_microstep: 443.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:53,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.99 | bwd_microstep: 689.73 | bwd_inner_microstep: 689.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:54,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.84 | bwd_microstep: 614.86 | bwd_inner_microstep: 614.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:48:55,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.28 | bwd_microstep: 448.42 | bwd_inner_microstep: 448.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:48:56,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 344.23 | bwd_microstep: 726.93 | bwd_inner_microstep: 726.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:57,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.12 | bwd_microstep: 459.71 | bwd_inner_microstep: 459.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:48:59,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.49 | optimizer_gradients: 14.20 | optimizer_step: 23.67
[2024-12-05 14:48:59,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.51 | bwd_microstep: 1615.60 | bwd_inner_microstep: 1603.06 | bwd_allreduce_microstep: 12.43 | step_microstep: 113.20
[2024-12-05 14:48:59,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4643.20 | bwd: 9650.41 | bwd_inner: 9636.22 | bwd_allreduce: 13.09 | step: 114.38

 59%|█████▉    | 226/384 [1:05:55<47:04, 17.88s/it]
                                                   
{'loss': 0.1032, 'grad_norm': 1.607485055923462, 'learning_rate': 3.828667696639589e-07, 'epoch': 1.17}

 59%|█████▉    | 226/384 [1:05:55<47:04, 17.88s/it][2024-12-05 14:49:00,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.07 | bwd_microstep: 397.85 | bwd_inner_microstep: 397.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:49:01,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.45 | bwd_microstep: 357.63 | bwd_inner_microstep: 357.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:02,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.80 | bwd_microstep: 434.15 | bwd_inner_microstep: 434.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:03,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.98 | bwd_microstep: 359.47 | bwd_inner_microstep: 359.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:04,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.08 | bwd_microstep: 428.33 | bwd_inner_microstep: 428.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:05,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.85 | bwd_microstep: 425.58 | bwd_inner_microstep: 425.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:06,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.68 | bwd_microstep: 451.25 | bwd_inner_microstep: 451.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:49:07,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.43 | bwd_microstep: 417.10 | bwd_inner_microstep: 416.96 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:49:08,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.77 | bwd_microstep: 683.17 | bwd_inner_microstep: 683.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:09,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.06 | bwd_microstep: 537.35 | bwd_inner_microstep: 537.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:49:10,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.86 | bwd_microstep: 408.85 | bwd_inner_microstep: 408.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:11,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.15 | bwd_microstep: 481.60 | bwd_inner_microstep: 481.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:12,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.97 | bwd_microstep: 350.62 | bwd_inner_microstep: 350.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:49:13,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.49 | bwd_microstep: 436.93 | bwd_inner_microstep: 436.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:14,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.57 | bwd_microstep: 419.61 | bwd_inner_microstep: 419.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:49:17,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.48 | optimizer_gradients: 14.01 | optimizer_step: 23.18
[2024-12-05 14:49:17,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.43 | bwd_microstep: 2553.99 | bwd_inner_microstep: 2541.87 | bwd_allreduce_microstep: 12.04 | step_microstep: 113.05
[2024-12-05 14:49:17,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4212.73 | bwd: 9143.45 | bwd_inner: 9129.70 | bwd_allreduce: 12.67 | step: 114.09

 59%|█████▉    | 227/384 [1:06:14<46:57, 17.94s/it]
                                                   
{'loss': 0.1116, 'grad_norm': 1.5336533784866333, 'learning_rate': 3.7876592856081e-07, 'epoch': 1.18}

 59%|█████▉    | 227/384 [1:06:14<46:57, 17.94s/it][2024-12-05 14:49:19,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.90 | bwd_microstep: 557.33 | bwd_inner_microstep: 557.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:49:20,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.83 | bwd_microstep: 427.17 | bwd_inner_microstep: 427.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:49:21,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.68 | bwd_microstep: 599.11 | bwd_inner_microstep: 598.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:49:22,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.66 | bwd_microstep: 540.40 | bwd_inner_microstep: 540.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:49:23,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.21 | bwd_microstep: 489.03 | bwd_inner_microstep: 488.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:49:24,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.68 | bwd_microstep: 483.48 | bwd_inner_microstep: 483.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:49:25,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.37 | bwd_microstep: 346.52 | bwd_inner_microstep: 346.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:26,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 330.78 | bwd_microstep: 635.16 | bwd_inner_microstep: 635.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:27,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.10 | bwd_microstep: 395.89 | bwd_inner_microstep: 395.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:49:28,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.75 | bwd_microstep: 433.46 | bwd_inner_microstep: 433.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:29,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.85 | bwd_microstep: 485.29 | bwd_inner_microstep: 485.13 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:49:30,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.84 | bwd_microstep: 434.84 | bwd_inner_microstep: 434.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:49:31,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.99 | bwd_microstep: 437.88 | bwd_inner_microstep: 437.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:32,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 382.96 | bwd_microstep: 833.77 | bwd_inner_microstep: 833.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:33,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.80 | bwd_microstep: 478.29 | bwd_inner_microstep: 478.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:36,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.42 | optimizer_gradients: 14.08 | optimizer_step: 23.53
[2024-12-05 14:49:36,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.24 | bwd_microstep: 2086.97 | bwd_inner_microstep: 2074.96 | bwd_allreduce_microstep: 11.92 | step_microstep: 112.68
[2024-12-05 14:49:36,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4564.72 | bwd: 9664.60 | bwd_inner: 9650.95 | bwd_allreduce: 12.57 | step: 113.80

 59%|█████▉    | 228/384 [1:06:32<46:54, 18.04s/it]
                                                   
{'loss': 0.1243, 'grad_norm': 1.5528396368026733, 'learning_rate': 3.7467373387063964e-07, 'epoch': 1.19}

 59%|█████▉    | 228/384 [1:06:32<46:54, 18.04s/it][2024-12-05 14:49:36,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.32 | bwd_microstep: 403.76 | bwd_inner_microstep: 403.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:37,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.63 | bwd_microstep: 489.49 | bwd_inner_microstep: 489.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:39,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.30 | bwd_microstep: 587.55 | bwd_inner_microstep: 587.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:40,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.34 | bwd_microstep: 623.47 | bwd_inner_microstep: 623.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:41,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.20 | bwd_microstep: 433.51 | bwd_inner_microstep: 433.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:49:42,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.66 | bwd_microstep: 482.04 | bwd_inner_microstep: 481.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:43,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.40 | bwd_microstep: 480.59 | bwd_inner_microstep: 480.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:44,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.70 | bwd_microstep: 707.79 | bwd_inner_microstep: 707.65 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:49:45,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.72 | bwd_microstep: 688.33 | bwd_inner_microstep: 688.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:49:46,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.85 | bwd_microstep: 404.20 | bwd_inner_microstep: 404.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:47,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.83 | bwd_microstep: 436.53 | bwd_inner_microstep: 436.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:48,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.75 | bwd_microstep: 397.14 | bwd_inner_microstep: 397.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:49,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.69 | bwd_microstep: 428.49 | bwd_inner_microstep: 428.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:49:50,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 359.06 | bwd_microstep: 740.71 | bwd_inner_microstep: 740.58 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:49:52,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.87 | bwd_microstep: 525.29 | bwd_inner_microstep: 525.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:49:53,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.18 | optimizer_gradients: 14.03 | optimizer_step: 23.23
[2024-12-05 14:49:53,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.22 | bwd_microstep: 800.91 | bwd_inner_microstep: 788.81 | bwd_allreduce_microstep: 12.01 | step_microstep: 111.89
[2024-12-05 14:49:53,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4613.61 | bwd: 8629.80 | bwd_inner: 8616.05 | bwd_allreduce: 12.62 | step: 112.87

 60%|█████▉    | 229/384 [1:06:49<46:06, 17.85s/it]
                                                   
{'loss': 0.1437, 'grad_norm': 1.743318796157837, 'learning_rate': 3.7059047744873955e-07, 'epoch': 1.19}

 60%|█████▉    | 229/384 [1:06:49<46:06, 17.85s/it][2024-12-05 14:49:54,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 343.47 | bwd_microstep: 691.40 | bwd_inner_microstep: 691.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:49:55,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.63 | bwd_microstep: 345.85 | bwd_inner_microstep: 345.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:49:56,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.71 | bwd_microstep: 580.28 | bwd_inner_microstep: 580.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:49:57,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.44 | bwd_microstep: 503.64 | bwd_inner_microstep: 503.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:49:58,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.99 | bwd_microstep: 407.57 | bwd_inner_microstep: 407.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:49:59,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.98 | bwd_microstep: 582.34 | bwd_inner_microstep: 582.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:50:00,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.51 | bwd_microstep: 445.87 | bwd_inner_microstep: 445.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:01,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.55 | bwd_microstep: 410.78 | bwd_inner_microstep: 410.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:50:02,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.38 | bwd_microstep: 396.91 | bwd_inner_microstep: 396.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:50:03,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.24 | bwd_microstep: 349.26 | bwd_inner_microstep: 349.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:50:04,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.68 | bwd_microstep: 677.04 | bwd_inner_microstep: 676.77 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.16
[2024-12-05 14:50:05,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.82 | bwd_microstep: 356.28 | bwd_inner_microstep: 356.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:50:06,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.06 | bwd_microstep: 679.84 | bwd_inner_microstep: 679.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:50:07,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.91 | bwd_microstep: 594.14 | bwd_inner_microstep: 594.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:50:08,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.39 | bwd_microstep: 380.50 | bwd_inner_microstep: 380.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:50:10,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.29 | optimizer_gradients: 14.05 | optimizer_step: 23.52
[2024-12-05 14:50:10,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.45 | bwd_microstep: 1176.84 | bwd_inner_microstep: 1164.83 | bwd_allreduce_microstep: 11.92 | step_microstep: 112.84
[2024-12-05 14:50:10,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4477.26 | bwd: 8578.55 | bwd_inner: 8564.82 | bwd_allreduce: 12.58 | step: 114.05

 60%|█████▉    | 230/384 [1:07:07<45:24, 17.69s/it]
                                                   
{'loss': 0.131, 'grad_norm': 1.6484525203704834, 'learning_rate': 3.665164505129241e-07, 'epoch': 1.2}

 60%|█████▉    | 230/384 [1:07:07<45:24, 17.69s/it][2024-12-05 14:50:11,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.30 | bwd_microstep: 375.38 | bwd_inner_microstep: 375.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:12,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.57 | bwd_microstep: 408.45 | bwd_inner_microstep: 408.31 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:50:13,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.19 | bwd_microstep: 438.91 | bwd_inner_microstep: 438.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:14,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.00 | bwd_microstep: 606.37 | bwd_inner_microstep: 606.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:50:15,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.22 | bwd_microstep: 556.11 | bwd_inner_microstep: 555.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:50:16,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.27 | bwd_microstep: 718.40 | bwd_inner_microstep: 718.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:18,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 359.26 | bwd_microstep: 737.91 | bwd_inner_microstep: 737.78 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:50:19,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.72 | bwd_microstep: 447.58 | bwd_inner_microstep: 447.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:20,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 341.55 | bwd_microstep: 682.41 | bwd_inner_microstep: 682.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:21,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.32 | bwd_microstep: 430.51 | bwd_inner_microstep: 430.36 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:50:22,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.04 | bwd_microstep: 593.13 | bwd_inner_microstep: 593.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:50:23,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.51 | bwd_microstep: 675.66 | bwd_inner_microstep: 675.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:24,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.84 | bwd_microstep: 433.79 | bwd_inner_microstep: 433.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:50:25,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.87 | bwd_microstep: 487.20 | bwd_inner_microstep: 487.06 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:50:27,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.94 | bwd_microstep: 688.82 | bwd_inner_microstep: 688.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:28,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.81 | optimizer_gradients: 14.02 | optimizer_step: 23.29
[2024-12-05 14:50:28,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.50 | bwd_microstep: 808.03 | bwd_inner_microstep: 796.04 | bwd_allreduce_microstep: 11.89 | step_microstep: 112.79
[2024-12-05 14:50:28,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4827.19 | bwd: 9088.71 | bwd_inner: 9075.00 | bwd_allreduce: 12.53 | step: 113.81

 60%|██████    | 231/384 [1:07:24<45:15, 17.75s/it]
                                                   
{'loss': 0.1268, 'grad_norm': 1.6531956195831299, 'learning_rate': 3.6245194362276094e-07, 'epoch': 1.2}

 60%|██████    | 231/384 [1:07:24<45:15, 17.75s/it][2024-12-05 14:50:29,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.24 | bwd_microstep: 413.55 | bwd_inner_microstep: 413.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:30,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.24 | bwd_microstep: 452.60 | bwd_inner_microstep: 452.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:31,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.49 | bwd_microstep: 430.36 | bwd_inner_microstep: 430.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:32,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.24 | bwd_microstep: 430.67 | bwd_inner_microstep: 430.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:33,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 244.03 | bwd_microstep: 369.51 | bwd_inner_microstep: 369.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:34,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.57 | bwd_microstep: 529.64 | bwd_inner_microstep: 529.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:35,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 401.44 | bwd_microstep: 894.82 | bwd_inner_microstep: 894.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:36,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.10 | bwd_microstep: 397.87 | bwd_inner_microstep: 397.73 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:50:37,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.68 | bwd_microstep: 459.61 | bwd_inner_microstep: 459.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:38,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.18 | bwd_microstep: 413.00 | bwd_inner_microstep: 412.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:50:39,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.22 | bwd_microstep: 436.18 | bwd_inner_microstep: 436.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:40,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.13 | bwd_microstep: 439.45 | bwd_inner_microstep: 439.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:41,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.70 | bwd_microstep: 563.54 | bwd_inner_microstep: 563.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:42,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.61 | bwd_microstep: 454.36 | bwd_inner_microstep: 454.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:43,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.40 | bwd_microstep: 393.34 | bwd_inner_microstep: 393.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:46,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.62 | optimizer_gradients: 14.07 | optimizer_step: 23.16
[2024-12-05 14:50:46,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.13 | bwd_microstep: 2456.77 | bwd_inner_microstep: 2444.78 | bwd_allreduce_microstep: 11.91 | step_microstep: 112.32
[2024-12-05 14:50:46,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4422.49 | bwd: 9535.23 | bwd_inner: 9521.61 | bwd_allreduce: 12.52 | step: 113.29

 60%|██████    | 232/384 [1:07:42<44:58, 17.75s/it]
                                                   
{'loss': 0.1261, 'grad_norm': 1.491297960281372, 'learning_rate': 3.5839724665884795e-07, 'epoch': 1.21}

 60%|██████    | 232/384 [1:07:42<44:58, 17.75s/it][2024-12-05 14:50:47,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.30 | bwd_microstep: 397.09 | bwd_inner_microstep: 396.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:48,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.54 | bwd_microstep: 435.17 | bwd_inner_microstep: 435.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:49,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.62 | bwd_microstep: 589.03 | bwd_inner_microstep: 588.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:50:50,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.65 | bwd_microstep: 528.56 | bwd_inner_microstep: 528.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:51,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.13 | bwd_microstep: 447.98 | bwd_inner_microstep: 447.82 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 14:50:52,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.15 | bwd_microstep: 432.80 | bwd_inner_microstep: 432.63 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:50:53,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.79 | bwd_microstep: 403.34 | bwd_inner_microstep: 403.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:54,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.95 | bwd_microstep: 459.51 | bwd_inner_microstep: 459.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:55,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.03 | bwd_microstep: 412.87 | bwd_inner_microstep: 412.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:56,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 222.77 | bwd_microstep: 321.28 | bwd_inner_microstep: 321.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:50:57,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.18 | bwd_microstep: 480.91 | bwd_inner_microstep: 480.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:50:58,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.14 | bwd_microstep: 458.53 | bwd_inner_microstep: 458.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:50:59,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.12 | bwd_microstep: 592.77 | bwd_inner_microstep: 592.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:51:00,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 386.01 | bwd_microstep: 832.09 | bwd_inner_microstep: 831.94 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:51:01,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.07 | bwd_microstep: 681.32 | bwd_inner_microstep: 681.18 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 14:51:04,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.91 | optimizer_gradients: 14.04 | optimizer_step: 23.15
[2024-12-05 14:51:04,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.74 | bwd_microstep: 1148.60 | bwd_inner_microstep: 1136.45 | bwd_allreduce_microstep: 12.05 | step_microstep: 111.86
[2024-12-05 14:51:04,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4523.28 | bwd: 8621.76 | bwd_inner: 8607.81 | bwd_allreduce: 12.72 | step: 112.98

 61%|██████    | 233/384 [1:08:00<44:33, 17.70s/it]
                                                   
{'loss': 0.1233, 'grad_norm': 1.5535728931427002, 'learning_rate': 3.5435264880213937e-07, 'epoch': 1.21}

 61%|██████    | 233/384 [1:08:00<44:33, 17.70s/it][2024-12-05 14:51:05,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.26 | bwd_microstep: 383.60 | bwd_inner_microstep: 383.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:06,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.61 | bwd_microstep: 483.82 | bwd_inner_microstep: 483.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:07,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.32 | bwd_microstep: 431.37 | bwd_inner_microstep: 431.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:08,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.95 | bwd_microstep: 489.81 | bwd_inner_microstep: 489.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:09,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.90 | bwd_microstep: 597.02 | bwd_inner_microstep: 596.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:10,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.26 | bwd_microstep: 445.58 | bwd_inner_microstep: 445.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:11,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.83 | bwd_microstep: 429.12 | bwd_inner_microstep: 428.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:12,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.74 | bwd_microstep: 590.41 | bwd_inner_microstep: 590.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:51:13,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.98 | bwd_microstep: 440.58 | bwd_inner_microstep: 440.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:14,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.82 | bwd_microstep: 448.38 | bwd_inner_microstep: 448.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:15,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 242.05 | bwd_microstep: 372.26 | bwd_inner_microstep: 372.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:15,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.42 | bwd_microstep: 449.39 | bwd_inner_microstep: 449.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:51:16,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.98 | bwd_microstep: 438.69 | bwd_inner_microstep: 438.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:17,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.24 | bwd_microstep: 419.04 | bwd_inner_microstep: 418.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:51:19,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.65 | bwd_microstep: 457.50 | bwd_inner_microstep: 457.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:51:21,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.35 | optimizer_gradients: 14.11 | optimizer_step: 23.33
[2024-12-05 14:51:21,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.16 | bwd_microstep: 2241.61 | bwd_inner_microstep: 2229.45 | bwd_allreduce_microstep: 12.06 | step_microstep: 112.55
[2024-12-05 14:51:21,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4297.22 | bwd: 9118.16 | bwd_inner: 9104.33 | bwd_allreduce: 12.68 | step: 113.70

 61%|██████    | 234/384 [1:08:18<44:21, 17.74s/it]
                                                   
{'loss': 0.1331, 'grad_norm': 1.6621695756912231, 'learning_rate': 3.50318438513321e-07, 'epoch': 1.22}

 61%|██████    | 234/384 [1:08:18<44:21, 17.74s/it][2024-12-05 14:51:22,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.94 | bwd_microstep: 514.32 | bwd_inner_microstep: 514.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:51:24,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 376.26 | bwd_microstep: 744.06 | bwd_inner_microstep: 743.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:51:25,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.23 | bwd_microstep: 422.40 | bwd_inner_microstep: 422.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:51:26,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.55 | bwd_microstep: 594.44 | bwd_inner_microstep: 594.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:51:27,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 391.11 | bwd_microstep: 864.74 | bwd_inner_microstep: 864.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:51:28,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 229.09 | bwd_microstep: 328.89 | bwd_inner_microstep: 328.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:51:29,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.17 | bwd_microstep: 428.21 | bwd_inner_microstep: 428.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:51:30,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.22 | bwd_microstep: 352.72 | bwd_inner_microstep: 352.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:51:31,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.70 | bwd_microstep: 568.92 | bwd_inner_microstep: 568.68 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:51:32,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.75 | bwd_microstep: 613.83 | bwd_inner_microstep: 613.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:51:33,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.24 | bwd_microstep: 424.29 | bwd_inner_microstep: 424.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:51:34,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.42 | bwd_microstep: 425.31 | bwd_inner_microstep: 425.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:35,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.13 | bwd_microstep: 438.00 | bwd_inner_microstep: 437.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:36,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.83 | bwd_microstep: 386.43 | bwd_inner_microstep: 386.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:37,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.37 | bwd_microstep: 481.94 | bwd_inner_microstep: 481.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:38,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.63 | optimizer_gradients: 14.04 | optimizer_step: 24.17
[2024-12-05 14:51:38,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.59 | bwd_microstep: 1269.27 | bwd_inner_microstep: 1257.05 | bwd_allreduce_microstep: 12.12 | step_microstep: 112.39
[2024-12-05 14:51:38,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4601.70 | bwd: 8857.71 | bwd_inner: 8843.75 | bwd_allreduce: 12.74 | step: 113.47

 61%|██████    | 235/384 [1:08:35<43:33, 17.54s/it]
                                                   
{'loss': 0.1229, 'grad_norm': 1.5829371213912964, 'learning_rate': 3.462949035122376e-07, 'epoch': 1.22}

 61%|██████    | 235/384 [1:08:35<43:33, 17.54s/it][2024-12-05 14:51:39,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.03 | bwd_microstep: 413.09 | bwd_inner_microstep: 412.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:40,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.36 | bwd_microstep: 588.63 | bwd_inner_microstep: 588.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:41,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.45 | bwd_microstep: 487.78 | bwd_inner_microstep: 487.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:42,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.39 | bwd_microstep: 607.91 | bwd_inner_microstep: 607.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:43,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.82 | bwd_microstep: 404.28 | bwd_inner_microstep: 404.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:44,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.47 | bwd_microstep: 440.52 | bwd_inner_microstep: 440.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:51:45,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.21 | bwd_microstep: 562.09 | bwd_inner_microstep: 561.95 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:51:46,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.79 | bwd_microstep: 450.06 | bwd_inner_microstep: 449.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:51:47,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.97 | bwd_microstep: 447.67 | bwd_inner_microstep: 447.46 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12
[2024-12-05 14:51:48,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.35 | bwd_microstep: 484.22 | bwd_inner_microstep: 484.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:49,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.62 | bwd_microstep: 418.40 | bwd_inner_microstep: 418.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:50,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.23 | bwd_microstep: 438.76 | bwd_inner_microstep: 438.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:51,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.50 | bwd_microstep: 417.32 | bwd_inner_microstep: 417.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:52,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.78 | bwd_microstep: 443.38 | bwd_inner_microstep: 443.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:53,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.44 | bwd_microstep: 718.49 | bwd_inner_microstep: 718.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:55,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.25 | optimizer_gradients: 14.09 | optimizer_step: 23.24
[2024-12-05 14:51:55,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.39 | bwd_microstep: 1874.19 | bwd_inner_microstep: 1862.04 | bwd_allreduce_microstep: 12.06 | step_microstep: 112.08
[2024-12-05 14:51:55,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4437.84 | bwd: 9196.79 | bwd_inner: 9182.89 | bwd_allreduce: 12.76 | step: 113.30

 61%|██████▏   | 236/384 [1:08:51<42:39, 17.29s/it]
                                                   
{'loss': 0.1434, 'grad_norm': 1.7693301439285278, 'learning_rate': 3.4228233075737223e-07, 'epoch': 1.23}

 61%|██████▏   | 236/384 [1:08:51<42:39, 17.29s/it][2024-12-05 14:51:56,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.88 | bwd_microstep: 409.37 | bwd_inner_microstep: 409.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:51:57,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.68 | bwd_microstep: 476.51 | bwd_inner_microstep: 476.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:51:58,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.89 | bwd_microstep: 582.93 | bwd_inner_microstep: 582.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:51:59,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.02 | bwd_microstep: 487.14 | bwd_inner_microstep: 487.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:52:00,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 349.63 | bwd_microstep: 710.90 | bwd_inner_microstep: 710.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:52:01,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.73 | bwd_microstep: 490.92 | bwd_inner_microstep: 490.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:52:02,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.29 | bwd_microstep: 457.38 | bwd_inner_microstep: 457.23 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 14:52:03,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.56 | bwd_microstep: 593.77 | bwd_inner_microstep: 593.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:52:05,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.33 | bwd_microstep: 571.08 | bwd_inner_microstep: 570.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:52:06,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.85 | bwd_microstep: 533.72 | bwd_inner_microstep: 533.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:52:06,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.60 | bwd_microstep: 512.25 | bwd_inner_microstep: 512.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:52:07,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.92 | bwd_microstep: 485.65 | bwd_inner_microstep: 485.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:52:08,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.46 | bwd_microstep: 413.38 | bwd_inner_microstep: 413.26 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:52:10,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 366.87 | bwd_microstep: 784.14 | bwd_inner_microstep: 784.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:52:11,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.81 | bwd_microstep: 425.69 | bwd_inner_microstep: 425.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:52:12,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.71 | optimizer_gradients: 14.02 | optimizer_step: 23.03
[2024-12-05 14:52:12,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.04 | bwd_microstep: 804.48 | bwd_inner_microstep: 792.37 | bwd_allreduce_microstep: 12.03 | step_microstep: 112.14
[2024-12-05 14:52:12,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4671.62 | bwd: 8739.33 | bwd_inner: 8725.55 | bwd_allreduce: 12.68 | step: 113.31

 62%|██████▏   | 237/384 [1:09:08<42:04, 17.18s/it]
                                                   
{'loss': 0.1213, 'grad_norm': 1.6113789081573486, 'learning_rate': 3.3828100642538093e-07, 'epoch': 1.23}

 62%|██████▏   | 237/384 [1:09:08<42:04, 17.18s/it][2024-12-05 14:52:13,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.19 | bwd_microstep: 594.23 | bwd_inner_microstep: 594.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:52:14,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.37 | bwd_microstep: 682.92 | bwd_inner_microstep: 682.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:52:15,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.66 | bwd_microstep: 632.64 | bwd_inner_microstep: 632.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:52:17,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 343.84 | bwd_microstep: 691.80 | bwd_inner_microstep: 691.66 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:52:18,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.10 | bwd_microstep: 622.41 | bwd_inner_microstep: 622.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:52:19,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.61 | bwd_microstep: 567.23 | bwd_inner_microstep: 567.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:52:20,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.51 | bwd_microstep: 430.41 | bwd_inner_microstep: 430.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:52:21,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.88 | bwd_microstep: 401.93 | bwd_inner_microstep: 401.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:52:22,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.61 | bwd_microstep: 484.23 | bwd_inner_microstep: 484.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:52:22,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.47 | bwd_microstep: 424.16 | bwd_inner_microstep: 424.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:52:23,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.19 | bwd_microstep: 451.84 | bwd_inner_microstep: 451.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:52:25,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.30 | bwd_microstep: 592.81 | bwd_inner_microstep: 592.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:52:26,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.86 | bwd_microstep: 563.08 | bwd_inner_microstep: 562.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:52:27,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 343.33 | bwd_microstep: 661.00 | bwd_inner_microstep: 660.88 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:52:28,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.82 | bwd_microstep: 428.87 | bwd_inner_microstep: 428.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:52:29,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.16 | optimizer_gradients: 14.02 | optimizer_step: 23.06
[2024-12-05 14:52:29,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.61 | bwd_microstep: 696.28 | bwd_inner_microstep: 684.40 | bwd_allreduce_microstep: 11.80 | step_microstep: 114.33
[2024-12-05 14:52:29,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4707.40 | bwd: 8925.84 | bwd_inner: 8912.31 | bwd_allreduce: 12.45 | step: 115.44

 62%|██████▏   | 238/384 [1:09:25<41:46, 17.17s/it]
                                                   
{'loss': 0.1191, 'grad_norm': 1.5799354314804077, 'learning_rate': 3.342912158906821e-07, 'epoch': 1.24}

 62%|██████▏   | 238/384 [1:09:25<41:46, 17.17s/it][2024-12-05 14:52:30,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.64 | bwd_microstep: 680.20 | bwd_inner_microstep: 679.97 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.16
[2024-12-05 14:52:32,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.98 | bwd_microstep: 589.56 | bwd_inner_microstep: 589.40 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:52:32,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.25 | bwd_microstep: 450.71 | bwd_inner_microstep: 450.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:52:33,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.87 | bwd_microstep: 463.29 | bwd_inner_microstep: 463.15 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:52:34,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.91 | bwd_microstep: 460.12 | bwd_inner_microstep: 459.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:52:35,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.26 | bwd_microstep: 409.24 | bwd_inner_microstep: 409.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:52:36,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.46 | bwd_microstep: 427.83 | bwd_inner_microstep: 427.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:52:37,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.51 | bwd_microstep: 354.54 | bwd_inner_microstep: 354.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:52:38,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.00 | bwd_microstep: 450.77 | bwd_inner_microstep: 450.62 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:52:39,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.34 | bwd_microstep: 410.89 | bwd_inner_microstep: 410.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:52:39,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.00 | bwd_microstep: 437.55 | bwd_inner_microstep: 437.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:52:41,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.98 | bwd_microstep: 591.69 | bwd_inner_microstep: 591.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:52:42,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 358.91 | bwd_microstep: 738.10 | bwd_inner_microstep: 737.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:52:43,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.45 | bwd_microstep: 448.30 | bwd_inner_microstep: 448.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:52:43,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.79 | bwd_microstep: 428.42 | bwd_inner_microstep: 428.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:52:46,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.56 | optimizer_gradients: 14.06 | optimizer_step: 23.23
[2024-12-05 14:52:46,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.85 | bwd_microstep: 1869.16 | bwd_inner_microstep: 1857.18 | bwd_allreduce_microstep: 11.88 | step_microstep: 114.15
[2024-12-05 14:52:46,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4488.29 | bwd: 9210.39 | bwd_inner: 9196.65 | bwd_allreduce: 12.58 | step: 115.28

 62%|██████▏   | 239/384 [1:09:42<41:08, 17.02s/it]
                                                   
{'loss': 0.1133, 'grad_norm': 1.5467253923416138, 'learning_rate': 3.3031324370510396e-07, 'epoch': 1.24}

 62%|██████▏   | 239/384 [1:09:42<41:08, 17.02s/it][2024-12-05 14:52:47,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.13 | bwd_microstep: 458.80 | bwd_inner_microstep: 458.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:52:48,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.24 | bwd_microstep: 562.78 | bwd_inner_microstep: 562.64 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 14:52:49,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.26 | bwd_microstep: 403.39 | bwd_inner_microstep: 403.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:52:50,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.04 | bwd_microstep: 627.85 | bwd_inner_microstep: 627.72 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:52:51,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.46 | bwd_microstep: 449.04 | bwd_inner_microstep: 448.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:52:52,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.22 | bwd_microstep: 673.46 | bwd_inner_microstep: 673.31 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:52:53,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.38 | bwd_microstep: 560.98 | bwd_inner_microstep: 560.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:52:54,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.08 | bwd_microstep: 455.10 | bwd_inner_microstep: 454.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:52:55,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.57 | bwd_microstep: 402.04 | bwd_inner_microstep: 401.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:52:56,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.22 | bwd_microstep: 599.77 | bwd_inner_microstep: 599.62 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:52:57,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.24 | bwd_microstep: 446.81 | bwd_inner_microstep: 446.67 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:52:58,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.82 | bwd_microstep: 431.37 | bwd_inner_microstep: 431.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:52:59,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.13 | bwd_microstep: 479.08 | bwd_inner_microstep: 478.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:53:00,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.56 | bwd_microstep: 684.00 | bwd_inner_microstep: 683.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:53:01,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.20 | bwd_microstep: 552.14 | bwd_inner_microstep: 552.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:53:02,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.53 | optimizer_gradients: 14.02 | optimizer_step: 23.25
[2024-12-05 14:53:02,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.89 | bwd_microstep: 802.62 | bwd_inner_microstep: 790.49 | bwd_allreduce_microstep: 12.03 | step_microstep: 112.22
[2024-12-05 14:53:02,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4570.50 | bwd: 8589.24 | bwd_inner: 8575.38 | bwd_allreduce: 12.71 | step: 113.38

 62%|██████▎   | 240/384 [1:09:59<40:25, 16.84s/it]
                                                   
{'loss': 0.1391, 'grad_norm': 1.8147982358932495, 'learning_rate': 3.263473735775899e-07, 'epoch': 1.25}

 62%|██████▎   | 240/384 [1:09:59<40:25, 16.84s/it][2024-12-05 14:53:04,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 364.23 | bwd_microstep: 741.71 | bwd_inner_microstep: 741.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:53:05,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.75 | bwd_microstep: 538.57 | bwd_inner_microstep: 538.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:53:06,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.74 | bwd_microstep: 488.50 | bwd_inner_microstep: 488.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:53:07,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.88 | bwd_microstep: 628.81 | bwd_inner_microstep: 628.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:53:07,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.68 | bwd_microstep: 409.51 | bwd_inner_microstep: 409.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:53:09,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.88 | bwd_microstep: 450.02 | bwd_inner_microstep: 449.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:53:10,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.75 | bwd_microstep: 567.19 | bwd_inner_microstep: 567.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:53:10,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.19 | bwd_microstep: 435.48 | bwd_inner_microstep: 435.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:53:11,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.98 | bwd_microstep: 463.43 | bwd_inner_microstep: 463.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:53:12,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.09 | bwd_microstep: 420.01 | bwd_inner_microstep: 419.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:53:14,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.28 | bwd_microstep: 683.73 | bwd_inner_microstep: 683.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:53:14,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.16 | bwd_microstep: 438.74 | bwd_inner_microstep: 438.60 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:53:16,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.46 | bwd_microstep: 414.08 | bwd_inner_microstep: 413.91 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:53:17,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.01 | bwd_microstep: 438.96 | bwd_inner_microstep: 438.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:53:18,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.04 | bwd_microstep: 412.32 | bwd_inner_microstep: 412.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:53:19,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.54 | optimizer_gradients: 14.02 | optimizer_step: 23.31
[2024-12-05 14:53:19,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.05 | bwd_microstep: 838.05 | bwd_inner_microstep: 825.86 | bwd_allreduce_microstep: 12.09 | step_microstep: 112.34
[2024-12-05 14:53:19,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4607.28 | bwd: 8369.10 | bwd_inner: 8355.21 | bwd_allreduce: 12.75 | step: 113.51

 63%|██████▎   | 241/384 [1:10:15<40:08, 16.84s/it]
                                                   
{'loss': 0.1339, 'grad_norm': 1.702359676361084, 'learning_rate': 3.2239388835396484e-07, 'epoch': 1.25}

 63%|██████▎   | 241/384 [1:10:15<40:08, 16.84s/it][2024-12-05 14:53:20,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.81 | bwd_microstep: 378.70 | bwd_inner_microstep: 378.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:53:21,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.74 | bwd_microstep: 585.07 | bwd_inner_microstep: 584.92 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:53:22,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.70 | bwd_microstep: 356.96 | bwd_inner_microstep: 356.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:53:23,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.44 | bwd_microstep: 566.92 | bwd_inner_microstep: 566.79 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:53:24,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.75 | bwd_microstep: 491.38 | bwd_inner_microstep: 491.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:53:25,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.33 | bwd_microstep: 432.95 | bwd_inner_microstep: 432.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:53:27,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 401.29 | bwd_microstep: 905.28 | bwd_inner_microstep: 905.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:53:28,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.75 | bwd_microstep: 522.20 | bwd_inner_microstep: 522.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:53:28,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.59 | bwd_microstep: 351.82 | bwd_inner_microstep: 351.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:53:30,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.94 | bwd_microstep: 435.24 | bwd_inner_microstep: 435.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:53:31,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.82 | bwd_microstep: 430.55 | bwd_inner_microstep: 430.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:53:32,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.29 | bwd_microstep: 489.08 | bwd_inner_microstep: 488.89 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08
[2024-12-05 14:53:33,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 345.26 | bwd_microstep: 700.84 | bwd_inner_microstep: 700.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:53:34,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.51 | bwd_microstep: 426.96 | bwd_inner_microstep: 426.82 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:53:35,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.30 | bwd_microstep: 612.04 | bwd_inner_microstep: 611.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:53:37,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.30 | optimizer_gradients: 14.14 | optimizer_step: 23.21
[2024-12-05 14:53:37,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.28 | bwd_microstep: 750.73 | bwd_inner_microstep: 738.66 | bwd_allreduce_microstep: 11.97 | step_microstep: 202.93
[2024-12-05 14:53:37,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4582.84 | bwd: 8436.75 | bwd_inner: 8422.94 | bwd_allreduce: 12.64 | step: 204.08

 63%|██████▎   | 242/384 [1:10:33<40:21, 17.05s/it]
                                                   
{'loss': 0.1565, 'grad_norm': 1.7504138946533203, 'learning_rate': 3.184530699967627e-07, 'epoch': 1.26}

 63%|██████▎   | 242/384 [1:10:33<40:21, 17.05s/it][2024-12-05 14:53:38,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.40 | bwd_microstep: 358.77 | bwd_inner_microstep: 358.63 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:53:38,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.55 | bwd_microstep: 409.85 | bwd_inner_microstep: 409.71 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.06
[2024-12-05 14:53:39,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.57 | bwd_microstep: 452.74 | bwd_inner_microstep: 452.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:53:40,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.08 | bwd_microstep: 413.26 | bwd_inner_microstep: 413.10 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:53:41,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 358.27 | bwd_microstep: 438.93 | bwd_inner_microstep: 438.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:53:42,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.34 | bwd_microstep: 406.64 | bwd_inner_microstep: 406.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:53:43,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.26 | bwd_microstep: 445.83 | bwd_inner_microstep: 445.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:53:44,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.21 | bwd_microstep: 355.85 | bwd_inner_microstep: 355.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:53:45,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.97 | bwd_microstep: 569.77 | bwd_inner_microstep: 569.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:53:46,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.35 | bwd_microstep: 447.00 | bwd_inner_microstep: 446.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:53:47,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.65 | bwd_microstep: 403.05 | bwd_inner_microstep: 402.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:53:48,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.30 | bwd_microstep: 413.60 | bwd_inner_microstep: 413.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:53:49,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.65 | bwd_microstep: 487.83 | bwd_inner_microstep: 487.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:53:50,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.85 | bwd_microstep: 583.54 | bwd_inner_microstep: 583.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:53:50,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.19 | bwd_microstep: 440.81 | bwd_inner_microstep: 440.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:53:54,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.38 | optimizer_gradients: 14.05 | optimizer_step: 23.51
[2024-12-05 14:53:54,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.30 | bwd_microstep: 2663.22 | bwd_inner_microstep: 2651.12 | bwd_allreduce_microstep: 11.99 | step_microstep: 112.53
[2024-12-05 14:53:54,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4313.01 | bwd: 9290.69 | bwd_inner: 9276.91 | bwd_allreduce: 12.61 | step: 113.61

 63%|██████▎   | 243/384 [1:10:50<40:05, 17.06s/it]
                                                   
{'loss': 0.1266, 'grad_norm': 1.7226463556289673, 'learning_rate': 3.1452519956511614e-07, 'epoch': 1.26}

 63%|██████▎   | 243/384 [1:10:50<40:05, 17.06s/it][2024-12-05 14:53:55,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.32 | bwd_microstep: 373.01 | bwd_inner_microstep: 372.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:53:56,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.84 | bwd_microstep: 447.75 | bwd_inner_microstep: 447.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:53:56,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.04 | bwd_microstep: 426.52 | bwd_inner_microstep: 426.38 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:53:58,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.92 | bwd_microstep: 455.57 | bwd_inner_microstep: 455.40 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.06
[2024-12-05 14:53:59,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.61 | bwd_microstep: 480.78 | bwd_inner_microstep: 480.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:54:00,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.13 | bwd_microstep: 496.84 | bwd_inner_microstep: 496.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:54:01,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.55 | bwd_microstep: 414.38 | bwd_inner_microstep: 414.25 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:54:02,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 406.42 | bwd_microstep: 898.34 | bwd_inner_microstep: 898.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:54:03,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.48 | bwd_microstep: 399.89 | bwd_inner_microstep: 399.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:54:04,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.22 | bwd_microstep: 411.58 | bwd_inner_microstep: 411.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:54:05,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.31 | bwd_microstep: 502.24 | bwd_inner_microstep: 502.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:54:06,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.74 | bwd_microstep: 359.54 | bwd_inner_microstep: 359.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:54:07,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.57 | bwd_microstep: 542.88 | bwd_inner_microstep: 542.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:54:08,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.52 | bwd_microstep: 500.79 | bwd_inner_microstep: 500.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:54:09,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.93 | bwd_microstep: 465.43 | bwd_inner_microstep: 465.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:54:11,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.57 | optimizer_gradients: 14.13 | optimizer_step: 23.32
[2024-12-05 14:54:11,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.84 | bwd_microstep: 1577.55 | bwd_inner_microstep: 1565.21 | bwd_allreduce_microstep: 12.24 | step_microstep: 112.56
[2024-12-05 14:54:11,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4433.49 | bwd: 8753.07 | bwd_inner: 8739.07 | bwd_allreduce: 12.92 | step: 113.63

 64%|██████▎   | 244/384 [1:11:07<40:03, 17.17s/it]
                                                   
{'loss': 0.1347, 'grad_norm': 1.8186461925506592, 'learning_rate': 3.1061055719471197e-07, 'epoch': 1.27}

 64%|██████▎   | 244/384 [1:11:07<40:03, 17.17s/it][2024-12-05 14:54:12,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.60 | bwd_microstep: 484.25 | bwd_inner_microstep: 484.06 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07
[2024-12-05 14:54:13,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.18 | bwd_microstep: 521.52 | bwd_inner_microstep: 521.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:54:14,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.54 | bwd_microstep: 430.77 | bwd_inner_microstep: 430.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:54:15,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.99 | bwd_microstep: 489.74 | bwd_inner_microstep: 489.60 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:54:16,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.29 | bwd_microstep: 465.50 | bwd_inner_microstep: 465.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:54:17,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.55 | bwd_microstep: 400.83 | bwd_inner_microstep: 400.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:54:18,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.41 | bwd_microstep: 429.36 | bwd_inner_microstep: 429.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:54:19,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.26 | bwd_microstep: 452.93 | bwd_inner_microstep: 452.79 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:54:20,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.26 | bwd_microstep: 437.85 | bwd_inner_microstep: 437.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:54:21,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.11 | bwd_microstep: 456.80 | bwd_inner_microstep: 456.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:54:21,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.40 | bwd_microstep: 353.53 | bwd_inner_microstep: 353.38 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:54:22,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.10 | bwd_microstep: 437.09 | bwd_inner_microstep: 436.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:54:23,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.43 | bwd_microstep: 518.63 | bwd_inner_microstep: 518.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:54:24,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.78 | bwd_microstep: 440.03 | bwd_inner_microstep: 439.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:54:25,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.31 | bwd_microstep: 563.83 | bwd_inner_microstep: 563.69 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:54:28,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 67.80 | optimizer_gradients: 14.05 | optimizer_step: 23.15
[2024-12-05 14:54:28,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.08 | bwd_microstep: 2259.31 | bwd_inner_microstep: 2246.81 | bwd_allreduce_microstep: 12.40 | step_microstep: 121.78
[2024-12-05 14:54:28,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4299.36 | bwd: 9141.98 | bwd_inner: 9127.74 | bwd_allreduce: 13.10 | step: 122.96

 64%|██████▍   | 245/384 [1:11:25<39:44, 17.15s/it]
                                                   
{'loss': 0.1115, 'grad_norm': 1.6384614706039429, 'learning_rate': 3.0670942207781204e-07, 'epoch': 1.27}

 64%|██████▍   | 245/384 [1:11:25<39:44, 17.15s/it][2024-12-05 14:54:29,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.33 | bwd_microstep: 414.75 | bwd_inner_microstep: 414.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:54:31,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.24 | bwd_microstep: 485.95 | bwd_inner_microstep: 485.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:54:31,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.13 | bwd_microstep: 403.65 | bwd_inner_microstep: 403.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:54:32,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.08 | bwd_microstep: 486.01 | bwd_inner_microstep: 485.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:54:33,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.25 | bwd_microstep: 494.68 | bwd_inner_microstep: 494.47 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:54:34,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.17 | bwd_microstep: 482.33 | bwd_inner_microstep: 482.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:54:35,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.36 | bwd_microstep: 347.76 | bwd_inner_microstep: 347.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:54:36,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.77 | bwd_microstep: 453.73 | bwd_inner_microstep: 453.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:54:37,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.99 | bwd_microstep: 449.28 | bwd_inner_microstep: 449.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:54:38,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.88 | bwd_microstep: 398.32 | bwd_inner_microstep: 398.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:54:39,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.71 | bwd_microstep: 464.57 | bwd_inner_microstep: 464.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:54:39,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.94 | bwd_microstep: 436.33 | bwd_inner_microstep: 436.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:54:41,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.60 | bwd_microstep: 451.87 | bwd_inner_microstep: 451.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:54:41,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.81 | bwd_microstep: 374.86 | bwd_inner_microstep: 374.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:54:42,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.13 | bwd_microstep: 353.61 | bwd_inner_microstep: 353.49 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:54:45,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.39 | optimizer_gradients: 14.04 | optimizer_step: 23.41
[2024-12-05 14:54:45,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.58 | bwd_microstep: 2264.64 | bwd_inner_microstep: 2252.42 | bwd_allreduce_microstep: 12.12 | step_microstep: 112.40
[2024-12-05 14:54:45,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4192.02 | bwd: 8762.34 | bwd_inner: 8748.41 | bwd_allreduce: 12.77 | step: 113.43

 64%|██████▍   | 246/384 [1:11:41<39:09, 17.02s/it]
                                                   
{'loss': 0.129, 'grad_norm': 1.7369823455810547, 'learning_rate': 3.028220724433408e-07, 'epoch': 1.28}

 64%|██████▍   | 246/384 [1:11:41<39:09, 17.02s/it][2024-12-05 14:54:46,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.39 | bwd_microstep: 403.40 | bwd_inner_microstep: 403.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:54:47,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.68 | bwd_microstep: 584.65 | bwd_inner_microstep: 584.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:54:48,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.00 | bwd_microstep: 411.24 | bwd_inner_microstep: 411.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:54:49,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.78 | bwd_microstep: 411.59 | bwd_inner_microstep: 411.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:54:50,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.85 | bwd_microstep: 355.21 | bwd_inner_microstep: 355.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:54:51,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.71 | bwd_microstep: 430.67 | bwd_inner_microstep: 430.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:54:52,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 348.51 | bwd_microstep: 721.96 | bwd_inner_microstep: 721.81 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 14:54:53,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.72 | bwd_microstep: 493.20 | bwd_inner_microstep: 493.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:54:54,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.65 | bwd_microstep: 503.87 | bwd_inner_microstep: 503.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:54:55,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.19 | bwd_microstep: 439.78 | bwd_inner_microstep: 439.64 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:54:56,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.26 | bwd_microstep: 569.35 | bwd_inner_microstep: 569.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:54:57,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.10 | bwd_microstep: 412.46 | bwd_inner_microstep: 412.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:54:58,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.78 | bwd_microstep: 625.14 | bwd_inner_microstep: 624.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:55:00,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.04 | bwd_microstep: 625.80 | bwd_inner_microstep: 625.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:55:01,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 378.93 | bwd_microstep: 684.33 | bwd_inner_microstep: 684.19 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:55:03,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.26 | optimizer_gradients: 14.00 | optimizer_step: 23.36
[2024-12-05 14:55:03,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.23 | bwd_microstep: 1500.77 | bwd_inner_microstep: 1488.78 | bwd_allreduce_microstep: 11.89 | step_microstep: 112.40
[2024-12-05 14:55:03,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4568.86 | bwd: 9173.40 | bwd_inner: 9159.72 | bwd_allreduce: 12.54 | step: 113.55

 64%|██████▍   | 247/384 [1:11:59<39:32, 17.32s/it]
                                                   
{'loss': 0.1385, 'grad_norm': 1.6780822277069092, 'learning_rate': 2.989487855370421e-07, 'epoch': 1.28}

 64%|██████▍   | 247/384 [1:11:59<39:32, 17.32s/it][2024-12-05 14:55:04,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.40 | bwd_microstep: 421.83 | bwd_inner_microstep: 421.70 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:55:05,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.20 | bwd_microstep: 437.53 | bwd_inner_microstep: 437.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:06,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.19 | bwd_microstep: 553.43 | bwd_inner_microstep: 553.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:55:07,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.92 | bwd_microstep: 403.27 | bwd_inner_microstep: 403.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:08,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.45 | bwd_microstep: 429.26 | bwd_inner_microstep: 429.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:09,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.71 | bwd_microstep: 488.66 | bwd_inner_microstep: 488.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:10,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.17 | bwd_microstep: 451.81 | bwd_inner_microstep: 451.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:11,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.73 | bwd_microstep: 439.64 | bwd_inner_microstep: 439.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:55:12,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.21 | bwd_microstep: 357.52 | bwd_inner_microstep: 357.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:55:13,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.48 | bwd_microstep: 594.99 | bwd_inner_microstep: 594.82 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 14:55:14,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.05 | bwd_microstep: 403.87 | bwd_inner_microstep: 403.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:15,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.23 | bwd_microstep: 429.25 | bwd_inner_microstep: 429.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:55:16,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.96 | bwd_microstep: 554.09 | bwd_inner_microstep: 553.95 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:55:17,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.08 | bwd_microstep: 533.44 | bwd_inner_microstep: 533.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:18,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.47 | bwd_microstep: 436.11 | bwd_inner_microstep: 435.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.05
[2024-12-05 14:55:21,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.54 | optimizer_gradients: 14.11 | optimizer_step: 23.24
[2024-12-05 14:55:21,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 358.58 | bwd_microstep: 2389.12 | bwd_inner_microstep: 2377.16 | bwd_allreduce_microstep: 11.87 | step_microstep: 112.74
[2024-12-05 14:55:21,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4443.91 | bwd: 9323.82 | bwd_inner: 9310.15 | bwd_allreduce: 12.53 | step: 113.74

 65%|██████▍   | 248/384 [1:12:17<39:43, 17.52s/it]
                                                   
{'loss': 0.1435, 'grad_norm': 1.87428879737854, 'learning_rate': 2.9508983760170634e-07, 'epoch': 1.29}

 65%|██████▍   | 248/384 [1:12:17<39:43, 17.52s/it][2024-12-05 14:55:22,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.94 | bwd_microstep: 579.27 | bwd_inner_microstep: 579.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:23,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.63 | bwd_microstep: 596.64 | bwd_inner_microstep: 596.51 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:55:24,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.22 | bwd_microstep: 430.83 | bwd_inner_microstep: 430.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:25,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.22 | bwd_microstep: 414.26 | bwd_inner_microstep: 414.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:27,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 349.74 | bwd_microstep: 723.32 | bwd_inner_microstep: 723.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:28,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.37 | bwd_microstep: 450.24 | bwd_inner_microstep: 450.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:29,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.02 | bwd_microstep: 571.68 | bwd_inner_microstep: 571.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:30,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.70 | bwd_microstep: 449.83 | bwd_inner_microstep: 449.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:31,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 369.04 | bwd_microstep: 773.76 | bwd_inner_microstep: 773.63 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:55:32,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 244.67 | bwd_microstep: 374.56 | bwd_inner_microstep: 374.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:33,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.84 | bwd_microstep: 593.17 | bwd_inner_microstep: 593.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:34,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.57 | bwd_microstep: 551.01 | bwd_inner_microstep: 550.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:35,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.16 | bwd_microstep: 488.56 | bwd_inner_microstep: 488.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:36,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.53 | bwd_microstep: 429.65 | bwd_inner_microstep: 429.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:37,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.17 | bwd_microstep: 436.50 | bwd_inner_microstep: 436.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:39,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.20 | optimizer_gradients: 14.01 | optimizer_step: 23.43
[2024-12-05 14:55:39,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.49 | bwd_microstep: 1249.52 | bwd_inner_microstep: 1237.61 | bwd_allreduce_microstep: 11.82 | step_microstep: 113.03
[2024-12-05 14:55:39,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4621.35 | bwd: 9112.81 | bwd_inner: 9099.28 | bwd_allreduce: 12.44 | step: 113.93

 65%|██████▍   | 249/384 [1:12:35<39:33, 17.58s/it]
                                                   
{'loss': 0.1491, 'grad_norm': 1.7940468788146973, 'learning_rate': 2.9124550385746856e-07, 'epoch': 1.29}

 65%|██████▍   | 249/384 [1:12:35<39:33, 17.58s/it][2024-12-05 14:55:40,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.00 | bwd_microstep: 417.30 | bwd_inner_microstep: 417.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:41,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.80 | bwd_microstep: 411.55 | bwd_inner_microstep: 411.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:55:42,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.49 | bwd_microstep: 396.02 | bwd_inner_microstep: 395.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:42,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.74 | bwd_microstep: 357.94 | bwd_inner_microstep: 357.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:55:43,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.92 | bwd_microstep: 433.59 | bwd_inner_microstep: 433.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:44,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.95 | bwd_microstep: 449.54 | bwd_inner_microstep: 449.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:45,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.41 | bwd_microstep: 435.01 | bwd_inner_microstep: 434.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:46,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.47 | bwd_microstep: 439.18 | bwd_inner_microstep: 439.04 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:55:47,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.28 | bwd_microstep: 567.20 | bwd_inner_microstep: 567.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:48,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.72 | bwd_microstep: 427.17 | bwd_inner_microstep: 427.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:49,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.81 | bwd_microstep: 426.77 | bwd_inner_microstep: 426.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:50,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.96 | bwd_microstep: 618.50 | bwd_inner_microstep: 618.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:51,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.71 | bwd_microstep: 508.88 | bwd_inner_microstep: 508.73 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:55:53,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.47 | bwd_microstep: 623.95 | bwd_inner_microstep: 623.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:55:53,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.03 | bwd_microstep: 433.75 | bwd_inner_microstep: 433.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:55:57,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.03 | optimizer_gradients: 14.01 | optimizer_step: 23.43
[2024-12-05 14:55:57,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.08 | bwd_microstep: 2966.31 | bwd_inner_microstep: 2954.36 | bwd_allreduce_microstep: 11.86 | step_microstep: 112.12
[2024-12-05 14:55:57,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4362.92 | bwd: 9912.63 | bwd_inner: 9899.05 | bwd_allreduce: 12.50 | step: 113.08

 65%|██████▌   | 250/384 [1:12:53<39:43, 17.79s/it]
                                                   
{'loss': 0.131, 'grad_norm': 1.712099552154541, 'learning_rate': 2.8741605848217976e-07, 'epoch': 1.3}

 65%|██████▌   | 250/384 [1:12:53<39:43, 17.79s/it][2024-12-05 14:55:58,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.57 | bwd_microstep: 521.91 | bwd_inner_microstep: 521.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:55:59,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.98 | bwd_microstep: 472.95 | bwd_inner_microstep: 472.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:00,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.95 | bwd_microstep: 612.86 | bwd_inner_microstep: 612.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:01,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.99 | bwd_microstep: 522.25 | bwd_inner_microstep: 522.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:03,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 370.46 | bwd_microstep: 782.28 | bwd_inner_microstep: 782.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:04,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.65 | bwd_microstep: 430.62 | bwd_inner_microstep: 430.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:05,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.03 | bwd_microstep: 456.05 | bwd_inner_microstep: 455.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:06,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 346.60 | bwd_microstep: 711.34 | bwd_inner_microstep: 711.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:07,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.92 | bwd_microstep: 530.58 | bwd_inner_microstep: 530.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:56:08,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.04 | bwd_microstep: 617.38 | bwd_inner_microstep: 617.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:56:09,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.50 | bwd_microstep: 347.88 | bwd_inner_microstep: 347.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:10,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.79 | bwd_microstep: 589.80 | bwd_inner_microstep: 589.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:11,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.93 | bwd_microstep: 459.70 | bwd_inner_microstep: 459.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:56:12,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.42 | bwd_microstep: 437.56 | bwd_inner_microstep: 437.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:56:13,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.36 | bwd_microstep: 681.64 | bwd_inner_microstep: 681.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:56:15,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.96 | optimizer_gradients: 14.06 | optimizer_step: 23.55
[2024-12-05 14:56:15,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.82 | bwd_microstep: 929.28 | bwd_inner_microstep: 917.26 | bwd_allreduce_microstep: 11.93 | step_microstep: 112.29
[2024-12-05 14:56:15,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4751.04 | bwd: 9104.08 | bwd_inner: 9090.43 | bwd_allreduce: 12.54 | step: 113.26

 65%|██████▌   | 251/384 [1:13:11<39:27, 17.80s/it]
                                                   
{'loss': 0.1145, 'grad_norm': 1.6154824495315552, 'learning_rate': 2.8360177459185263e-07, 'epoch': 1.3}

 65%|██████▌   | 251/384 [1:13:11<39:27, 17.80s/it][2024-12-05 14:56:16,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.73 | bwd_microstep: 555.14 | bwd_inner_microstep: 555.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:56:17,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.02 | bwd_microstep: 489.39 | bwd_inner_microstep: 489.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:18,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.03 | bwd_microstep: 410.39 | bwd_inner_microstep: 410.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:56:19,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.02 | bwd_microstep: 413.02 | bwd_inner_microstep: 412.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:19,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.20 | bwd_microstep: 343.47 | bwd_inner_microstep: 343.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:21,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.11 | bwd_microstep: 730.28 | bwd_inner_microstep: 730.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:22,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.97 | bwd_microstep: 678.82 | bwd_inner_microstep: 678.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:56:23,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.11 | bwd_microstep: 641.71 | bwd_inner_microstep: 641.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:56:24,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.60 | bwd_microstep: 592.42 | bwd_inner_microstep: 592.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:56:25,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.55 | bwd_microstep: 398.12 | bwd_inner_microstep: 398.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:56:26,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.22 | bwd_microstep: 622.83 | bwd_inner_microstep: 622.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:56:27,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.77 | bwd_microstep: 593.38 | bwd_inner_microstep: 593.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:28,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.27 | bwd_microstep: 521.82 | bwd_inner_microstep: 521.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:29,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.73 | bwd_microstep: 399.20 | bwd_inner_microstep: 399.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:30,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.88 | bwd_microstep: 487.95 | bwd_inner_microstep: 487.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:32,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.67 | optimizer_gradients: 14.02 | optimizer_step: 23.18
[2024-12-05 14:56:32,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.19 | bwd_microstep: 901.01 | bwd_inner_microstep: 888.86 | bwd_allreduce_microstep: 12.05 | step_microstep: 113.99
[2024-12-05 14:56:32,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4714.50 | bwd: 8778.94 | bwd_inner: 8765.19 | bwd_allreduce: 12.68 | step: 115.01

 66%|██████▌   | 252/384 [1:13:28<38:50, 17.65s/it]
                                                   
{'loss': 0.1232, 'grad_norm': 1.6371185779571533, 'learning_rate': 2.7980292422118277e-07, 'epoch': 1.31}

 66%|██████▌   | 252/384 [1:13:28<38:50, 17.65s/it][2024-12-05 14:56:33,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.83 | bwd_microstep: 423.41 | bwd_inner_microstep: 423.27 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:56:34,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.75 | bwd_microstep: 612.24 | bwd_inner_microstep: 612.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:56:35,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.71 | bwd_microstep: 480.67 | bwd_inner_microstep: 480.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:56:36,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.08 | bwd_microstep: 448.38 | bwd_inner_microstep: 448.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:56:37,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.11 | bwd_microstep: 479.15 | bwd_inner_microstep: 479.01 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:56:38,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.85 | bwd_microstep: 568.01 | bwd_inner_microstep: 567.86 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:56:39,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.74 | bwd_microstep: 438.40 | bwd_inner_microstep: 438.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:40,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.16 | bwd_microstep: 326.41 | bwd_inner_microstep: 326.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:56:42,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 387.53 | bwd_microstep: 870.85 | bwd_inner_microstep: 870.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:56:43,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.30 | bwd_microstep: 433.73 | bwd_inner_microstep: 433.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:56:44,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.29 | bwd_microstep: 556.22 | bwd_inner_microstep: 556.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:56:45,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.32 | bwd_microstep: 486.34 | bwd_inner_microstep: 486.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:56:46,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.28 | bwd_microstep: 621.12 | bwd_inner_microstep: 620.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:56:47,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.21 | bwd_microstep: 432.17 | bwd_inner_microstep: 432.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:56:48,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.42 | bwd_microstep: 378.55 | bwd_inner_microstep: 378.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:56:50,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.21 | optimizer_gradients: 14.07 | optimizer_step: 23.24
[2024-12-05 14:56:50,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.72 | bwd_microstep: 959.23 | bwd_inner_microstep: 947.03 | bwd_allreduce_microstep: 12.11 | step_microstep: 112.03
[2024-12-05 14:56:50,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4482.37 | bwd: 8514.88 | bwd_inner: 8501.04 | bwd_allreduce: 12.75 | step: 113.17

 66%|██████▌   | 253/384 [1:13:46<38:26, 17.61s/it]
                                                   
{'loss': 0.1274, 'grad_norm': 1.8156847953796387, 'learning_rate': 2.7601977830414766e-07, 'epoch': 1.32}

 66%|██████▌   | 253/384 [1:13:46<38:26, 17.61s/it][2024-12-05 14:56:51,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 344.20 | bwd_microstep: 678.08 | bwd_inner_microstep: 677.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:56:52,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.89 | bwd_microstep: 614.68 | bwd_inner_microstep: 614.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:56:54,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.44 | bwd_microstep: 438.36 | bwd_inner_microstep: 438.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 14:56:55,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.42 | bwd_microstep: 700.99 | bwd_inner_microstep: 700.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:56,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.71 | bwd_microstep: 689.03 | bwd_inner_microstep: 688.90 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:56:58,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 365.58 | bwd_microstep: 775.85 | bwd_inner_microstep: 775.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:56:59,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.75 | bwd_microstep: 423.47 | bwd_inner_microstep: 423.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:57:00,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.33 | bwd_microstep: 429.51 | bwd_inner_microstep: 429.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:57:01,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.83 | bwd_microstep: 588.40 | bwd_inner_microstep: 588.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:57:02,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.64 | bwd_microstep: 475.00 | bwd_inner_microstep: 474.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:57:03,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.45 | bwd_microstep: 422.34 | bwd_inner_microstep: 422.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:57:04,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.71 | bwd_microstep: 422.45 | bwd_inner_microstep: 422.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:57:06,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.66 | bwd_microstep: 478.43 | bwd_inner_microstep: 478.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:57:07,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.90 | bwd_microstep: 669.24 | bwd_inner_microstep: 669.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:57:08,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.51 | bwd_microstep: 628.82 | bwd_inner_microstep: 628.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:57:10,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.56 | optimizer_gradients: 14.07 | optimizer_step: 23.06
[2024-12-05 14:57:10,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.69 | bwd_microstep: 1174.71 | bwd_inner_microstep: 1162.70 | bwd_allreduce_microstep: 11.92 | step_microstep: 112.28
[2024-12-05 14:57:10,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4846.78 | bwd: 9609.35 | bwd_inner: 9595.76 | bwd_allreduce: 12.51 | step: 113.28

 66%|██████▌   | 254/384 [1:14:07<40:14, 18.57s/it]
                                                   
{'loss': 0.1168, 'grad_norm': 1.6596229076385498, 'learning_rate': 2.72252606654683e-07, 'epoch': 1.32}

 66%|██████▌   | 254/384 [1:14:07<40:14, 18.57s/it][2024-12-05 14:57:12,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.20 | bwd_microstep: 555.24 | bwd_inner_microstep: 555.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:57:13,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.28 | bwd_microstep: 415.48 | bwd_inner_microstep: 415.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:57:14,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.55 | bwd_microstep: 480.41 | bwd_inner_microstep: 480.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:57:15,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.60 | bwd_microstep: 523.05 | bwd_inner_microstep: 522.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:57:16,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.28 | bwd_microstep: 431.12 | bwd_inner_microstep: 430.97 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:57:17,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.07 | bwd_microstep: 430.86 | bwd_inner_microstep: 430.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:57:18,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.17 | bwd_microstep: 448.79 | bwd_inner_microstep: 448.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:57:19,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.38 | bwd_microstep: 446.67 | bwd_inner_microstep: 446.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:57:21,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.56 | bwd_microstep: 684.97 | bwd_inner_microstep: 684.84 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 14:57:21,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.58 | bwd_microstep: 432.71 | bwd_inner_microstep: 432.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:57:23,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.78 | bwd_microstep: 411.56 | bwd_inner_microstep: 411.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:57:24,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.61 | bwd_microstep: 486.78 | bwd_inner_microstep: 486.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:57:25,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.27 | bwd_microstep: 685.22 | bwd_inner_microstep: 685.09 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:57:26,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.97 | bwd_microstep: 490.81 | bwd_inner_microstep: 490.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:57:27,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.32 | bwd_microstep: 437.55 | bwd_inner_microstep: 437.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:57:31,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.92 | optimizer_gradients: 14.05 | optimizer_step: 23.07
[2024-12-05 14:57:31,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.25 | bwd_microstep: 3508.92 | bwd_inner_microstep: 3496.79 | bwd_allreduce_microstep: 12.02 | step_microstep: 113.23
[2024-12-05 14:57:31,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4430.95 | bwd: 10870.14 | bwd_inner: 10856.37 | bwd_allreduce: 12.65 | step: 114.32

 66%|██████▋   | 255/384 [1:14:28<41:28, 19.29s/it]
                                                   
{'loss': 0.1139, 'grad_norm': 1.6106423139572144, 'learning_rate': 2.685016779474396e-07, 'epoch': 1.33}

 66%|██████▋   | 255/384 [1:14:28<41:28, 19.29s/it][2024-12-05 14:57:32,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.37 | bwd_microstep: 408.01 | bwd_inner_microstep: 407.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:57:33,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.29 | bwd_microstep: 454.02 | bwd_inner_microstep: 453.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:57:35,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.53 | bwd_microstep: 493.94 | bwd_inner_microstep: 493.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:57:36,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.84 | bwd_microstep: 435.79 | bwd_inner_microstep: 435.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:57:36,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.32 | bwd_microstep: 484.43 | bwd_inner_microstep: 484.29 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:57:37,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.19 | bwd_microstep: 339.65 | bwd_inner_microstep: 339.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:57:39,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.47 | bwd_microstep: 485.94 | bwd_inner_microstep: 485.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:57:40,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.15 | bwd_microstep: 430.61 | bwd_inner_microstep: 430.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:57:41,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.01 | bwd_microstep: 558.95 | bwd_inner_microstep: 558.78 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 14:57:42,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.53 | bwd_microstep: 578.85 | bwd_inner_microstep: 578.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:57:43,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.07 | bwd_microstep: 447.97 | bwd_inner_microstep: 447.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:57:45,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 363.40 | bwd_microstep: 776.94 | bwd_inner_microstep: 776.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:57:46,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.49 | bwd_microstep: 471.90 | bwd_inner_microstep: 471.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:57:47,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.78 | bwd_microstep: 424.80 | bwd_inner_microstep: 424.65 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:57:48,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.51 | bwd_microstep: 409.50 | bwd_inner_microstep: 409.31 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09
[2024-12-05 14:57:50,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.41 | optimizer_gradients: 14.02 | optimizer_step: 23.11
[2024-12-05 14:57:50,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.48 | bwd_microstep: 1441.83 | bwd_inner_microstep: 1429.68 | bwd_allreduce_microstep: 12.06 | step_microstep: 112.11
[2024-12-05 14:57:50,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4388.48 | bwd: 8643.11 | bwd_inner: 8629.22 | bwd_allreduce: 12.77 | step: 113.28

 67%|██████▋   | 256/384 [1:14:46<40:42, 19.08s/it]
                                                   
{'loss': 0.1297, 'grad_norm': 1.6363728046417236, 'learning_rate': 2.6476725969862226e-07, 'epoch': 1.33}

 67%|██████▋   | 256/384 [1:14:46<40:42, 19.08s/it][2024-12-05 14:57:51,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.97 | bwd_microstep: 396.43 | bwd_inner_microstep: 396.29 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 14:57:52,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.46 | bwd_microstep: 484.09 | bwd_inner_microstep: 483.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:57:53,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.86 | bwd_microstep: 460.48 | bwd_inner_microstep: 460.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:57:54,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.94 | bwd_microstep: 439.68 | bwd_inner_microstep: 439.49 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 14:57:55,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.62 | bwd_microstep: 490.22 | bwd_inner_microstep: 490.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:57:57,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 350.62 | bwd_microstep: 716.94 | bwd_inner_microstep: 716.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:57:58,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.63 | bwd_microstep: 430.88 | bwd_inner_microstep: 430.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:57:59,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.07 | bwd_microstep: 561.92 | bwd_inner_microstep: 561.79 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.05
[2024-12-05 14:58:00,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.04 | bwd_microstep: 400.83 | bwd_inner_microstep: 400.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:58:01,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.91 | bwd_microstep: 453.51 | bwd_inner_microstep: 453.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:58:03,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.80 | bwd_microstep: 677.27 | bwd_inner_microstep: 677.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:58:04,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.69 | bwd_microstep: 458.71 | bwd_inner_microstep: 458.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.05
[2024-12-05 14:58:05,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.34 | bwd_microstep: 409.50 | bwd_inner_microstep: 409.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:58:06,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.03 | bwd_microstep: 427.31 | bwd_inner_microstep: 427.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:58:07,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.22 | bwd_microstep: 449.48 | bwd_inner_microstep: 449.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:58:09,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.67 | optimizer_gradients: 14.11 | optimizer_step: 23.34
[2024-12-05 14:58:09,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.43 | bwd_microstep: 1595.48 | bwd_inner_microstep: 1582.91 | bwd_allreduce_microstep: 12.47 | step_microstep: 113.93
[2024-12-05 14:58:09,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4438.70 | bwd: 8852.74 | bwd_inner: 8838.50 | bwd_allreduce: 13.10 | step: 114.96

 67%|██████▋   | 257/384 [1:15:06<40:36, 19.18s/it]
                                                   
{'loss': 0.1191, 'grad_norm': 1.4978957176208496, 'learning_rate': 2.6104961824690964e-07, 'epoch': 1.34}

 67%|██████▋   | 257/384 [1:15:06<40:36, 19.18s/it][2024-12-05 14:58:11,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.78 | bwd_microstep: 403.96 | bwd_inner_microstep: 403.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:58:11,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.85 | bwd_microstep: 437.81 | bwd_inner_microstep: 437.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:58:12,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.31 | bwd_microstep: 487.35 | bwd_inner_microstep: 487.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:58:13,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.40 | bwd_microstep: 486.95 | bwd_inner_microstep: 486.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:58:15,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.29 | bwd_microstep: 608.33 | bwd_inner_microstep: 608.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:58:16,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.30 | bwd_microstep: 621.47 | bwd_inner_microstep: 621.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:58:17,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.36 | bwd_microstep: 403.67 | bwd_inner_microstep: 403.54 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:58:19,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.97 | bwd_microstep: 521.60 | bwd_inner_microstep: 521.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:58:20,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.08 | bwd_microstep: 538.77 | bwd_inner_microstep: 538.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:58:21,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.58 | bwd_microstep: 402.77 | bwd_inner_microstep: 402.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:58:22,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.42 | bwd_microstep: 352.30 | bwd_inner_microstep: 352.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:58:23,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.98 | bwd_microstep: 423.26 | bwd_inner_microstep: 423.12 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:58:24,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.40 | bwd_microstep: 628.68 | bwd_inner_microstep: 628.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:58:25,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.87 | bwd_microstep: 399.37 | bwd_inner_microstep: 399.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:58:26,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.62 | bwd_microstep: 434.19 | bwd_inner_microstep: 434.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:58:29,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.44 | optimizer_gradients: 14.06 | optimizer_step: 23.25
[2024-12-05 14:58:29,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.36 | bwd_microstep: 1891.35 | bwd_inner_microstep: 1879.10 | bwd_allreduce_microstep: 12.17 | step_microstep: 112.22
[2024-12-05 14:58:29,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4410.63 | bwd: 9041.82 | bwd_inner: 9027.88 | bwd_allreduce: 12.81 | step: 113.31

 67%|██████▋   | 258/384 [1:15:25<40:19, 19.20s/it]
                                                   
{'loss': 0.1236, 'grad_norm': 1.5889379978179932, 'learning_rate': 2.5734901873445956e-07, 'epoch': 1.34}

 67%|██████▋   | 258/384 [1:15:25<40:19, 19.20s/it][2024-12-05 14:58:30,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.05 | bwd_microstep: 383.33 | bwd_inner_microstep: 383.18 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:58:31,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.20 | bwd_microstep: 436.38 | bwd_inner_microstep: 436.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:58:32,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.43 | bwd_microstep: 430.88 | bwd_inner_microstep: 430.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:58:33,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.89 | bwd_microstep: 400.08 | bwd_inner_microstep: 399.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:58:34,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.38 | bwd_microstep: 684.20 | bwd_inner_microstep: 684.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:58:35,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.57 | bwd_microstep: 453.46 | bwd_inner_microstep: 453.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:58:36,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.36 | bwd_microstep: 461.84 | bwd_inner_microstep: 461.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:58:37,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.62 | bwd_microstep: 481.65 | bwd_inner_microstep: 481.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:58:38,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.19 | bwd_microstep: 488.99 | bwd_inner_microstep: 488.84 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11
[2024-12-05 14:58:39,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.23 | bwd_microstep: 517.03 | bwd_inner_microstep: 516.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:58:40,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.56 | bwd_microstep: 396.77 | bwd_inner_microstep: 396.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:58:41,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.49 | bwd_microstep: 434.77 | bwd_inner_microstep: 434.62 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:58:42,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.70 | bwd_microstep: 400.97 | bwd_inner_microstep: 400.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:58:43,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.12 | bwd_microstep: 405.30 | bwd_inner_microstep: 405.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:58:44,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.02 | bwd_microstep: 402.67 | bwd_inner_microstep: 402.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 14:58:47,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.46 | optimizer_gradients: 14.04 | optimizer_step: 23.29
[2024-12-05 14:58:47,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.06 | bwd_microstep: 2220.42 | bwd_inner_microstep: 2208.43 | bwd_allreduce_microstep: 11.90 | step_microstep: 112.56
[2024-12-05 14:58:47,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4277.94 | bwd: 8998.73 | bwd_inner: 8985.08 | bwd_allreduce: 12.57 | step: 113.76

 67%|██████▋   | 259/384 [1:15:43<39:28, 18.95s/it]
                                                   
{'loss': 0.1053, 'grad_norm': 1.534178376197815, 'learning_rate': 2.536657250879988e-07, 'epoch': 1.35}

 67%|██████▋   | 259/384 [1:15:43<39:28, 18.95s/it][2024-12-05 14:58:48,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.44 | bwd_microstep: 397.71 | bwd_inner_microstep: 397.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:58:49,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.72 | bwd_microstep: 437.84 | bwd_inner_microstep: 437.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:58:50,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 229.59 | bwd_microstep: 338.38 | bwd_inner_microstep: 338.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:58:51,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.19 | bwd_microstep: 689.38 | bwd_inner_microstep: 689.21 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 14:58:52,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.56 | bwd_microstep: 463.94 | bwd_inner_microstep: 463.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:58:53,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.94 | bwd_microstep: 488.10 | bwd_inner_microstep: 487.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:58:54,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.20 | bwd_microstep: 542.08 | bwd_inner_microstep: 541.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:58:55,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.76 | bwd_microstep: 492.95 | bwd_inner_microstep: 492.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:58:56,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.37 | bwd_microstep: 411.84 | bwd_inner_microstep: 411.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:58:57,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.16 | bwd_microstep: 593.67 | bwd_inner_microstep: 593.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:58:58,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.29 | bwd_microstep: 361.06 | bwd_inner_microstep: 360.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:59:00,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.18 | bwd_microstep: 454.57 | bwd_inner_microstep: 454.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:59:01,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.26 | bwd_microstep: 559.73 | bwd_inner_microstep: 559.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:59:02,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.97 | bwd_microstep: 398.69 | bwd_inner_microstep: 398.55 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:59:03,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.86 | bwd_microstep: 491.17 | bwd_inner_microstep: 491.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:59:06,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.19 | optimizer_gradients: 14.30 | optimizer_step: 23.76
[2024-12-05 14:59:06,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.63 | bwd_microstep: 2450.69 | bwd_inner_microstep: 2438.21 | bwd_allreduce_microstep: 12.37 | step_microstep: 116.03
[2024-12-05 14:59:06,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4361.19 | bwd: 9571.80 | bwd_inner: 9557.65 | bwd_allreduce: 13.00 | step: 117.09

 68%|██████▊   | 260/384 [1:16:03<39:25, 19.07s/it]
                                                   
{'loss': 0.1427, 'grad_norm': 1.8409833908081055, 'learning_rate': 2.500000000000001e-07, 'epoch': 1.35}

 68%|██████▊   | 260/384 [1:16:03<39:25, 19.07s/it][2024-12-05 14:59:08,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.86 | bwd_microstep: 448.25 | bwd_inner_microstep: 448.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:59:09,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.18 | bwd_microstep: 462.35 | bwd_inner_microstep: 462.17 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 14:59:10,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.81 | bwd_microstep: 406.96 | bwd_inner_microstep: 406.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:59:12,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.99 | bwd_microstep: 479.82 | bwd_inner_microstep: 479.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:59:13,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.65 | bwd_microstep: 505.10 | bwd_inner_microstep: 504.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:59:14,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.59 | bwd_microstep: 396.11 | bwd_inner_microstep: 395.97 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:59:15,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.75 | bwd_microstep: 432.08 | bwd_inner_microstep: 431.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:59:16,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.86 | bwd_microstep: 582.52 | bwd_inner_microstep: 582.39 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:59:17,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.58 | bwd_microstep: 481.32 | bwd_inner_microstep: 481.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:59:18,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.60 | bwd_microstep: 414.52 | bwd_inner_microstep: 414.38 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:59:19,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.47 | bwd_microstep: 434.94 | bwd_inner_microstep: 434.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:59:20,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.02 | bwd_microstep: 448.44 | bwd_inner_microstep: 448.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:59:21,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.14 | bwd_microstep: 440.97 | bwd_inner_microstep: 440.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:59:23,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.93 | bwd_microstep: 447.32 | bwd_inner_microstep: 447.17 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:59:24,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.36 | bwd_microstep: 413.05 | bwd_inner_microstep: 412.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:59:27,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.48 | optimizer_gradients: 14.03 | optimizer_step: 23.26
[2024-12-05 14:59:27,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.58 | bwd_microstep: 2277.55 | bwd_inner_microstep: 2265.52 | bwd_allreduce_microstep: 11.94 | step_microstep: 112.07
[2024-12-05 14:59:27,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4290.45 | bwd: 9071.29 | bwd_inner: 9057.57 | bwd_allreduce: 12.60 | step: 113.08

 68%|██████▊   | 261/384 [1:16:23<40:07, 19.57s/it]
                                                   
{'loss': 0.1194, 'grad_norm': 1.7446589469909668, 'learning_rate': 2.4635210490994647e-07, 'epoch': 1.36}

 68%|██████▊   | 261/384 [1:16:23<40:07, 19.57s/it][2024-12-05 14:59:28,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.63 | bwd_microstep: 406.93 | bwd_inner_microstep: 406.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:59:29,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.97 | bwd_microstep: 396.50 | bwd_inner_microstep: 396.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:59:30,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.51 | bwd_microstep: 463.96 | bwd_inner_microstep: 463.80 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 14:59:31,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.23 | bwd_microstep: 404.60 | bwd_inner_microstep: 404.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:59:33,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.82 | bwd_microstep: 586.34 | bwd_inner_microstep: 586.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:59:34,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.56 | bwd_microstep: 448.94 | bwd_inner_microstep: 448.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:59:35,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 345.42 | bwd_microstep: 707.14 | bwd_inner_microstep: 707.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:59:36,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.69 | bwd_microstep: 445.29 | bwd_inner_microstep: 445.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:59:37,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.67 | bwd_microstep: 398.23 | bwd_inner_microstep: 398.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:59:39,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.77 | bwd_microstep: 583.51 | bwd_inner_microstep: 583.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:59:40,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.56 | bwd_microstep: 421.13 | bwd_inner_microstep: 421.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:59:41,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.08 | bwd_microstep: 683.96 | bwd_inner_microstep: 683.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:59:42,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.14 | bwd_microstep: 450.02 | bwd_inner_microstep: 449.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:59:43,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.45 | bwd_microstep: 408.14 | bwd_inner_microstep: 408.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:59:44,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.16 | bwd_microstep: 354.41 | bwd_inner_microstep: 354.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:59:46,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.25 | optimizer_gradients: 14.05 | optimizer_step: 23.28
[2024-12-05 14:59:46,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.44 | bwd_microstep: 1154.69 | bwd_inner_microstep: 1142.61 | bwd_allreduce_microstep: 11.99 | step_microstep: 112.02
[2024-12-05 14:59:46,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4379.16 | bwd: 8313.78 | bwd_inner: 8300.11 | bwd_allreduce: 12.63 | step: 113.12

 68%|██████▊   | 262/384 [1:16:42<39:18, 19.33s/it]
                                                   
{'loss': 0.1351, 'grad_norm': 1.7886688709259033, 'learning_rate': 2.427222999856857e-07, 'epoch': 1.36}

 68%|██████▊   | 262/384 [1:16:42<39:18, 19.33s/it][2024-12-05 14:59:47,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.10 | bwd_microstep: 451.99 | bwd_inner_microstep: 451.85 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:59:48,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.75 | bwd_microstep: 713.18 | bwd_inner_microstep: 713.06 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 14:59:50,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.44 | bwd_microstep: 678.46 | bwd_inner_microstep: 678.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:59:51,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.74 | bwd_microstep: 692.75 | bwd_inner_microstep: 692.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 14:59:52,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.86 | bwd_microstep: 437.65 | bwd_inner_microstep: 437.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:59:53,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.17 | bwd_microstep: 443.36 | bwd_inner_microstep: 443.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:59:54,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.45 | bwd_microstep: 430.85 | bwd_inner_microstep: 430.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 14:59:55,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.63 | bwd_microstep: 453.88 | bwd_inner_microstep: 453.74 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 14:59:56,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.99 | bwd_microstep: 618.42 | bwd_inner_microstep: 618.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:59:57,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.15 | bwd_microstep: 415.60 | bwd_inner_microstep: 415.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:59:58,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.84 | bwd_microstep: 447.67 | bwd_inner_microstep: 447.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 14:59:59,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.83 | bwd_microstep: 437.43 | bwd_inner_microstep: 437.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:00:00,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.42 | bwd_microstep: 687.98 | bwd_inner_microstep: 687.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:00:01,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.89 | bwd_microstep: 448.92 | bwd_inner_microstep: 448.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:00:02,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.66 | bwd_microstep: 433.54 | bwd_inner_microstep: 433.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:00:04,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.70 | optimizer_gradients: 14.10 | optimizer_step: 23.11
[2024-12-05 15:00:04,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.68 | bwd_microstep: 1262.44 | bwd_inner_microstep: 1250.19 | bwd_allreduce_microstep: 12.15 | step_microstep: 112.62
[2024-12-05 15:00:04,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4589.68 | bwd: 9054.13 | bwd_inner: 9040.20 | bwd_allreduce: 12.80 | step: 113.69

 68%|██████▊   | 263/384 [1:17:00<37:58, 18.83s/it]
                                                   
{'loss': 0.1249, 'grad_norm': 1.6462031602859497, 'learning_rate': 2.391108441048753e-07, 'epoch': 1.37}

 68%|██████▊   | 263/384 [1:17:00<37:58, 18.83s/it][2024-12-05 15:00:05,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.35 | bwd_microstep: 454.54 | bwd_inner_microstep: 454.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:00:06,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.13 | bwd_microstep: 568.46 | bwd_inner_microstep: 568.31 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:00:06,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.98 | bwd_microstep: 439.80 | bwd_inner_microstep: 439.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:00:08,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.09 | bwd_microstep: 606.33 | bwd_inner_microstep: 606.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:00:09,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.86 | bwd_microstep: 405.12 | bwd_inner_microstep: 404.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:00:10,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.59 | bwd_microstep: 695.02 | bwd_inner_microstep: 694.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:00:11,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.68 | bwd_microstep: 359.16 | bwd_inner_microstep: 359.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:00:12,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.96 | bwd_microstep: 526.36 | bwd_inner_microstep: 526.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 15:00:12,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.64 | bwd_microstep: 452.01 | bwd_inner_microstep: 451.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:00:14,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.73 | bwd_microstep: 625.59 | bwd_inner_microstep: 625.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:00:15,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.28 | bwd_microstep: 435.64 | bwd_inner_microstep: 435.46 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10
[2024-12-05 15:00:16,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 366.06 | bwd_microstep: 773.44 | bwd_inner_microstep: 773.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:00:17,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.75 | bwd_microstep: 439.41 | bwd_inner_microstep: 439.27 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:00:18,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.52 | bwd_microstep: 448.58 | bwd_inner_microstep: 448.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:00:19,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.95 | bwd_microstep: 475.98 | bwd_inner_microstep: 475.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:00:20,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.11 | optimizer_gradients: 14.05 | optimizer_step: 23.26
[2024-12-05 15:00:20,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.76 | bwd_microstep: 907.68 | bwd_inner_microstep: 895.51 | bwd_allreduce_microstep: 12.07 | step_microstep: 111.84
[2024-12-05 15:00:20,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4552.40 | bwd: 8613.10 | bwd_inner: 8599.18 | bwd_allreduce: 12.74 | step: 113.11

 69%|██████▉   | 264/384 [1:17:16<36:17, 18.14s/it]
                                                   
{'loss': 0.1482, 'grad_norm': 1.8700019121170044, 'learning_rate': 2.355179948365189e-07, 'epoch': 1.37}

 69%|██████▉   | 264/384 [1:17:16<36:17, 18.14s/it][2024-12-05 15:00:21,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.67 | bwd_microstep: 378.74 | bwd_inner_microstep: 378.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:00:22,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.38 | bwd_microstep: 534.25 | bwd_inner_microstep: 534.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:00:23,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.02 | bwd_microstep: 438.39 | bwd_inner_microstep: 438.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:00:24,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.66 | bwd_microstep: 442.45 | bwd_inner_microstep: 442.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:00:25,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.19 | bwd_microstep: 445.03 | bwd_inner_microstep: 444.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:00:26,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.89 | bwd_microstep: 411.69 | bwd_inner_microstep: 411.56 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:00:27,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.65 | bwd_microstep: 527.47 | bwd_inner_microstep: 527.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:00:27,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.08 | bwd_microstep: 354.31 | bwd_inner_microstep: 354.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:00:28,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.71 | bwd_microstep: 399.43 | bwd_inner_microstep: 399.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:00:29,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.80 | bwd_microstep: 609.50 | bwd_inner_microstep: 609.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:00:30,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.04 | bwd_microstep: 583.47 | bwd_inner_microstep: 583.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:00:31,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.14 | bwd_microstep: 435.19 | bwd_inner_microstep: 435.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:00:32,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.49 | bwd_microstep: 389.34 | bwd_inner_microstep: 389.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:00:33,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.98 | bwd_microstep: 348.22 | bwd_inner_microstep: 348.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:00:33,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.62 | bwd_microstep: 443.33 | bwd_inner_microstep: 443.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:00:36,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.43 | optimizer_gradients: 14.00 | optimizer_step: 23.20
[2024-12-05 15:00:36,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.93 | bwd_microstep: 2101.83 | bwd_inner_microstep: 2089.90 | bwd_allreduce_microstep: 11.82 | step_microstep: 113.00
[2024-12-05 15:00:36,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4273.28 | bwd: 8842.64 | bwd_inner: 8829.13 | bwd_allreduce: 12.45 | step: 114.10

 69%|██████▉   | 265/384 [1:17:32<34:37, 17.46s/it]
                                                   
{'loss': 0.134, 'grad_norm': 1.8244132995605469, 'learning_rate': 2.3194400842259687e-07, 'epoch': 1.38}

 69%|██████▉   | 265/384 [1:17:32<34:37, 17.46s/it][2024-12-05 15:00:37,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 379.79 | bwd_microstep: 803.14 | bwd_inner_microstep: 803.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:00:38,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.61 | bwd_microstep: 428.24 | bwd_inner_microstep: 428.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:00:40,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 473.18 | bwd_microstep: 1130.79 | bwd_inner_microstep: 1130.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:00:41,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.63 | bwd_microstep: 553.25 | bwd_inner_microstep: 553.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:00:42,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 367.08 | bwd_microstep: 772.38 | bwd_inner_microstep: 772.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:00:43,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.32 | bwd_microstep: 395.00 | bwd_inner_microstep: 394.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:00:44,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.00 | bwd_microstep: 474.38 | bwd_inner_microstep: 474.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:00:45,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.64 | bwd_microstep: 482.04 | bwd_inner_microstep: 481.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:00:45,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.86 | bwd_microstep: 397.52 | bwd_inner_microstep: 397.37 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.06
[2024-12-05 15:00:46,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.02 | bwd_microstep: 432.90 | bwd_inner_microstep: 432.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:00:47,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.89 | bwd_microstep: 585.75 | bwd_inner_microstep: 585.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:00:48,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.01 | bwd_microstep: 476.59 | bwd_inner_microstep: 476.47 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:00:49,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.14 | bwd_microstep: 574.68 | bwd_inner_microstep: 574.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:00:50,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.43 | bwd_microstep: 395.81 | bwd_inner_microstep: 395.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:00:51,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.95 | bwd_microstep: 476.00 | bwd_inner_microstep: 475.86 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 15:00:52,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.02 | optimizer_gradients: 14.04 | optimizer_step: 23.11
[2024-12-05 15:00:52,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.15 | bwd_microstep: 933.48 | bwd_inner_microstep: 921.38 | bwd_allreduce_microstep: 12.03 | step_microstep: 114.13
[2024-12-05 15:00:52,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4779.76 | bwd: 9311.95 | bwd_inner: 9298.25 | bwd_allreduce: 12.65 | step: 115.12

 69%|██████▉   | 266/384 [1:17:49<33:40, 17.12s/it]
                                                   
{'loss': 0.1258, 'grad_norm': 1.8189918994903564, 'learning_rate': 2.283891397597908e-07, 'epoch': 1.38}

 69%|██████▉   | 266/384 [1:17:49<33:40, 17.12s/it][2024-12-05 15:00:53,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 241.65 | bwd_microstep: 336.11 | bwd_inner_microstep: 335.97 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:00:54,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.75 | bwd_microstep: 456.74 | bwd_inner_microstep: 456.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:00:55,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.00 | bwd_microstep: 401.85 | bwd_inner_microstep: 401.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:00:56,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.09 | bwd_microstep: 677.70 | bwd_inner_microstep: 677.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:00:57,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.87 | bwd_microstep: 520.53 | bwd_inner_microstep: 520.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:00:58,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.49 | bwd_microstep: 379.73 | bwd_inner_microstep: 379.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:00:59,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.41 | bwd_microstep: 454.94 | bwd_inner_microstep: 454.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:00:59,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.53 | bwd_microstep: 352.71 | bwd_inner_microstep: 352.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:01:00,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.43 | bwd_microstep: 483.30 | bwd_inner_microstep: 483.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:01:02,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.99 | bwd_microstep: 622.71 | bwd_inner_microstep: 622.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:01:03,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.23 | bwd_microstep: 677.11 | bwd_inner_microstep: 676.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:01:04,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.28 | bwd_microstep: 479.04 | bwd_inner_microstep: 478.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 15:01:04,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.80 | bwd_microstep: 415.46 | bwd_inner_microstep: 415.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:01:05,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.87 | bwd_microstep: 425.41 | bwd_inner_microstep: 425.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:01:06,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.27 | bwd_microstep: 456.62 | bwd_inner_microstep: 456.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:01:09,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.23 | optimizer_gradients: 14.01 | optimizer_step: 23.06
[2024-12-05 15:01:09,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.65 | bwd_microstep: 2288.45 | bwd_inner_microstep: 2276.28 | bwd_allreduce_microstep: 12.07 | step_microstep: 111.63
[2024-12-05 15:01:09,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4390.39 | bwd: 9428.39 | bwd_inner: 9414.56 | bwd_allreduce: 12.71 | step: 112.74

 70%|██████▉   | 267/384 [1:18:05<33:00, 16.93s/it]
                                                   
{'loss': 0.1131, 'grad_norm': 1.5552480220794678, 'learning_rate': 2.2485364238130433e-07, 'epoch': 1.39}

 70%|██████▉   | 267/384 [1:18:05<33:00, 16.93s/it][2024-12-05 15:01:10,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.21 | bwd_microstep: 448.10 | bwd_inner_microstep: 447.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:01:10,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.81 | bwd_microstep: 444.20 | bwd_inner_microstep: 444.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:01:11,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 242.75 | bwd_microstep: 371.52 | bwd_inner_microstep: 371.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:01:12,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.58 | bwd_microstep: 415.17 | bwd_inner_microstep: 415.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:01:13,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.62 | bwd_microstep: 495.37 | bwd_inner_microstep: 495.22 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:01:14,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.31 | bwd_microstep: 526.51 | bwd_inner_microstep: 526.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:01:15,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.96 | bwd_microstep: 454.57 | bwd_inner_microstep: 454.42 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:01:15,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.44 | bwd_microstep: 446.59 | bwd_inner_microstep: 446.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:01:16,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.30 | bwd_microstep: 410.38 | bwd_inner_microstep: 410.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:01:17,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.48 | bwd_microstep: 542.68 | bwd_inner_microstep: 542.54 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:01:18,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.28 | bwd_microstep: 354.29 | bwd_inner_microstep: 354.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.11
[2024-12-05 15:01:19,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 343.32 | bwd_microstep: 705.22 | bwd_inner_microstep: 705.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:01:20,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.28 | bwd_microstep: 408.48 | bwd_inner_microstep: 408.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:01:21,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.75 | bwd_microstep: 411.27 | bwd_inner_microstep: 411.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:01:21,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.21 | bwd_microstep: 357.87 | bwd_inner_microstep: 357.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:01:24,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.71 | optimizer_gradients: 14.07 | optimizer_step: 23.14
[2024-12-05 15:01:24,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.62 | bwd_microstep: 1898.42 | bwd_inner_microstep: 1886.41 | bwd_allreduce_microstep: 11.91 | step_microstep: 112.54
[2024-12-05 15:01:24,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4283.96 | bwd: 8690.61 | bwd_inner: 8676.95 | bwd_allreduce: 12.58 | step: 113.73

 70%|██████▉   | 268/384 [1:18:20<31:39, 16.37s/it]
                                                   
{'loss': 0.1119, 'grad_norm': 1.659328579902649, 'learning_rate': 2.2133776843878183e-07, 'epoch': 1.39}

 70%|██████▉   | 268/384 [1:18:20<31:39, 16.37s/it][2024-12-05 15:01:25,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.15 | bwd_microstep: 419.95 | bwd_inner_microstep: 419.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:01:26,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.61 | bwd_microstep: 429.68 | bwd_inner_microstep: 429.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:01:27,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.96 | bwd_microstep: 490.63 | bwd_inner_microstep: 490.50 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:01:27,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.21 | bwd_microstep: 404.38 | bwd_inner_microstep: 404.23 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:01:28,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.55 | bwd_microstep: 454.04 | bwd_inner_microstep: 453.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:01:29,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.18 | bwd_microstep: 482.24 | bwd_inner_microstep: 482.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:01:30,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.49 | bwd_microstep: 355.22 | bwd_inner_microstep: 355.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:01:31,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.86 | bwd_microstep: 440.30 | bwd_inner_microstep: 440.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:01:31,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.52 | bwd_microstep: 427.72 | bwd_inner_microstep: 427.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:01:33,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.50 | bwd_microstep: 615.63 | bwd_inner_microstep: 615.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:01:34,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.29 | bwd_microstep: 672.83 | bwd_inner_microstep: 672.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:01:35,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.43 | bwd_microstep: 406.33 | bwd_inner_microstep: 406.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:01:36,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.58 | bwd_microstep: 682.38 | bwd_inner_microstep: 682.25 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:01:37,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 406.48 | bwd_microstep: 911.53 | bwd_inner_microstep: 911.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:01:38,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.22 | bwd_microstep: 405.41 | bwd_inner_microstep: 405.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:01:40,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.66 | optimizer_gradients: 14.06 | optimizer_step: 23.17
[2024-12-05 15:01:40,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 355.07 | bwd_microstep: 1005.13 | bwd_inner_microstep: 993.15 | bwd_allreduce_microstep: 11.87 | step_microstep: 113.56
[2024-12-05 15:01:40,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4625.18 | bwd: 8603.40 | bwd_inner: 8589.81 | bwd_allreduce: 12.51 | step: 114.58

 70%|███████   | 269/384 [1:18:36<31:01, 16.19s/it]
                                                   
{'loss': 0.0851, 'grad_norm': 1.3867595195770264, 'learning_rate': 2.1784176868432375e-07, 'epoch': 1.4}

 70%|███████   | 269/384 [1:18:36<31:01, 16.19s/it][2024-12-05 15:01:41,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 342.21 | bwd_microstep: 652.47 | bwd_inner_microstep: 652.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:01:42,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.11 | bwd_microstep: 421.43 | bwd_inner_microstep: 421.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:01:43,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.40 | bwd_microstep: 458.99 | bwd_inner_microstep: 458.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:01:44,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.38 | bwd_microstep: 489.96 | bwd_inner_microstep: 489.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:01:45,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.78 | bwd_microstep: 442.65 | bwd_inner_microstep: 442.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:01:46,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.66 | bwd_microstep: 603.68 | bwd_inner_microstep: 603.52 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07
[2024-12-05 15:01:46,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.47 | bwd_microstep: 460.15 | bwd_inner_microstep: 460.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:01:47,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.65 | bwd_microstep: 439.56 | bwd_inner_microstep: 439.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:01:48,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.79 | bwd_microstep: 433.19 | bwd_inner_microstep: 433.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:01:49,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.41 | bwd_microstep: 455.02 | bwd_inner_microstep: 454.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:01:50,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.79 | bwd_microstep: 407.04 | bwd_inner_microstep: 406.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:01:51,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.11 | bwd_microstep: 680.02 | bwd_inner_microstep: 679.87 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 15:01:52,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.34 | bwd_microstep: 482.17 | bwd_inner_microstep: 482.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:01:53,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.30 | bwd_microstep: 452.54 | bwd_inner_microstep: 452.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:01:54,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.31 | bwd_microstep: 461.36 | bwd_inner_microstep: 461.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:01:57,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.58 | optimizer_gradients: 14.04 | optimizer_step: 23.12
[2024-12-05 15:01:57,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.88 | bwd_microstep: 2674.34 | bwd_inner_microstep: 2662.44 | bwd_allreduce_microstep: 11.79 | step_microstep: 112.04
[2024-12-05 15:01:57,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4467.66 | bwd: 10014.52 | bwd_inner: 10000.93 | bwd_allreduce: 12.44 | step: 113.04

 70%|███████   | 270/384 [1:18:53<31:22, 16.52s/it]
                                                   
{'loss': 0.1158, 'grad_norm': 1.7647099494934082, 'learning_rate': 2.1436589245260372e-07, 'epoch': 1.4}

 70%|███████   | 270/384 [1:18:53<31:22, 16.52s/it][2024-12-05 15:01:58,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.41 | bwd_microstep: 396.10 | bwd_inner_microstep: 395.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:01:58,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.86 | bwd_microstep: 401.32 | bwd_inner_microstep: 401.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:01:59,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.05 | bwd_microstep: 413.56 | bwd_inner_microstep: 413.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:02:00,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.51 | bwd_microstep: 451.79 | bwd_inner_microstep: 451.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:01,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.47 | bwd_microstep: 432.79 | bwd_inner_microstep: 432.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:02,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.95 | bwd_microstep: 433.56 | bwd_inner_microstep: 433.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:02:03,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.01 | bwd_microstep: 446.36 | bwd_inner_microstep: 446.21 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:02:04,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.77 | bwd_microstep: 412.77 | bwd_inner_microstep: 412.64 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:02:05,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.51 | bwd_microstep: 404.63 | bwd_inner_microstep: 404.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:02:06,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.68 | bwd_microstep: 616.04 | bwd_inner_microstep: 615.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:02:07,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.46 | bwd_microstep: 484.73 | bwd_inner_microstep: 484.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:02:08,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.56 | bwd_microstep: 613.19 | bwd_inner_microstep: 613.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:02:09,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.05 | bwd_microstep: 485.14 | bwd_inner_microstep: 485.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:10,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.45 | bwd_microstep: 530.66 | bwd_inner_microstep: 530.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:02:11,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.58 | bwd_microstep: 455.21 | bwd_inner_microstep: 455.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:13,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.20 | optimizer_gradients: 14.02 | optimizer_step: 23.25
[2024-12-05 15:02:13,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.29 | bwd_microstep: 2044.32 | bwd_inner_microstep: 2032.06 | bwd_allreduce_microstep: 12.15 | step_microstep: 111.97
[2024-12-05 15:02:13,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4423.68 | bwd: 9022.14 | bwd_inner: 9008.26 | bwd_allreduce: 12.79 | step: 112.99

 71%|███████   | 271/384 [1:19:09<30:53, 16.40s/it]
                                                   
{'loss': 0.1527, 'grad_norm': 1.8926544189453125, 'learning_rate': 2.109103876430864e-07, 'epoch': 1.41}

 71%|███████   | 271/384 [1:19:09<30:53, 16.40s/it][2024-12-05 15:02:14,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.80 | bwd_microstep: 404.08 | bwd_inner_microstep: 403.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:15,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.74 | bwd_microstep: 560.94 | bwd_inner_microstep: 560.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:02:16,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.28 | bwd_microstep: 453.12 | bwd_inner_microstep: 452.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:17,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.11 | bwd_microstep: 414.03 | bwd_inner_microstep: 413.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:02:17,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.97 | bwd_microstep: 356.52 | bwd_inner_microstep: 356.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:18,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.16 | bwd_microstep: 445.32 | bwd_inner_microstep: 445.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:02:19,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.90 | bwd_microstep: 489.64 | bwd_inner_microstep: 489.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:20,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.20 | bwd_microstep: 593.43 | bwd_inner_microstep: 593.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:02:21,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.93 | bwd_microstep: 461.49 | bwd_inner_microstep: 461.30 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:02:22,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.14 | bwd_microstep: 454.14 | bwd_inner_microstep: 454.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:02:23,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.63 | bwd_microstep: 457.71 | bwd_inner_microstep: 457.53 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 15:02:24,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.46 | bwd_microstep: 413.23 | bwd_inner_microstep: 413.05 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 15:02:25,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.90 | bwd_microstep: 412.34 | bwd_inner_microstep: 412.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:02:26,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.50 | bwd_microstep: 455.01 | bwd_inner_microstep: 454.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:02:27,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.81 | bwd_microstep: 449.13 | bwd_inner_microstep: 449.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:29,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.43 | optimizer_gradients: 14.06 | optimizer_step: 23.08
[2024-12-05 15:02:29,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.44 | bwd_microstep: 2185.62 | bwd_inner_microstep: 2173.46 | bwd_allreduce_microstep: 12.07 | step_microstep: 112.33
[2024-12-05 15:02:29,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4320.03 | bwd: 9005.74 | bwd_inner: 8991.83 | bwd_allreduce: 12.76 | step: 113.46

 71%|███████   | 272/384 [1:19:26<30:34, 16.38s/it]
                                                   
{'loss': 0.1108, 'grad_norm': 1.839390516281128, 'learning_rate': 2.074755007023461e-07, 'epoch': 1.41}

 71%|███████   | 272/384 [1:19:26<30:34, 16.38s/it][2024-12-05 15:02:30,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.72 | bwd_microstep: 426.56 | bwd_inner_microstep: 426.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:02:31,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.57 | bwd_microstep: 455.32 | bwd_inner_microstep: 455.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:02:32,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.26 | bwd_microstep: 518.10 | bwd_inner_microstep: 517.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:33,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.90 | bwd_microstep: 448.72 | bwd_inner_microstep: 448.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:34,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.04 | bwd_microstep: 489.04 | bwd_inner_microstep: 488.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:35,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.06 | bwd_microstep: 458.14 | bwd_inner_microstep: 458.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:36,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.12 | bwd_microstep: 456.83 | bwd_inner_microstep: 456.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:37,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.45 | bwd_microstep: 428.07 | bwd_inner_microstep: 427.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:37,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.95 | bwd_microstep: 450.60 | bwd_inner_microstep: 450.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:38,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.80 | bwd_microstep: 488.28 | bwd_inner_microstep: 488.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:02:39,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.90 | bwd_microstep: 593.79 | bwd_inner_microstep: 593.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:40,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.61 | bwd_microstep: 449.83 | bwd_inner_microstep: 449.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:02:41,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.91 | bwd_microstep: 481.99 | bwd_inner_microstep: 481.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:02:42,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.14 | bwd_microstep: 536.10 | bwd_inner_microstep: 535.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:02:43,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.45 | bwd_microstep: 626.82 | bwd_inner_microstep: 626.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:02:46,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.04 | optimizer_gradients: 14.03 | optimizer_step: 23.10
[2024-12-05 15:02:46,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 352.94 | bwd_microstep: 1659.46 | bwd_inner_microstep: 1647.38 | bwd_allreduce_microstep: 12.00 | step_microstep: 111.78
[2024-12-05 15:02:46,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4487.87 | bwd: 8967.65 | bwd_inner: 8953.93 | bwd_allreduce: 12.61 | step: 112.90

 71%|███████   | 273/384 [1:19:42<30:10, 16.31s/it]
                                                   
{'loss': 0.1508, 'grad_norm': 1.797500729560852, 'learning_rate': 2.040614766064913e-07, 'epoch': 1.42}

 71%|███████   | 273/384 [1:19:42<30:10, 16.31s/it][2024-12-05 15:02:47,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 378.54 | bwd_microstep: 815.86 | bwd_inner_microstep: 815.67 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10
[2024-12-05 15:02:48,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.45 | bwd_microstep: 615.36 | bwd_inner_microstep: 615.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:02:49,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.95 | bwd_microstep: 618.52 | bwd_inner_microstep: 618.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:50,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.23 | bwd_microstep: 356.87 | bwd_inner_microstep: 356.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:02:51,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.87 | bwd_microstep: 524.40 | bwd_inner_microstep: 524.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:52,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.93 | bwd_microstep: 453.51 | bwd_inner_microstep: 453.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:02:53,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.72 | bwd_microstep: 432.57 | bwd_inner_microstep: 432.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:02:54,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.69 | bwd_microstep: 619.18 | bwd_inner_microstep: 619.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:02:55,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.91 | bwd_microstep: 567.78 | bwd_inner_microstep: 567.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:56,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.29 | bwd_microstep: 407.81 | bwd_inner_microstep: 407.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:02:56,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.76 | bwd_microstep: 408.16 | bwd_inner_microstep: 408.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:02:57,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.78 | bwd_microstep: 427.40 | bwd_inner_microstep: 427.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:58,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.47 | bwd_microstep: 449.72 | bwd_inner_microstep: 449.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:02:59,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.39 | bwd_microstep: 485.97 | bwd_inner_microstep: 485.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:03:00,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.84 | bwd_microstep: 477.29 | bwd_inner_microstep: 477.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:03:02,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.44 | optimizer_gradients: 14.00 | optimizer_step: 23.37
[2024-12-05 15:03:02,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.69 | bwd_microstep: 1245.62 | bwd_inner_microstep: 1233.67 | bwd_allreduce_microstep: 11.86 | step_microstep: 112.35
[2024-12-05 15:03:02,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4484.56 | bwd: 8906.01 | bwd_inner: 8892.40 | bwd_allreduce: 12.50 | step: 113.51

 71%|███████▏  | 274/384 [1:19:58<29:50, 16.28s/it]
                                                   
{'loss': 0.1242, 'grad_norm': 1.7864497900009155, 'learning_rate': 2.0066855884369243e-07, 'epoch': 1.42}

 71%|███████▏  | 274/384 [1:19:58<29:50, 16.28s/it][2024-12-05 15:03:03,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.90 | bwd_microstep: 396.83 | bwd_inner_microstep: 396.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:03:04,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.43 | bwd_microstep: 547.69 | bwd_inner_microstep: 547.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:03:05,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.96 | bwd_microstep: 552.36 | bwd_inner_microstep: 552.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:03:05,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.17 | bwd_microstep: 417.90 | bwd_inner_microstep: 417.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:03:07,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 354.39 | bwd_microstep: 759.59 | bwd_inner_microstep: 759.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:03:08,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.39 | bwd_microstep: 567.41 | bwd_inner_microstep: 567.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:03:09,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.17 | bwd_microstep: 427.05 | bwd_inner_microstep: 426.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:03:10,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.71 | bwd_microstep: 488.08 | bwd_inner_microstep: 487.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:03:11,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 378.69 | bwd_microstep: 833.38 | bwd_inner_microstep: 833.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:03:12,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.96 | bwd_microstep: 459.18 | bwd_inner_microstep: 459.05 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:03:13,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.64 | bwd_microstep: 429.71 | bwd_inner_microstep: 429.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:03:14,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.89 | bwd_microstep: 479.72 | bwd_inner_microstep: 479.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:03:15,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.89 | bwd_microstep: 410.63 | bwd_inner_microstep: 410.49 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:03:15,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.24 | bwd_microstep: 452.93 | bwd_inner_microstep: 452.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:03:16,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.15 | bwd_microstep: 552.78 | bwd_inner_microstep: 552.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:03:18,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.26 | optimizer_gradients: 14.02 | optimizer_step: 23.30
[2024-12-05 15:03:18,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.76 | bwd_microstep: 1173.09 | bwd_inner_microstep: 1161.01 | bwd_allreduce_microstep: 11.99 | step_microstep: 111.85
[2024-12-05 15:03:18,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4560.41 | bwd: 8948.33 | bwd_inner: 8934.63 | bwd_allreduce: 12.63 | step: 112.93

 72%|███████▏  | 275/384 [1:20:14<29:37, 16.31s/it]
                                                   
{'loss': 0.122, 'grad_norm': 1.853615641593933, 'learning_rate': 1.9729698939681644e-07, 'epoch': 1.43}

 72%|███████▏  | 275/384 [1:20:14<29:37, 16.31s/it][2024-12-05 15:03:19,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.36 | bwd_microstep: 410.98 | bwd_inner_microstep: 410.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:03:20,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.29 | bwd_microstep: 481.07 | bwd_inner_microstep: 480.92 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:03:22,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.25 | bwd_microstep: 436.98 | bwd_inner_microstep: 436.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:03:23,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.58 | bwd_microstep: 580.54 | bwd_inner_microstep: 580.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:03:24,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.38 | bwd_microstep: 426.50 | bwd_inner_microstep: 426.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:03:25,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.26 | bwd_microstep: 424.07 | bwd_inner_microstep: 423.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:03:26,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.33 | bwd_microstep: 448.79 | bwd_inner_microstep: 448.66 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:03:27,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.50 | bwd_microstep: 457.49 | bwd_inner_microstep: 457.30 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 15:03:28,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.64 | bwd_microstep: 676.54 | bwd_inner_microstep: 676.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:03:29,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.55 | bwd_microstep: 437.22 | bwd_inner_microstep: 437.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:03:30,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.35 | bwd_microstep: 439.92 | bwd_inner_microstep: 439.78 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:03:30,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.58 | bwd_microstep: 427.09 | bwd_inner_microstep: 426.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:03:32,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 341.66 | bwd_microstep: 693.51 | bwd_inner_microstep: 693.37 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:03:32,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.90 | bwd_microstep: 410.54 | bwd_inner_microstep: 410.40 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:03:33,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.00 | bwd_microstep: 430.87 | bwd_inner_microstep: 430.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:03:35,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.21 | optimizer_gradients: 14.02 | optimizer_step: 23.16
[2024-12-05 15:03:35,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.74 | bwd_microstep: 1566.60 | bwd_inner_microstep: 1554.45 | bwd_allreduce_microstep: 12.04 | step_microstep: 111.75
[2024-12-05 15:03:35,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4481.44 | bwd: 8748.72 | bwd_inner: 8734.83 | bwd_allreduce: 12.70 | step: 112.86

 72%|███████▏  | 276/384 [1:20:32<29:52, 16.60s/it]
                                                   
{'loss': 0.1212, 'grad_norm': 1.6054631471633911, 'learning_rate': 1.9394700872616853e-07, 'epoch': 1.43}

 72%|███████▏  | 276/384 [1:20:32<29:52, 16.60s/it][2024-12-05 15:03:36,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.24 | bwd_microstep: 362.23 | bwd_inner_microstep: 362.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:03:37,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.93 | bwd_microstep: 428.12 | bwd_inner_microstep: 428.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:03:38,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 359.22 | bwd_microstep: 752.58 | bwd_inner_microstep: 752.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:03:39,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.05 | bwd_microstep: 432.25 | bwd_inner_microstep: 432.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:03:40,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.57 | bwd_microstep: 430.33 | bwd_inner_microstep: 430.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:03:41,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.16 | bwd_microstep: 345.29 | bwd_inner_microstep: 345.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:03:42,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.66 | bwd_microstep: 390.79 | bwd_inner_microstep: 390.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:03:43,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.13 | bwd_microstep: 454.55 | bwd_inner_microstep: 454.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:03:44,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.92 | bwd_microstep: 688.24 | bwd_inner_microstep: 688.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:03:45,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.46 | bwd_microstep: 411.46 | bwd_inner_microstep: 411.29 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 15:03:46,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.12 | bwd_microstep: 582.80 | bwd_inner_microstep: 582.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:03:47,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.36 | bwd_microstep: 413.66 | bwd_inner_microstep: 413.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:03:48,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.01 | bwd_microstep: 554.62 | bwd_inner_microstep: 554.45 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:03:49,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.45 | bwd_microstep: 452.85 | bwd_inner_microstep: 452.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:03:50,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.96 | bwd_microstep: 482.14 | bwd_inner_microstep: 482.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:03:52,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.43 | optimizer_gradients: 14.05 | optimizer_step: 23.13
[2024-12-05 15:03:52,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.70 | bwd_microstep: 1493.53 | bwd_inner_microstep: 1481.54 | bwd_allreduce_microstep: 11.88 | step_microstep: 112.42
[2024-12-05 15:03:52,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4505.02 | bwd: 8675.45 | bwd_inner: 8661.74 | bwd_allreduce: 12.54 | step: 113.45

 72%|███████▏  | 277/384 [1:20:48<29:32, 16.57s/it]
                                                   
{'loss': 0.1101, 'grad_norm': 1.632055640220642, 'learning_rate': 1.906188557523427e-07, 'epoch': 1.44}

 72%|███████▏  | 277/384 [1:20:48<29:32, 16.57s/it][2024-12-05 15:03:53,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.33 | bwd_microstep: 371.17 | bwd_inner_microstep: 371.03 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:03:54,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.27 | bwd_microstep: 586.05 | bwd_inner_microstep: 585.86 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 15:03:55,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.33 | bwd_microstep: 445.86 | bwd_inner_microstep: 445.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:03:56,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.81 | bwd_microstep: 627.00 | bwd_inner_microstep: 626.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:03:57,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.27 | bwd_microstep: 476.94 | bwd_inner_microstep: 476.80 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:03:58,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.30 | bwd_microstep: 416.12 | bwd_inner_microstep: 415.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:03:59,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.92 | bwd_microstep: 519.57 | bwd_inner_microstep: 519.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:04:00,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.94 | bwd_microstep: 589.44 | bwd_inner_microstep: 589.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:04:01,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.16 | bwd_microstep: 535.10 | bwd_inner_microstep: 534.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:04:02,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.36 | bwd_microstep: 397.94 | bwd_inner_microstep: 397.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:04:03,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.03 | bwd_microstep: 416.06 | bwd_inner_microstep: 415.81 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.15
[2024-12-05 15:04:04,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.30 | bwd_microstep: 562.17 | bwd_inner_microstep: 562.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:04:05,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.67 | bwd_microstep: 620.74 | bwd_inner_microstep: 620.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:04:06,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.56 | bwd_microstep: 596.35 | bwd_inner_microstep: 596.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:04:07,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.41 | bwd_microstep: 399.88 | bwd_inner_microstep: 399.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:04:08,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.20 | optimizer_gradients: 14.07 | optimizer_step: 23.15
[2024-12-05 15:04:08,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.34 | bwd_microstep: 899.12 | bwd_inner_microstep: 886.96 | bwd_allreduce_microstep: 12.06 | step_microstep: 112.06
[2024-12-05 15:04:08,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4521.08 | bwd: 8459.52 | bwd_inner: 8445.57 | bwd_allreduce: 12.75 | step: 113.26

 72%|███████▏  | 278/384 [1:21:05<29:14, 16.56s/it]
                                                   
{'loss': 0.1217, 'grad_norm': 1.6560664176940918, 'learning_rate': 1.873127678391816e-07, 'epoch': 1.45}

 72%|███████▏  | 278/384 [1:21:05<29:14, 16.56s/it][2024-12-05 15:04:10,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.22 | bwd_microstep: 658.87 | bwd_inner_microstep: 658.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:04:10,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.30 | bwd_microstep: 428.06 | bwd_inner_microstep: 427.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:11,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.44 | bwd_microstep: 401.70 | bwd_inner_microstep: 401.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:12,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.05 | bwd_microstep: 353.38 | bwd_inner_microstep: 353.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:13,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.66 | bwd_microstep: 615.93 | bwd_inner_microstep: 615.76 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 15:04:14,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.05 | bwd_microstep: 428.55 | bwd_inner_microstep: 428.37 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 15:04:15,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.04 | bwd_microstep: 410.64 | bwd_inner_microstep: 410.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:04:16,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.02 | bwd_microstep: 486.87 | bwd_inner_microstep: 486.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:17,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.76 | bwd_microstep: 593.87 | bwd_inner_microstep: 593.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:18,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.46 | bwd_microstep: 448.51 | bwd_inner_microstep: 448.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:19,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.37 | bwd_microstep: 355.44 | bwd_inner_microstep: 355.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:19,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.25 | bwd_microstep: 407.45 | bwd_inner_microstep: 407.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:20,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.61 | bwd_microstep: 529.45 | bwd_inner_microstep: 529.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:04:21,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.88 | bwd_microstep: 433.74 | bwd_inner_microstep: 433.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:22,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.49 | bwd_microstep: 427.18 | bwd_inner_microstep: 427.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:04:24,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.47 | optimizer_gradients: 14.12 | optimizer_step: 23.26
[2024-12-05 15:04:24,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.12 | bwd_microstep: 1641.08 | bwd_inner_microstep: 1628.96 | bwd_allreduce_microstep: 12.03 | step_microstep: 114.00
[2024-12-05 15:04:24,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4407.77 | bwd: 8620.73 | bwd_inner: 8606.89 | bwd_allreduce: 12.68 | step: 115.05

 73%|███████▎  | 279/384 [1:21:21<28:39, 16.38s/it]
                                                   
{'loss': 0.0988, 'grad_norm': 1.4159197807312012, 'learning_rate': 1.8402898077684803e-07, 'epoch': 1.45}

 73%|███████▎  | 279/384 [1:21:21<28:39, 16.38s/it][2024-12-05 15:04:25,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.35 | bwd_microstep: 373.30 | bwd_inner_microstep: 373.12 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 15:04:26,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.52 | bwd_microstep: 412.26 | bwd_inner_microstep: 412.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:04:27,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.68 | bwd_microstep: 482.60 | bwd_inner_microstep: 482.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:28,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.96 | bwd_microstep: 357.22 | bwd_inner_microstep: 357.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:28,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.18 | bwd_microstep: 413.93 | bwd_inner_microstep: 413.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:29,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.02 | bwd_microstep: 583.38 | bwd_inner_microstep: 583.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:30,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.32 | bwd_microstep: 435.10 | bwd_inner_microstep: 434.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:04:31,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.69 | bwd_microstep: 584.88 | bwd_inner_microstep: 584.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 15:04:32,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.84 | bwd_microstep: 438.28 | bwd_inner_microstep: 438.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:33,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.66 | bwd_microstep: 687.60 | bwd_inner_microstep: 687.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:04:34,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.67 | bwd_microstep: 445.92 | bwd_inner_microstep: 445.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:04:35,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.49 | bwd_microstep: 519.42 | bwd_inner_microstep: 519.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:36,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.56 | bwd_microstep: 408.36 | bwd_inner_microstep: 408.19 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:04:37,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.69 | bwd_microstep: 396.99 | bwd_inner_microstep: 396.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:38,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.64 | bwd_microstep: 443.74 | bwd_inner_microstep: 443.60 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:04:41,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.94 | optimizer_gradients: 14.05 | optimizer_step: 23.34
[2024-12-05 15:04:41,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.41 | bwd_microstep: 2830.04 | bwd_inner_microstep: 2818.09 | bwd_allreduce_microstep: 11.87 | step_microstep: 115.02
[2024-12-05 15:04:41,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4335.73 | bwd: 9813.05 | bwd_inner: 9799.37 | bwd_allreduce: 12.52 | step: 116.05

 73%|███████▎  | 280/384 [1:21:37<28:29, 16.44s/it]
                                                   
{'loss': 0.1293, 'grad_norm': 1.6314151287078857, 'learning_rate': 1.8076772876500828e-07, 'epoch': 1.46}

 73%|███████▎  | 280/384 [1:21:37<28:29, 16.44s/it][2024-12-05 15:04:42,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.57 | bwd_microstep: 407.99 | bwd_inner_microstep: 407.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:43,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.79 | bwd_microstep: 411.08 | bwd_inner_microstep: 410.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:04:43,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.56 | bwd_microstep: 421.80 | bwd_inner_microstep: 421.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:44,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.49 | bwd_microstep: 593.97 | bwd_inner_microstep: 593.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:46,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.70 | bwd_microstep: 580.58 | bwd_inner_microstep: 580.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:47,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.36 | bwd_microstep: 569.86 | bwd_inner_microstep: 569.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:04:48,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.56 | bwd_microstep: 617.70 | bwd_inner_microstep: 617.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:48,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.18 | bwd_microstep: 461.14 | bwd_inner_microstep: 461.00 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.06
[2024-12-05 15:04:49,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.48 | bwd_microstep: 528.36 | bwd_inner_microstep: 528.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:50,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.90 | bwd_microstep: 405.16 | bwd_inner_microstep: 405.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:04:51,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.66 | bwd_microstep: 453.15 | bwd_inner_microstep: 453.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:04:52,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.32 | bwd_microstep: 449.57 | bwd_inner_microstep: 449.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:53,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.15 | bwd_microstep: 482.28 | bwd_inner_microstep: 482.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:54,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.78 | bwd_microstep: 403.21 | bwd_inner_microstep: 403.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:04:55,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.67 | bwd_microstep: 680.78 | bwd_inner_microstep: 680.63 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:04:57,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.55 | optimizer_gradients: 14.06 | optimizer_step: 23.12
[2024-12-05 15:04:57,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.03 | bwd_microstep: 2233.00 | bwd_inner_microstep: 2221.06 | bwd_allreduce_microstep: 11.85 | step_microstep: 112.19
[2024-12-05 15:04:57,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4485.30 | bwd: 9699.62 | bwd_inner: 9686.06 | bwd_allreduce: 12.49 | step: 113.18

 73%|███████▎  | 281/384 [1:21:54<28:15, 16.46s/it]
                                                   
{'loss': 0.1401, 'grad_norm': 1.7430942058563232, 'learning_rate': 1.775292443961291e-07, 'epoch': 1.46}

 73%|███████▎  | 281/384 [1:21:54<28:15, 16.46s/it][2024-12-05 15:04:59,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 378.67 | bwd_microstep: 788.79 | bwd_inner_microstep: 788.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:05:00,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.90 | bwd_microstep: 510.35 | bwd_inner_microstep: 510.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.05
[2024-12-05 15:05:01,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.49 | bwd_microstep: 587.51 | bwd_inner_microstep: 587.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:05:02,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.10 | bwd_microstep: 452.79 | bwd_inner_microstep: 452.67 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:05:03,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 343.16 | bwd_microstep: 700.47 | bwd_inner_microstep: 700.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:05:04,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.51 | bwd_microstep: 431.02 | bwd_inner_microstep: 430.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:05:05,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.83 | bwd_microstep: 442.95 | bwd_inner_microstep: 442.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:05:06,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 332.05 | bwd_microstep: 670.85 | bwd_inner_microstep: 670.71 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:05:07,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.04 | bwd_microstep: 489.84 | bwd_inner_microstep: 489.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:05:07,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.42 | bwd_microstep: 441.41 | bwd_inner_microstep: 441.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:05:08,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.01 | bwd_microstep: 450.79 | bwd_inner_microstep: 450.60 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 15:05:09,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.43 | bwd_microstep: 450.50 | bwd_inner_microstep: 450.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:05:10,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.49 | bwd_microstep: 529.28 | bwd_inner_microstep: 529.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:05:11,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.49 | bwd_microstep: 523.88 | bwd_inner_microstep: 523.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:05:12,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.52 | bwd_microstep: 460.85 | bwd_inner_microstep: 460.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:05:13,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.38 | optimizer_gradients: 14.03 | optimizer_step: 23.33
[2024-12-05 15:05:13,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.92 | bwd_microstep: 903.71 | bwd_inner_microstep: 891.55 | bwd_allreduce_microstep: 12.07 | step_microstep: 112.27
[2024-12-05 15:05:13,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4638.08 | bwd: 8834.99 | bwd_inner: 8821.14 | bwd_allreduce: 12.72 | step: 113.31

 73%|███████▎  | 282/384 [1:22:09<27:37, 16.25s/it]
                                                   
{'loss': 0.1275, 'grad_norm': 1.7857812643051147, 'learning_rate': 1.7431375863888898e-07, 'epoch': 1.47}

 73%|███████▎  | 282/384 [1:22:09<27:37, 16.25s/it][2024-12-05 15:05:14,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.86 | bwd_microstep: 402.46 | bwd_inner_microstep: 402.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:05:15,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.30 | bwd_microstep: 400.32 | bwd_inner_microstep: 400.18 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:05:16,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.04 | bwd_microstep: 459.75 | bwd_inner_microstep: 459.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:05:16,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 228.18 | bwd_microstep: 330.71 | bwd_inner_microstep: 330.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:05:18,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.38 | bwd_microstep: 684.38 | bwd_inner_microstep: 684.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:05:19,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.87 | bwd_microstep: 619.77 | bwd_inner_microstep: 619.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:05:20,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.70 | bwd_microstep: 426.70 | bwd_inner_microstep: 426.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:05:20,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.81 | bwd_microstep: 478.86 | bwd_inner_microstep: 478.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:05:21,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.50 | bwd_microstep: 439.99 | bwd_inner_microstep: 439.84 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 15:05:22,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.03 | bwd_microstep: 478.25 | bwd_inner_microstep: 478.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:05:23,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.38 | bwd_microstep: 521.94 | bwd_inner_microstep: 521.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:05:24,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.34 | bwd_microstep: 487.66 | bwd_inner_microstep: 487.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:05:25,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.84 | bwd_microstep: 479.59 | bwd_inner_microstep: 479.46 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:05:26,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.63 | bwd_microstep: 407.45 | bwd_inner_microstep: 407.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:05:27,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.72 | bwd_microstep: 491.95 | bwd_inner_microstep: 491.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:05:29,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.63 | optimizer_gradients: 14.02 | optimizer_step: 23.08
[2024-12-05 15:05:29,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.06 | bwd_microstep: 1920.63 | bwd_inner_microstep: 1908.39 | bwd_allreduce_microstep: 12.13 | step_microstep: 112.16
[2024-12-05 15:05:29,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4384.71 | bwd: 9030.43 | bwd_inner: 9016.56 | bwd_allreduce: 12.78 | step: 113.35

 74%|███████▎  | 283/384 [1:22:25<27:10, 16.14s/it]
                                                   
{'loss': 0.1061, 'grad_norm': 1.5308443307876587, 'learning_rate': 1.7112150082170568e-07, 'epoch': 1.47}

 74%|███████▎  | 283/384 [1:22:25<27:10, 16.14s/it][2024-12-05 15:05:30,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.68 | bwd_microstep: 557.73 | bwd_inner_microstep: 557.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:05:31,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.98 | bwd_microstep: 480.19 | bwd_inner_microstep: 480.05 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:05:32,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.20 | bwd_microstep: 435.01 | bwd_inner_microstep: 434.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:05:33,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.95 | bwd_microstep: 568.53 | bwd_inner_microstep: 568.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:05:34,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.76 | bwd_microstep: 489.04 | bwd_inner_microstep: 488.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 15:05:35,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.35 | bwd_microstep: 436.36 | bwd_inner_microstep: 436.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:05:35,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.25 | bwd_microstep: 447.83 | bwd_inner_microstep: 447.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:05:36,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.29 | bwd_microstep: 452.24 | bwd_inner_microstep: 452.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:05:37,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.43 | bwd_microstep: 404.68 | bwd_inner_microstep: 404.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:05:38,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.93 | bwd_microstep: 401.77 | bwd_inner_microstep: 401.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:05:39,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.00 | bwd_microstep: 428.51 | bwd_inner_microstep: 428.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:05:39,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.61 | bwd_microstep: 484.85 | bwd_inner_microstep: 484.73 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:05:40,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.73 | bwd_microstep: 558.73 | bwd_inner_microstep: 558.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:05:41,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.07 | bwd_microstep: 429.58 | bwd_inner_microstep: 429.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:05:42,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.54 | bwd_microstep: 401.80 | bwd_inner_microstep: 401.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:05:45,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.58 | optimizer_gradients: 14.02 | optimizer_step: 23.09
[2024-12-05 15:05:45,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 332.91 | bwd_microstep: 2228.68 | bwd_inner_microstep: 2216.55 | bwd_allreduce_microstep: 12.03 | step_microstep: 111.96
[2024-12-05 15:05:45,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4379.72 | bwd: 9205.51 | bwd_inner: 9191.78 | bwd_allreduce: 12.66 | step: 113.16

 74%|███████▍  | 284/384 [1:22:41<26:43, 16.03s/it]
                                                   
{'loss': 0.1119, 'grad_norm': 1.6288737058639526, 'learning_rate': 1.679526986163804e-07, 'epoch': 1.48}

 74%|███████▍  | 284/384 [1:22:41<26:43, 16.03s/it][2024-12-05 15:05:46,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.25 | bwd_microstep: 436.31 | bwd_inner_microstep: 436.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:05:47,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.69 | bwd_microstep: 407.69 | bwd_inner_microstep: 407.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:05:48,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.75 | bwd_microstep: 521.68 | bwd_inner_microstep: 521.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:05:48,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.05 | bwd_microstep: 409.90 | bwd_inner_microstep: 409.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:05:50,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 350.44 | bwd_microstep: 720.60 | bwd_inner_microstep: 720.45 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:05:51,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 375.23 | bwd_microstep: 780.52 | bwd_inner_microstep: 780.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:05:52,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.25 | bwd_microstep: 487.82 | bwd_inner_microstep: 487.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:05:53,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 352.32 | bwd_microstep: 710.87 | bwd_inner_microstep: 710.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:05:54,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 366.28 | bwd_microstep: 791.22 | bwd_inner_microstep: 791.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:05:55,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.29 | bwd_microstep: 567.97 | bwd_inner_microstep: 567.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:05:56,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.75 | bwd_microstep: 434.50 | bwd_inner_microstep: 434.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:05:57,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.15 | bwd_microstep: 481.31 | bwd_inner_microstep: 481.17 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:05:58,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.61 | bwd_microstep: 619.61 | bwd_inner_microstep: 619.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:05:59,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.51 | bwd_microstep: 400.76 | bwd_inner_microstep: 400.62 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:06:00,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.12 | bwd_microstep: 414.61 | bwd_inner_microstep: 414.48 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:06:02,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.23 | optimizer_gradients: 14.05 | optimizer_step: 23.23
[2024-12-05 15:06:02,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 402.21 | bwd_microstep: 1532.19 | bwd_inner_microstep: 1520.23 | bwd_allreduce_microstep: 11.87 | step_microstep: 111.87
[2024-12-05 15:06:02,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4858.95 | bwd: 9717.55 | bwd_inner: 9703.94 | bwd_allreduce: 12.52 | step: 112.95

 74%|███████▍  | 285/384 [1:22:58<26:58, 16.34s/it]
                                                   
{'loss': 0.1166, 'grad_norm': 1.704690933227539, 'learning_rate': 1.6480757802186068e-07, 'epoch': 1.48}

 74%|███████▍  | 285/384 [1:22:58<26:58, 16.34s/it][2024-12-05 15:06:03,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.41 | bwd_microstep: 368.21 | bwd_inner_microstep: 368.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:03,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.04 | bwd_microstep: 356.16 | bwd_inner_microstep: 356.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:05,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 380.53 | bwd_microstep: 841.89 | bwd_inner_microstep: 841.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:06:06,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 244.62 | bwd_microstep: 381.81 | bwd_inner_microstep: 381.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:06:06,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.89 | bwd_microstep: 448.58 | bwd_inner_microstep: 448.44 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:06:07,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.11 | bwd_microstep: 560.84 | bwd_inner_microstep: 560.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:06:08,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.95 | bwd_microstep: 538.41 | bwd_inner_microstep: 538.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:06:09,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.84 | bwd_microstep: 408.12 | bwd_inner_microstep: 407.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:10,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.87 | bwd_microstep: 487.47 | bwd_inner_microstep: 487.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:11,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.90 | bwd_microstep: 402.80 | bwd_inner_microstep: 402.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:12,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.92 | bwd_microstep: 446.72 | bwd_inner_microstep: 446.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:13,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.53 | bwd_microstep: 427.57 | bwd_inner_microstep: 427.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:06:14,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.05 | bwd_microstep: 448.91 | bwd_inner_microstep: 448.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:14,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.67 | bwd_microstep: 476.18 | bwd_inner_microstep: 476.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:16,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 359.48 | bwd_microstep: 745.30 | bwd_inner_microstep: 745.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:06:19,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.49 | optimizer_gradients: 14.02 | optimizer_step: 23.23
[2024-12-05 15:06:19,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 243.76 | bwd_microstep: 2978.16 | bwd_inner_microstep: 2966.18 | bwd_allreduce_microstep: 11.89 | step_microstep: 112.39
[2024-12-05 15:06:19,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4444.66 | bwd: 10317.13 | bwd_inner: 10303.56 | bwd_allreduce: 12.51 | step: 113.44

 74%|███████▍  | 286/384 [1:23:15<27:06, 16.59s/it]
                                                   
{'loss': 0.1045, 'grad_norm': 1.5033763647079468, 'learning_rate': 1.6168636334812125e-07, 'epoch': 1.49}

 74%|███████▍  | 286/384 [1:23:15<27:06, 16.59s/it][2024-12-05 15:06:20,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.30 | bwd_microstep: 534.69 | bwd_inner_microstep: 534.55 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.06
[2024-12-05 15:06:21,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.25 | bwd_microstep: 455.39 | bwd_inner_microstep: 455.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:22,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.11 | bwd_microstep: 540.34 | bwd_inner_microstep: 540.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:06:23,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 365.27 | bwd_microstep: 769.34 | bwd_inner_microstep: 769.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:06:24,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.97 | bwd_microstep: 448.47 | bwd_inner_microstep: 448.33 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:06:25,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.78 | bwd_microstep: 415.07 | bwd_inner_microstep: 414.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:26,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.17 | bwd_microstep: 534.47 | bwd_inner_microstep: 534.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:27,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.89 | bwd_microstep: 403.30 | bwd_inner_microstep: 403.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:27,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.57 | bwd_microstep: 482.33 | bwd_inner_microstep: 482.18 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:06:29,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.11 | bwd_microstep: 596.83 | bwd_inner_microstep: 596.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:06:29,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.65 | bwd_microstep: 478.47 | bwd_inner_microstep: 478.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:06:30,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.63 | bwd_microstep: 434.51 | bwd_inner_microstep: 434.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:31,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.50 | bwd_microstep: 445.99 | bwd_inner_microstep: 445.84 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:06:32,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.92 | bwd_microstep: 560.22 | bwd_inner_microstep: 560.08 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:06:33,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.05 | bwd_microstep: 407.42 | bwd_inner_microstep: 407.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:06:34,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.32 | optimizer_gradients: 14.07 | optimizer_step: 23.17
[2024-12-05 15:06:34,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.25 | bwd_microstep: 983.59 | bwd_inner_microstep: 971.51 | bwd_allreduce_microstep: 12.00 | step_microstep: 112.08
[2024-12-05 15:06:34,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4509.49 | bwd: 8490.46 | bwd_inner: 8476.68 | bwd_allreduce: 12.66 | step: 113.11

 75%|███████▍  | 287/384 [1:23:31<26:09, 16.18s/it]
                                                   
{'loss': 0.0959, 'grad_norm': 1.4401872158050537, 'learning_rate': 1.5858927720016706e-07, 'epoch': 1.49}

 75%|███████▍  | 287/384 [1:23:31<26:09, 16.18s/it][2024-12-05 15:06:35,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.41 | bwd_microstep: 423.64 | bwd_inner_microstep: 423.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:06:36,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.55 | bwd_microstep: 434.78 | bwd_inner_microstep: 434.63 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:06:37,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.73 | bwd_microstep: 447.49 | bwd_inner_microstep: 447.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:06:38,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.93 | bwd_microstep: 406.46 | bwd_inner_microstep: 406.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:06:39,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.59 | bwd_microstep: 446.08 | bwd_inner_microstep: 445.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:06:40,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 344.17 | bwd_microstep: 706.10 | bwd_inner_microstep: 705.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:06:41,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.08 | bwd_microstep: 478.37 | bwd_inner_microstep: 478.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:06:42,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.42 | bwd_microstep: 481.93 | bwd_inner_microstep: 481.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:06:42,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.95 | bwd_microstep: 427.18 | bwd_inner_microstep: 427.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:06:43,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.67 | bwd_microstep: 406.62 | bwd_inner_microstep: 406.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:06:44,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.43 | bwd_microstep: 409.78 | bwd_inner_microstep: 409.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:06:45,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.59 | bwd_microstep: 436.08 | bwd_inner_microstep: 435.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:06:46,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.01 | bwd_microstep: 606.95 | bwd_inner_microstep: 606.82 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:06:47,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.66 | bwd_microstep: 623.33 | bwd_inner_microstep: 623.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:06:48,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.88 | bwd_microstep: 410.48 | bwd_inner_microstep: 410.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:50,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.17 | optimizer_gradients: 14.56 | optimizer_step: 23.96
[2024-12-05 15:06:50,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.79 | bwd_microstep: 1869.99 | bwd_inner_microstep: 1857.96 | bwd_allreduce_microstep: 11.93 | step_microstep: 120.83
[2024-12-05 15:06:50,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4392.89 | bwd: 9015.25 | bwd_inner: 9001.57 | bwd_allreduce: 12.57 | step: 121.96

 75%|███████▌  | 288/384 [1:23:47<25:47, 16.12s/it]
                                                   
{'loss': 0.1124, 'grad_norm': 1.6706205606460571, 'learning_rate': 1.555165404621567e-07, 'epoch': 1.5}

 75%|███████▌  | 288/384 [1:23:47<25:47, 16.12s/it][2024-12-05 15:06:51,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.24 | bwd_microstep: 554.79 | bwd_inner_microstep: 554.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:52,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.80 | bwd_microstep: 431.36 | bwd_inner_microstep: 431.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:53,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.79 | bwd_microstep: 485.04 | bwd_inner_microstep: 484.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:54,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 346.33 | bwd_microstep: 714.94 | bwd_inner_microstep: 714.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:56,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.43 | bwd_microstep: 591.27 | bwd_inner_microstep: 591.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:56,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.07 | bwd_microstep: 397.69 | bwd_inner_microstep: 397.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:57,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.74 | bwd_microstep: 447.60 | bwd_inner_microstep: 447.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:06:58,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.85 | bwd_microstep: 411.73 | bwd_inner_microstep: 411.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:06:59,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.97 | bwd_microstep: 526.80 | bwd_inner_microstep: 526.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:00,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.69 | bwd_microstep: 487.63 | bwd_inner_microstep: 487.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:01,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.45 | bwd_microstep: 454.85 | bwd_inner_microstep: 454.69 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:07:02,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.56 | bwd_microstep: 551.56 | bwd_inner_microstep: 551.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:03,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.12 | bwd_microstep: 488.18 | bwd_inner_microstep: 488.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:07:04,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.23 | bwd_microstep: 448.12 | bwd_inner_microstep: 447.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:07:05,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.57 | bwd_microstep: 532.07 | bwd_inner_microstep: 531.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:07:07,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.03 | optimizer_gradients: 14.07 | optimizer_step: 23.14
[2024-12-05 15:07:07,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.28 | bwd_microstep: 928.09 | bwd_inner_microstep: 915.92 | bwd_allreduce_microstep: 12.07 | step_microstep: 111.77
[2024-12-05 15:07:07,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4512.19 | bwd: 8451.71 | bwd_inner: 8437.92 | bwd_allreduce: 12.70 | step: 112.83

 75%|███████▌  | 289/384 [1:24:03<25:37, 16.19s/it]
                                                   
{'loss': 0.1146, 'grad_norm': 1.8483508825302124, 'learning_rate': 1.5246837228164905e-07, 'epoch': 1.5}

 75%|███████▌  | 289/384 [1:24:03<25:37, 16.19s/it][2024-12-05 15:07:08,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.18 | bwd_microstep: 588.47 | bwd_inner_microstep: 588.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:09,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.22 | bwd_microstep: 378.44 | bwd_inner_microstep: 378.29 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:07:10,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.02 | bwd_microstep: 445.33 | bwd_inner_microstep: 445.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:10,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.74 | bwd_microstep: 445.07 | bwd_inner_microstep: 444.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:11,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.06 | bwd_microstep: 558.91 | bwd_inner_microstep: 558.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:07:12,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.65 | bwd_microstep: 403.39 | bwd_inner_microstep: 403.25 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:07:13,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.08 | bwd_microstep: 431.63 | bwd_inner_microstep: 431.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:07:14,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.29 | bwd_microstep: 435.81 | bwd_inner_microstep: 435.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:07:15,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.38 | bwd_microstep: 455.11 | bwd_inner_microstep: 454.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:16,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.94 | bwd_microstep: 504.22 | bwd_inner_microstep: 504.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:07:17,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.65 | bwd_microstep: 563.25 | bwd_inner_microstep: 563.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:18,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.23 | bwd_microstep: 448.90 | bwd_inner_microstep: 448.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:19,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.99 | bwd_microstep: 566.29 | bwd_inner_microstep: 566.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:07:20,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 390.30 | bwd_microstep: 859.43 | bwd_inner_microstep: 859.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:07:21,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 332.87 | bwd_microstep: 632.48 | bwd_inner_microstep: 632.35 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:07:23,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.51 | optimizer_gradients: 14.00 | optimizer_step: 23.10
[2024-12-05 15:07:23,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.22 | bwd_microstep: 1001.13 | bwd_inner_microstep: 989.21 | bwd_allreduce_microstep: 11.84 | step_microstep: 112.13
[2024-12-05 15:07:23,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4690.88 | bwd: 8717.84 | bwd_inner: 8704.22 | bwd_allreduce: 12.48 | step: 113.27

 76%|███████▌  | 290/384 [1:24:19<25:23, 16.21s/it]
                                                   
{'loss': 0.1413, 'grad_norm': 1.8255398273468018, 'learning_rate': 1.494449900539737e-07, 'epoch': 1.51}

 76%|███████▌  | 290/384 [1:24:19<25:23, 16.21s/it][2024-12-05 15:07:24,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.27 | bwd_microstep: 410.32 | bwd_inner_microstep: 410.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:07:25,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.41 | bwd_microstep: 429.51 | bwd_inner_microstep: 429.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:07:26,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.68 | bwd_microstep: 434.97 | bwd_inner_microstep: 434.85 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:07:27,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.21 | bwd_microstep: 494.81 | bwd_inner_microstep: 494.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:27,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.37 | bwd_microstep: 487.19 | bwd_inner_microstep: 487.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:28,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.81 | bwd_microstep: 589.24 | bwd_inner_microstep: 589.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:29,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.87 | bwd_microstep: 459.91 | bwd_inner_microstep: 459.76 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:07:30,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.46 | bwd_microstep: 432.89 | bwd_inner_microstep: 432.72 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 15:07:31,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.65 | bwd_microstep: 429.37 | bwd_inner_microstep: 429.25 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:07:32,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.45 | bwd_microstep: 477.02 | bwd_inner_microstep: 476.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:07:33,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.14 | bwd_microstep: 410.17 | bwd_inner_microstep: 410.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.35
[2024-12-05 15:07:34,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.06 | bwd_microstep: 582.65 | bwd_inner_microstep: 582.47 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 15:07:35,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 373.30 | bwd_microstep: 776.05 | bwd_inner_microstep: 775.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:07:36,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.76 | bwd_microstep: 410.49 | bwd_inner_microstep: 410.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:07:37,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.46 | bwd_microstep: 580.15 | bwd_inner_microstep: 580.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:39,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.43 | optimizer_gradients: 14.06 | optimizer_step: 23.15
[2024-12-05 15:07:39,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.89 | bwd_microstep: 1415.12 | bwd_inner_microstep: 1402.83 | bwd_allreduce_microstep: 12.18 | step_microstep: 112.15
[2024-12-05 15:07:39,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4600.86 | bwd: 8819.85 | bwd_inner: 8805.81 | bwd_allreduce: 12.85 | step: 113.47

 76%|███████▌  | 291/384 [1:24:35<25:00, 16.13s/it]
                                                   
{'loss': 0.1316, 'grad_norm': 1.8132373094558716, 'learning_rate': 1.4644660940672627e-07, 'epoch': 1.51}

 76%|███████▌  | 291/384 [1:24:35<25:00, 16.13s/it][2024-12-05 15:07:40,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.19 | bwd_microstep: 426.25 | bwd_inner_microstep: 426.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:07:41,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.24 | bwd_microstep: 451.06 | bwd_inner_microstep: 450.88 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 15:07:42,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.89 | bwd_microstep: 404.77 | bwd_inner_microstep: 404.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:07:42,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.72 | bwd_microstep: 426.61 | bwd_inner_microstep: 426.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:43,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.07 | bwd_microstep: 481.80 | bwd_inner_microstep: 481.63 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:07:44,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.96 | bwd_microstep: 349.64 | bwd_inner_microstep: 349.49 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:07:45,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.65 | bwd_microstep: 432.37 | bwd_inner_microstep: 432.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:46,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.98 | bwd_microstep: 615.30 | bwd_inner_microstep: 615.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:47,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.28 | bwd_microstep: 478.25 | bwd_inner_microstep: 478.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:07:48,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.40 | bwd_microstep: 413.10 | bwd_inner_microstep: 412.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:07:49,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.67 | bwd_microstep: 405.12 | bwd_inner_microstep: 405.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:07:50,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.53 | bwd_microstep: 464.19 | bwd_inner_microstep: 464.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:07:50,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.38 | bwd_microstep: 481.25 | bwd_inner_microstep: 481.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:51,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.06 | bwd_microstep: 487.50 | bwd_inner_microstep: 487.34 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:07:52,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.10 | bwd_microstep: 428.00 | bwd_inner_microstep: 427.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:55,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.42 | optimizer_gradients: 14.12 | optimizer_step: 23.30
[2024-12-05 15:07:55,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.56 | bwd_microstep: 1881.99 | bwd_inner_microstep: 1869.84 | bwd_allreduce_microstep: 12.07 | step_microstep: 112.33
[2024-12-05 15:07:55,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4392.73 | bwd: 8627.20 | bwd_inner: 8613.27 | bwd_allreduce: 12.75 | step: 113.49

 76%|███████▌  | 292/384 [1:24:51<24:29, 15.98s/it]
                                                   
{'loss': 0.1252, 'grad_norm': 1.7580801248550415, 'learning_rate': 1.434734441843899e-07, 'epoch': 1.52}

 76%|███████▌  | 292/384 [1:24:51<24:29, 15.98s/it][2024-12-05 15:07:55,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.99 | bwd_microstep: 423.35 | bwd_inner_microstep: 423.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:56,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 225.65 | bwd_microstep: 327.09 | bwd_inner_microstep: 326.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:07:57,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.91 | bwd_microstep: 425.29 | bwd_inner_microstep: 425.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:07:58,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.49 | bwd_microstep: 626.23 | bwd_inner_microstep: 626.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:07:59,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.73 | bwd_microstep: 416.75 | bwd_inner_microstep: 416.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:08:00,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.44 | bwd_microstep: 434.15 | bwd_inner_microstep: 434.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:08:01,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 397.39 | bwd_microstep: 882.55 | bwd_inner_microstep: 882.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:08:02,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.60 | bwd_microstep: 334.29 | bwd_inner_microstep: 334.14 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:08:03,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.65 | bwd_microstep: 482.59 | bwd_inner_microstep: 482.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:08:04,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.42 | bwd_microstep: 681.23 | bwd_inner_microstep: 681.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:08:05,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 343.72 | bwd_microstep: 705.36 | bwd_inner_microstep: 705.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:08:06,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.14 | bwd_microstep: 430.14 | bwd_inner_microstep: 430.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:08:07,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.59 | bwd_microstep: 401.42 | bwd_inner_microstep: 401.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:08:08,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.86 | bwd_microstep: 582.74 | bwd_inner_microstep: 582.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:08:08,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.44 | bwd_microstep: 394.88 | bwd_inner_microstep: 394.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:08:10,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.74 | optimizer_gradients: 13.99 | optimizer_step: 23.86
[2024-12-05 15:08:10,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.84 | bwd_microstep: 1268.44 | bwd_inner_microstep: 1256.33 | bwd_allreduce_microstep: 12.01 | step_microstep: 112.20
[2024-12-05 15:08:10,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4494.93 | bwd: 8816.47 | bwd_inner: 8802.71 | bwd_allreduce: 12.66 | step: 113.35

 76%|███████▋  | 293/384 [1:25:06<24:07, 15.90s/it]
                                                   
{'loss': 0.1087, 'grad_norm': 1.4802451133728027, 'learning_rate': 1.4052570643308375e-07, 'epoch': 1.52}

 76%|███████▋  | 293/384 [1:25:06<24:07, 15.90s/it][2024-12-05 15:08:11,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.05 | bwd_microstep: 452.84 | bwd_inner_microstep: 452.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:08:12,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.23 | bwd_microstep: 552.99 | bwd_inner_microstep: 552.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:08:13,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.67 | bwd_microstep: 614.34 | bwd_inner_microstep: 614.21 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:08:14,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.55 | bwd_microstep: 479.19 | bwd_inner_microstep: 479.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:08:15,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.53 | bwd_microstep: 553.64 | bwd_inner_microstep: 553.49 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 15:08:16,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.34 | bwd_microstep: 680.37 | bwd_inner_microstep: 680.22 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 15:08:17,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.27 | bwd_microstep: 412.62 | bwd_inner_microstep: 412.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:08:18,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.81 | bwd_microstep: 425.81 | bwd_inner_microstep: 425.67 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:08:19,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.80 | bwd_microstep: 432.08 | bwd_inner_microstep: 431.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:08:19,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.64 | bwd_microstep: 445.71 | bwd_inner_microstep: 445.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:08:20,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.44 | bwd_microstep: 586.52 | bwd_inner_microstep: 586.38 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:08:21,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.59 | bwd_microstep: 429.55 | bwd_inner_microstep: 429.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:08:22,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.06 | bwd_microstep: 432.30 | bwd_inner_microstep: 432.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:08:23,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.10 | bwd_microstep: 453.68 | bwd_inner_microstep: 453.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:08:23,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.19 | bwd_microstep: 425.66 | bwd_inner_microstep: 425.53 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 15:08:25,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.47 | optimizer_gradients: 14.01 | optimizer_step: 23.12
[2024-12-05 15:08:25,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.31 | bwd_microstep: 1290.02 | bwd_inner_microstep: 1277.92 | bwd_allreduce_microstep: 12.01 | step_microstep: 112.06
[2024-12-05 15:08:25,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4490.62 | bwd: 8667.32 | bwd_inner: 8653.48 | bwd_allreduce: 12.69 | step: 113.18

 77%|███████▋  | 294/384 [1:25:21<23:25, 15.62s/it]
                                                   
{'loss': 0.1063, 'grad_norm': 1.595434308052063, 'learning_rate': 1.376036063854401e-07, 'epoch': 1.53}

 77%|███████▋  | 294/384 [1:25:21<23:25, 15.62s/it][2024-12-05 15:08:26,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.73 | bwd_microstep: 467.37 | bwd_inner_microstep: 467.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:08:27,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.00 | bwd_microstep: 418.30 | bwd_inner_microstep: 418.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:08:28,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.32 | bwd_microstep: 624.47 | bwd_inner_microstep: 624.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:08:29,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.73 | bwd_microstep: 456.07 | bwd_inner_microstep: 455.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:08:30,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.89 | bwd_microstep: 560.93 | bwd_inner_microstep: 560.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:08:30,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.25 | bwd_microstep: 349.89 | bwd_inner_microstep: 349.75 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:08:31,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.54 | bwd_microstep: 446.11 | bwd_inner_microstep: 445.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:08:32,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.14 | bwd_microstep: 426.80 | bwd_inner_microstep: 426.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:08:33,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.32 | bwd_microstep: 532.87 | bwd_inner_microstep: 532.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:08:34,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.63 | bwd_microstep: 580.81 | bwd_inner_microstep: 580.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:08:35,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.66 | bwd_microstep: 442.68 | bwd_inner_microstep: 442.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:08:36,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.62 | bwd_microstep: 586.83 | bwd_inner_microstep: 586.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:08:37,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.22 | bwd_microstep: 475.86 | bwd_inner_microstep: 475.72 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:08:37,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.01 | bwd_microstep: 441.88 | bwd_inner_microstep: 441.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:08:38,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.71 | bwd_microstep: 482.10 | bwd_inner_microstep: 481.97 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:08:41,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.45 | optimizer_gradients: 14.04 | optimizer_step: 23.21
[2024-12-05 15:08:41,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.98 | bwd_microstep: 1922.33 | bwd_inner_microstep: 1910.44 | bwd_allreduce_microstep: 11.82 | step_microstep: 112.28
[2024-12-05 15:08:41,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4472.82 | bwd: 9215.30 | bwd_inner: 9201.83 | bwd_allreduce: 12.45 | step: 113.33

 77%|███████▋  | 295/384 [1:25:37<23:08, 15.60s/it]
                                                   
{'loss': 0.1071, 'grad_norm': 1.5052251815795898, 'learning_rate': 1.3470735244561027e-07, 'epoch': 1.53}

 77%|███████▋  | 295/384 [1:25:37<23:08, 15.60s/it][2024-12-05 15:08:42,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.67 | bwd_microstep: 411.71 | bwd_inner_microstep: 411.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:08:42,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.71 | bwd_microstep: 427.22 | bwd_inner_microstep: 427.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:08:43,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.02 | bwd_microstep: 533.39 | bwd_inner_microstep: 533.26 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:08:44,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.18 | bwd_microstep: 451.51 | bwd_inner_microstep: 451.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:08:45,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.47 | bwd_microstep: 472.86 | bwd_inner_microstep: 472.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:08:46,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.43 | bwd_microstep: 400.03 | bwd_inner_microstep: 399.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:08:47,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.02 | bwd_microstep: 463.21 | bwd_inner_microstep: 463.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:08:48,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.55 | bwd_microstep: 611.72 | bwd_inner_microstep: 611.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:08:48,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.06 | bwd_microstep: 400.48 | bwd_inner_microstep: 400.33 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 15:08:49,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.98 | bwd_microstep: 435.95 | bwd_inner_microstep: 435.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:08:50,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.04 | bwd_microstep: 405.51 | bwd_inner_microstep: 405.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:08:51,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.14 | bwd_microstep: 427.25 | bwd_inner_microstep: 427.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:08:52,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.71 | bwd_microstep: 529.40 | bwd_inner_microstep: 529.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:08:53,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.75 | bwd_microstep: 411.66 | bwd_inner_microstep: 411.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:08:54,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.16 | bwd_microstep: 527.45 | bwd_inner_microstep: 527.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:08:56,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.01 | optimizer_gradients: 14.01 | optimizer_step: 23.49
[2024-12-05 15:08:56,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.76 | bwd_microstep: 1733.63 | bwd_inner_microstep: 1721.47 | bwd_allreduce_microstep: 12.06 | step_microstep: 112.10
[2024-12-05 15:08:56,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4294.75 | bwd: 8642.96 | bwd_inner: 8629.18 | bwd_allreduce: 12.72 | step: 113.21

 77%|███████▋  | 296/384 [1:25:52<22:40, 15.46s/it]
                                                   
{'loss': 0.1003, 'grad_norm': 1.4966932535171509, 'learning_rate': 1.3183715117440142e-07, 'epoch': 1.54}

 77%|███████▋  | 296/384 [1:25:52<22:40, 15.46s/it][2024-12-05 15:08:57,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.51 | bwd_microstep: 381.03 | bwd_inner_microstep: 380.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:08:58,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.20 | bwd_microstep: 429.82 | bwd_inner_microstep: 429.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:08:58,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.15 | bwd_microstep: 416.35 | bwd_inner_microstep: 416.16 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08
[2024-12-05 15:08:59,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.80 | bwd_microstep: 448.06 | bwd_inner_microstep: 447.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:00,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.29 | bwd_microstep: 453.59 | bwd_inner_microstep: 453.44 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:09:01,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.49 | bwd_microstep: 493.99 | bwd_inner_microstep: 493.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:09:02,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.17 | bwd_microstep: 460.87 | bwd_inner_microstep: 460.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:09:04,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 382.74 | bwd_microstep: 838.37 | bwd_inner_microstep: 838.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:09:04,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.75 | bwd_microstep: 463.87 | bwd_inner_microstep: 463.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:09:05,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.09 | bwd_microstep: 439.89 | bwd_inner_microstep: 439.75 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:09:06,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.49 | bwd_microstep: 494.34 | bwd_inner_microstep: 494.17 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:09:07,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.29 | bwd_microstep: 437.13 | bwd_inner_microstep: 437.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:08,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.68 | bwd_microstep: 429.70 | bwd_inner_microstep: 429.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:09,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.60 | bwd_microstep: 451.77 | bwd_inner_microstep: 451.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:09:10,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.25 | bwd_microstep: 618.19 | bwd_inner_microstep: 618.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:12,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.23 | optimizer_gradients: 14.02 | optimizer_step: 23.19
[2024-12-05 15:09:12,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.19 | bwd_microstep: 1831.66 | bwd_inner_microstep: 1819.63 | bwd_allreduce_microstep: 11.94 | step_microstep: 111.87
[2024-12-05 15:09:12,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4475.74 | bwd: 9088.66 | bwd_inner: 9074.83 | bwd_allreduce: 12.61 | step: 112.92

 77%|███████▋  | 297/384 [1:26:08<22:46, 15.70s/it]
                                                   
{'loss': 0.1147, 'grad_norm': 1.5577093362808228, 'learning_rate': 1.2899320727454472e-07, 'epoch': 1.54}

 77%|███████▋  | 297/384 [1:26:08<22:46, 15.70s/it][2024-12-05 15:09:13,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.18 | bwd_microstep: 394.99 | bwd_inner_microstep: 394.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:14,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.57 | bwd_microstep: 452.40 | bwd_inner_microstep: 452.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:09:15,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.16 | bwd_microstep: 462.24 | bwd_inner_microstep: 462.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:16,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.06 | bwd_microstep: 395.35 | bwd_inner_microstep: 395.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:16,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.56 | bwd_microstep: 359.50 | bwd_inner_microstep: 359.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:17,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.68 | bwd_microstep: 464.32 | bwd_inner_microstep: 464.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:18,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.00 | bwd_microstep: 376.48 | bwd_inner_microstep: 376.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:19,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.75 | bwd_microstep: 388.52 | bwd_inner_microstep: 388.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:20,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.85 | bwd_microstep: 684.78 | bwd_inner_microstep: 684.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:09:21,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.30 | bwd_microstep: 343.85 | bwd_inner_microstep: 343.71 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:09:22,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.34 | bwd_microstep: 541.28 | bwd_inner_microstep: 541.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:23,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.09 | bwd_microstep: 425.73 | bwd_inner_microstep: 425.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:24,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.73 | bwd_microstep: 537.58 | bwd_inner_microstep: 537.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:09:25,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.99 | bwd_microstep: 549.37 | bwd_inner_microstep: 549.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:09:26,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.21 | bwd_microstep: 482.65 | bwd_inner_microstep: 482.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:29,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.42 | optimizer_gradients: 14.03 | optimizer_step: 23.14
[2024-12-05 15:09:29,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.23 | bwd_microstep: 2228.76 | bwd_inner_microstep: 2216.73 | bwd_allreduce_microstep: 11.94 | step_microstep: 112.06
[2024-12-05 15:09:29,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4331.74 | bwd: 9087.80 | bwd_inner: 9074.15 | bwd_allreduce: 12.56 | step: 113.04

 78%|███████▊  | 298/384 [1:26:25<22:47, 15.90s/it]
                                                   
{'loss': 0.1479, 'grad_norm': 2.081566572189331, 'learning_rate': 1.2617572357609562e-07, 'epoch': 1.55}

 78%|███████▊  | 298/384 [1:26:25<22:47, 15.90s/it][2024-12-05 15:09:29,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.51 | bwd_microstep: 380.57 | bwd_inner_microstep: 380.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:30,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.75 | bwd_microstep: 584.06 | bwd_inner_microstep: 583.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:31,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.74 | bwd_microstep: 410.64 | bwd_inner_microstep: 410.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:32,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.89 | bwd_microstep: 431.96 | bwd_inner_microstep: 431.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:33,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.38 | bwd_microstep: 398.07 | bwd_inner_microstep: 397.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:34,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.55 | bwd_microstep: 410.85 | bwd_inner_microstep: 410.72 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:09:35,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.85 | bwd_microstep: 425.92 | bwd_inner_microstep: 425.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:36,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.47 | bwd_microstep: 562.46 | bwd_inner_microstep: 562.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:36,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.55 | bwd_microstep: 358.77 | bwd_inner_microstep: 358.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:37,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.19 | bwd_microstep: 653.28 | bwd_inner_microstep: 653.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:09:38,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.87 | bwd_microstep: 347.20 | bwd_inner_microstep: 347.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:39,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 349.11 | bwd_microstep: 718.91 | bwd_inner_microstep: 718.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:09:40,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.40 | bwd_microstep: 358.46 | bwd_inner_microstep: 358.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:41,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.58 | bwd_microstep: 587.39 | bwd_inner_microstep: 587.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:09:42,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 358.15 | bwd_microstep: 745.39 | bwd_inner_microstep: 745.26 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:09:44,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.30 | optimizer_gradients: 14.01 | optimizer_step: 23.39
[2024-12-05 15:09:44,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.16 | bwd_microstep: 918.00 | bwd_inner_microstep: 906.03 | bwd_allreduce_microstep: 11.87 | step_microstep: 113.43
[2024-12-05 15:09:44,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4451.23 | bwd: 8291.93 | bwd_inner: 8278.31 | bwd_allreduce: 12.49 | step: 114.42

 78%|███████▊  | 299/384 [1:26:40<22:17, 15.73s/it]
                                                   
{'loss': 0.1061, 'grad_norm': 1.5348504781723022, 'learning_rate': 1.2338490102196825e-07, 'epoch': 1.55}

 78%|███████▊  | 299/384 [1:26:40<22:17, 15.73s/it][2024-12-05 15:09:45,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.77 | bwd_microstep: 507.27 | bwd_inner_microstep: 507.09 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 15:09:46,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.50 | bwd_microstep: 588.85 | bwd_inner_microstep: 588.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:47,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.20 | bwd_microstep: 451.38 | bwd_inner_microstep: 451.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:47,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.43 | bwd_microstep: 437.96 | bwd_inner_microstep: 437.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:48,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.79 | bwd_microstep: 432.23 | bwd_inner_microstep: 432.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:49,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.65 | bwd_microstep: 442.05 | bwd_inner_microstep: 441.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:50,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.55 | bwd_microstep: 582.65 | bwd_inner_microstep: 582.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:51,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.20 | bwd_microstep: 524.24 | bwd_inner_microstep: 524.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:09:52,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.05 | bwd_microstep: 433.05 | bwd_inner_microstep: 432.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:09:53,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.63 | bwd_microstep: 449.21 | bwd_inner_microstep: 449.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:09:54,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.27 | bwd_microstep: 461.34 | bwd_inner_microstep: 461.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:09:55,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 225.29 | bwd_microstep: 334.43 | bwd_inner_microstep: 334.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:09:55,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.05 | bwd_microstep: 512.72 | bwd_inner_microstep: 512.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:09:56,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.27 | bwd_microstep: 453.38 | bwd_inner_microstep: 453.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:09:57,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.60 | bwd_microstep: 448.70 | bwd_inner_microstep: 448.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:00,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.40 | optimizer_gradients: 14.11 | optimizer_step: 23.14
[2024-12-05 15:10:00,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.16 | bwd_microstep: 2423.13 | bwd_inner_microstep: 2410.96 | bwd_allreduce_microstep: 12.08 | step_microstep: 112.20
[2024-12-05 15:10:00,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4361.46 | bwd: 9482.58 | bwd_inner: 9468.79 | bwd_allreduce: 12.71 | step: 113.27

 78%|███████▊  | 300/384 [1:26:56<22:14, 15.89s/it]
                                                   
{'loss': 0.1359, 'grad_norm': 1.7641793489456177, 'learning_rate': 1.2062093865360457e-07, 'epoch': 1.56}

 78%|███████▊  | 300/384 [1:26:56<22:14, 15.89s/it][2024-12-05 15:10:01,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.68 | bwd_microstep: 329.18 | bwd_inner_microstep: 329.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:02,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.36 | bwd_microstep: 408.76 | bwd_inner_microstep: 408.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 15:10:02,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.43 | bwd_microstep: 418.04 | bwd_inner_microstep: 417.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:03,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.53 | bwd_microstep: 459.23 | bwd_inner_microstep: 459.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:04,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.65 | bwd_microstep: 428.23 | bwd_inner_microstep: 428.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:05,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.66 | bwd_microstep: 480.51 | bwd_inner_microstep: 480.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:06,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 357.65 | bwd_microstep: 745.48 | bwd_inner_microstep: 745.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:07,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.08 | bwd_microstep: 410.85 | bwd_inner_microstep: 410.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:10:08,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.96 | bwd_microstep: 438.61 | bwd_inner_microstep: 438.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:09,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.23 | bwd_microstep: 438.93 | bwd_inner_microstep: 438.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:09,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.12 | bwd_microstep: 436.56 | bwd_inner_microstep: 436.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:10,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.90 | bwd_microstep: 489.77 | bwd_inner_microstep: 489.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:11,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.15 | bwd_microstep: 405.10 | bwd_inner_microstep: 404.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:12,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.74 | bwd_microstep: 624.88 | bwd_inner_microstep: 624.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:13,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.76 | bwd_microstep: 541.41 | bwd_inner_microstep: 541.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:15,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.18 | optimizer_gradients: 14.04 | optimizer_step: 23.33
[2024-12-05 15:10:15,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.47 | bwd_microstep: 1411.74 | bwd_inner_microstep: 1399.60 | bwd_allreduce_microstep: 12.05 | step_microstep: 112.33
[2024-12-05 15:10:15,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4330.43 | bwd: 8467.28 | bwd_inner: 8453.53 | bwd_allreduce: 12.67 | step: 113.48

 78%|███████▊  | 301/384 [1:27:11<21:39, 15.66s/it]
                                                   
{'loss': 0.1069, 'grad_norm': 1.5747112035751343, 'learning_rate': 1.1788403359677767e-07, 'epoch': 1.56}

 78%|███████▊  | 301/384 [1:27:11<21:39, 15.66s/it][2024-12-05 15:10:16,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.54 | bwd_microstep: 415.69 | bwd_inner_microstep: 415.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:17,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.91 | bwd_microstep: 446.39 | bwd_inner_microstep: 446.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:18,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 385.85 | bwd_microstep: 841.13 | bwd_inner_microstep: 841.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:19,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.80 | bwd_microstep: 450.77 | bwd_inner_microstep: 450.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:20,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.70 | bwd_microstep: 437.48 | bwd_inner_microstep: 437.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:21,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.62 | bwd_microstep: 488.36 | bwd_inner_microstep: 488.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:21,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.27 | bwd_microstep: 420.16 | bwd_inner_microstep: 420.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:22,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.22 | bwd_microstep: 411.50 | bwd_inner_microstep: 411.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:23,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.33 | bwd_microstep: 447.76 | bwd_inner_microstep: 447.47 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.06
[2024-12-05 15:10:24,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.89 | bwd_microstep: 589.01 | bwd_inner_microstep: 588.87 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:10:25,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.73 | bwd_microstep: 358.04 | bwd_inner_microstep: 357.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:25,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.83 | bwd_microstep: 432.11 | bwd_inner_microstep: 431.93 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 15:10:26,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.51 | bwd_microstep: 499.37 | bwd_inner_microstep: 499.21 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 15:10:27,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.18 | bwd_microstep: 567.09 | bwd_inner_microstep: 566.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:28,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.23 | bwd_microstep: 348.26 | bwd_inner_microstep: 348.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:31,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.98 | optimizer_gradients: 14.08 | optimizer_step: 23.33
[2024-12-05 15:10:31,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.37 | bwd_microstep: 2477.91 | bwd_inner_microstep: 2465.76 | bwd_allreduce_microstep: 12.05 | step_microstep: 114.95
[2024-12-05 15:10:31,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4350.03 | bwd: 9631.16 | bwd_inner: 9617.03 | bwd_allreduce: 12.88 | step: 116.10

 79%|███████▊  | 302/384 [1:27:27<21:28, 15.71s/it]
                                                   
{'loss': 0.1077, 'grad_norm': 1.6333017349243164, 'learning_rate': 1.1517438104753385e-07, 'epoch': 1.57}

 79%|███████▊  | 302/384 [1:27:27<21:28, 15.71s/it][2024-12-05 15:10:32,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.63 | bwd_microstep: 495.95 | bwd_inner_microstep: 495.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:10:33,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.88 | bwd_microstep: 445.47 | bwd_inner_microstep: 445.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:34,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.83 | bwd_microstep: 478.75 | bwd_inner_microstep: 478.60 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:10:35,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.33 | bwd_microstep: 540.96 | bwd_inner_microstep: 540.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:10:36,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.73 | bwd_microstep: 456.16 | bwd_inner_microstep: 456.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:36,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.20 | bwd_microstep: 432.74 | bwd_inner_microstep: 432.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:37,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.62 | bwd_microstep: 409.07 | bwd_inner_microstep: 408.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 15:10:38,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.02 | bwd_microstep: 614.03 | bwd_inner_microstep: 613.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:10:39,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.86 | bwd_microstep: 403.75 | bwd_inner_microstep: 403.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:40,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.65 | bwd_microstep: 411.42 | bwd_inner_microstep: 411.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:41,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.39 | bwd_microstep: 354.73 | bwd_inner_microstep: 354.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:42,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.32 | bwd_microstep: 488.97 | bwd_inner_microstep: 488.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:42,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.66 | bwd_microstep: 427.52 | bwd_inner_microstep: 427.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:43,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.53 | bwd_microstep: 597.01 | bwd_inner_microstep: 596.87 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:10:44,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.81 | bwd_microstep: 414.41 | bwd_inner_microstep: 414.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:46,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.48 | optimizer_gradients: 14.08 | optimizer_step: 23.15
[2024-12-05 15:10:46,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.85 | bwd_microstep: 1392.52 | bwd_inner_microstep: 1380.18 | bwd_allreduce_microstep: 12.25 | step_microstep: 112.50
[2024-12-05 15:10:46,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4349.34 | bwd: 8363.47 | bwd_inner: 8349.50 | bwd_allreduce: 12.92 | step: 113.71

 79%|███████▉  | 303/384 [1:27:42<20:57, 15.52s/it]
                                                   
{'loss': 0.1118, 'grad_norm': 1.5666186809539795, 'learning_rate': 1.1249217425827062e-07, 'epoch': 1.58}

 79%|███████▉  | 303/384 [1:27:42<20:57, 15.52s/it][2024-12-05 15:10:47,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.08 | bwd_microstep: 468.96 | bwd_inner_microstep: 468.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:48,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.92 | bwd_microstep: 425.45 | bwd_inner_microstep: 425.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:49,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.31 | bwd_microstep: 452.53 | bwd_inner_microstep: 452.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:50,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.29 | bwd_microstep: 421.06 | bwd_inner_microstep: 420.91 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:10:50,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.48 | bwd_microstep: 441.70 | bwd_inner_microstep: 441.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:52,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 377.13 | bwd_microstep: 850.36 | bwd_inner_microstep: 850.14 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.08
[2024-12-05 15:10:52,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.37 | bwd_microstep: 371.55 | bwd_inner_microstep: 371.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:53,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.51 | bwd_microstep: 484.52 | bwd_inner_microstep: 484.37 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:10:54,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.97 | bwd_microstep: 407.89 | bwd_inner_microstep: 407.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:55,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.36 | bwd_microstep: 449.19 | bwd_inner_microstep: 449.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:56,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.69 | bwd_microstep: 410.26 | bwd_inner_microstep: 410.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:56,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.95 | bwd_microstep: 480.29 | bwd_inner_microstep: 480.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:10:57,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.37 | bwd_microstep: 431.95 | bwd_inner_microstep: 431.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:10:59,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 402.69 | bwd_microstep: 907.98 | bwd_inner_microstep: 907.84 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:10:59,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.11 | bwd_microstep: 439.74 | bwd_inner_microstep: 439.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:01,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 64.02 | optimizer_gradients: 14.05 | optimizer_step: 23.14
[2024-12-05 15:11:01,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.59 | bwd_microstep: 1453.98 | bwd_inner_microstep: 1441.63 | bwd_allreduce_microstep: 12.27 | step_microstep: 117.94
[2024-12-05 15:11:01,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4431.85 | bwd: 8897.38 | bwd_inner: 8883.28 | bwd_allreduce: 13.01 | step: 119.07

 79%|███████▉  | 304/384 [1:27:58<20:32, 15.40s/it]
                                                   
{'loss': 0.1043, 'grad_norm': 1.5051664113998413, 'learning_rate': 1.0983760452395413e-07, 'epoch': 1.58}

 79%|███████▉  | 304/384 [1:27:58<20:32, 15.40s/it][2024-12-05 15:11:02,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.39 | bwd_microstep: 413.90 | bwd_inner_microstep: 413.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:03,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.15 | bwd_microstep: 489.71 | bwd_inner_microstep: 489.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:04,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.85 | bwd_microstep: 430.53 | bwd_inner_microstep: 430.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:05,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.77 | bwd_microstep: 509.12 | bwd_inner_microstep: 508.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.22
[2024-12-05 15:11:06,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.37 | bwd_microstep: 459.87 | bwd_inner_microstep: 459.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:06,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.27 | bwd_microstep: 414.09 | bwd_inner_microstep: 413.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:11:07,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.72 | bwd_microstep: 455.55 | bwd_inner_microstep: 455.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:11:08,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.64 | bwd_microstep: 526.43 | bwd_inner_microstep: 526.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:11:09,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.99 | bwd_microstep: 425.24 | bwd_inner_microstep: 425.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:11:09,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.45 | bwd_microstep: 355.74 | bwd_inner_microstep: 355.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:11:10,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.75 | bwd_microstep: 480.62 | bwd_inner_microstep: 480.48 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:11:11,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.91 | bwd_microstep: 445.66 | bwd_inner_microstep: 445.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:11:12,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.82 | bwd_microstep: 353.76 | bwd_inner_microstep: 353.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:11:12,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.62 | bwd_microstep: 402.27 | bwd_inner_microstep: 402.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:11:13,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.38 | bwd_microstep: 680.41 | bwd_inner_microstep: 680.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:11:16,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.77 | optimizer_gradients: 14.02 | optimizer_step: 23.12
[2024-12-05 15:11:16,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.67 | bwd_microstep: 1848.93 | bwd_inner_microstep: 1836.69 | bwd_allreduce_microstep: 12.14 | step_microstep: 112.53
[2024-12-05 15:11:16,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4278.79 | bwd: 8691.84 | bwd_inner: 8677.98 | bwd_allreduce: 12.78 | step: 113.83

 79%|███████▉  | 305/384 [1:28:12<19:55, 15.13s/it]
                                                   
{'loss': 0.1327, 'grad_norm': 1.7494423389434814, 'learning_rate': 1.07210861168476e-07, 'epoch': 1.59}

 79%|███████▉  | 305/384 [1:28:12<19:55, 15.13s/it][2024-12-05 15:11:17,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.20 | bwd_microstep: 418.73 | bwd_inner_microstep: 418.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:11:17,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.95 | bwd_microstep: 432.12 | bwd_inner_microstep: 432.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:11:18,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.82 | bwd_microstep: 563.84 | bwd_inner_microstep: 563.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:19,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.46 | bwd_microstep: 410.95 | bwd_inner_microstep: 410.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:11:20,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.22 | bwd_microstep: 482.90 | bwd_inner_microstep: 482.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:21,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.71 | bwd_microstep: 489.13 | bwd_inner_microstep: 489.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:11:22,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.95 | bwd_microstep: 437.80 | bwd_inner_microstep: 437.61 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 15:11:22,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.46 | bwd_microstep: 437.30 | bwd_inner_microstep: 437.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:23,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.07 | bwd_microstep: 399.61 | bwd_inner_microstep: 399.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:24,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.94 | bwd_microstep: 488.53 | bwd_inner_microstep: 488.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.25
[2024-12-05 15:11:25,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.79 | bwd_microstep: 459.70 | bwd_inner_microstep: 459.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:26,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 344.71 | bwd_microstep: 709.86 | bwd_inner_microstep: 709.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:11:27,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.16 | bwd_microstep: 432.01 | bwd_inner_microstep: 431.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:27,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.80 | bwd_microstep: 477.96 | bwd_inner_microstep: 477.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:11:28,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.87 | bwd_microstep: 431.27 | bwd_inner_microstep: 431.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:31,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.57 | optimizer_gradients: 14.27 | optimizer_step: 23.32
[2024-12-05 15:11:31,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.81 | bwd_microstep: 2386.62 | bwd_inner_microstep: 2374.41 | bwd_allreduce_microstep: 12.10 | step_microstep: 119.69
[2024-12-05 15:11:31,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4364.96 | bwd: 9458.33 | bwd_inner: 9444.46 | bwd_allreduce: 12.76 | step: 121.07

 80%|███████▉  | 306/384 [1:28:27<19:45, 15.20s/it]
                                                   
{'loss': 0.0938, 'grad_norm': 1.5114164352416992, 'learning_rate': 1.0461213153115079e-07, 'epoch': 1.59}

 80%|███████▉  | 306/384 [1:28:27<19:45, 15.20s/it][2024-12-05 15:11:32,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.73 | bwd_microstep: 559.56 | bwd_inner_microstep: 559.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:33,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.73 | bwd_microstep: 475.83 | bwd_inner_microstep: 475.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:11:34,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.55 | bwd_microstep: 433.75 | bwd_inner_microstep: 433.62 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:11:35,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.51 | bwd_microstep: 475.95 | bwd_inner_microstep: 475.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:36,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.35 | bwd_microstep: 389.57 | bwd_inner_microstep: 389.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:37,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.88 | bwd_microstep: 426.87 | bwd_inner_microstep: 426.74 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:11:38,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.06 | bwd_microstep: 483.73 | bwd_inner_microstep: 483.59 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:11:39,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.39 | bwd_microstep: 408.26 | bwd_inner_microstep: 408.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:11:40,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.48 | bwd_microstep: 354.75 | bwd_inner_microstep: 354.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:40,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.42 | bwd_microstep: 433.69 | bwd_inner_microstep: 433.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:41,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.93 | bwd_microstep: 416.78 | bwd_inner_microstep: 416.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:42,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.82 | bwd_microstep: 455.58 | bwd_inner_microstep: 455.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:43,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.80 | bwd_microstep: 523.01 | bwd_inner_microstep: 522.80 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 15:11:44,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.40 | bwd_microstep: 620.84 | bwd_inner_microstep: 620.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:45,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.79 | bwd_microstep: 406.16 | bwd_inner_microstep: 406.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:11:48,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.85 | optimizer_gradients: 14.09 | optimizer_step: 23.11
[2024-12-05 15:11:48,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.33 | bwd_microstep: 2383.81 | bwd_inner_microstep: 2371.63 | bwd_allreduce_microstep: 12.07 | step_microstep: 112.69
[2024-12-05 15:11:48,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4307.21 | bwd: 9248.15 | bwd_inner: 9234.26 | bwd_allreduce: 12.75 | step: 113.91

 80%|███████▉  | 307/384 [1:28:44<20:11, 15.74s/it]
                                                   
{'loss': 0.1056, 'grad_norm': 1.562703251838684, 'learning_rate': 1.0204160095335479e-07, 'epoch': 1.6}

 80%|███████▉  | 307/384 [1:28:44<20:11, 15.74s/it][2024-12-05 15:11:49,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.20 | bwd_microstep: 420.82 | bwd_inner_microstep: 420.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:11:50,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.92 | bwd_microstep: 436.80 | bwd_inner_microstep: 436.46 | bwd_allreduce_microstep: 0.24 | step_microstep: 0.08
[2024-12-05 15:11:51,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.68 | bwd_microstep: 397.30 | bwd_inner_microstep: 397.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:11:52,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 344.89 | bwd_microstep: 716.11 | bwd_inner_microstep: 715.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:53,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.27 | bwd_microstep: 461.25 | bwd_inner_microstep: 461.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:11:54,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.72 | bwd_microstep: 442.17 | bwd_inner_microstep: 442.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:55,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.62 | bwd_microstep: 460.87 | bwd_inner_microstep: 460.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:56,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.62 | bwd_microstep: 459.24 | bwd_inner_microstep: 459.10 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.20
[2024-12-05 15:11:57,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.00 | bwd_microstep: 460.10 | bwd_inner_microstep: 459.97 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:11:58,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.06 | bwd_microstep: 360.96 | bwd_inner_microstep: 360.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:59,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.97 | bwd_microstep: 458.20 | bwd_inner_microstep: 458.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:11:59,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.00 | bwd_microstep: 466.31 | bwd_inner_microstep: 466.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:00,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.02 | bwd_microstep: 519.73 | bwd_inner_microstep: 519.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:01,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.42 | bwd_microstep: 408.29 | bwd_inner_microstep: 408.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:02,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.41 | bwd_microstep: 433.52 | bwd_inner_microstep: 433.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:05,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.18 | optimizer_gradients: 14.07 | optimizer_step: 23.20
[2024-12-05 15:12:05,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.05 | bwd_microstep: 2755.80 | bwd_inner_microstep: 2743.67 | bwd_allreduce_microstep: 12.04 | step_microstep: 112.60
[2024-12-05 15:12:05,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4270.91 | bwd: 9657.46 | bwd_inner: 9643.48 | bwd_allreduce: 12.88 | step: 113.91

 80%|████████  | 308/384 [1:29:01<20:27, 16.14s/it]
                                                   
{'loss': 0.1223, 'grad_norm': 1.6739459037780762, 'learning_rate': 9.94994527653078e-08, 'epoch': 1.6}

 80%|████████  | 308/384 [1:29:01<20:27, 16.14s/it][2024-12-05 15:12:06,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.53 | bwd_microstep: 559.87 | bwd_inner_microstep: 559.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:07,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.31 | bwd_microstep: 491.63 | bwd_inner_microstep: 491.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:12:08,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 330.94 | bwd_microstep: 674.56 | bwd_inner_microstep: 674.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:09,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.42 | bwd_microstep: 617.99 | bwd_inner_microstep: 617.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:12:10,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.95 | bwd_microstep: 476.30 | bwd_inner_microstep: 476.16 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:12:11,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.82 | bwd_microstep: 535.64 | bwd_inner_microstep: 535.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:12,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.51 | bwd_microstep: 526.73 | bwd_inner_microstep: 526.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:12:13,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.65 | bwd_microstep: 522.19 | bwd_inner_microstep: 522.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:14,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 331.68 | bwd_microstep: 679.01 | bwd_inner_microstep: 678.87 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:12:15,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.26 | bwd_microstep: 530.15 | bwd_inner_microstep: 530.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:12:16,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.39 | bwd_microstep: 418.12 | bwd_inner_microstep: 418.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:17,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.51 | bwd_microstep: 433.39 | bwd_inner_microstep: 433.24 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:12:18,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.25 | bwd_microstep: 492.99 | bwd_inner_microstep: 492.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:12:19,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 344.92 | bwd_microstep: 722.37 | bwd_inner_microstep: 722.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:20,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.66 | bwd_microstep: 686.99 | bwd_inner_microstep: 686.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:21,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.53 | optimizer_gradients: 14.03 | optimizer_step: 23.50
[2024-12-05 15:12:21,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.57 | bwd_microstep: 907.19 | bwd_inner_microstep: 895.03 | bwd_allreduce_microstep: 12.07 | step_microstep: 112.68
[2024-12-05 15:12:21,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4800.43 | bwd: 9275.11 | bwd_inner: 9261.29 | bwd_allreduce: 12.74 | step: 113.83

 80%|████████  | 309/384 [1:29:18<20:13, 16.18s/it]
                                                   
{'loss': 0.1201, 'grad_norm': 1.6659190654754639, 'learning_rate': 9.69858682729976e-08, 'epoch': 1.61}

 80%|████████  | 309/384 [1:29:18<20:13, 16.18s/it][2024-12-05 15:12:22,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.25 | bwd_microstep: 378.23 | bwd_inner_microstep: 378.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:12:23,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.90 | bwd_microstep: 450.22 | bwd_inner_microstep: 450.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:12:24,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.66 | bwd_microstep: 410.68 | bwd_inner_microstep: 410.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:25,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.12 | bwd_microstep: 417.68 | bwd_inner_microstep: 417.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:26,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.35 | bwd_microstep: 463.41 | bwd_inner_microstep: 463.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:12:27,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 354.28 | bwd_microstep: 746.22 | bwd_inner_microstep: 746.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:28,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.47 | bwd_microstep: 540.67 | bwd_inner_microstep: 540.50 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:12:29,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 332.71 | bwd_microstep: 633.44 | bwd_inner_microstep: 633.29 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:12:30,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.40 | bwd_microstep: 489.78 | bwd_inner_microstep: 489.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:31,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.81 | bwd_microstep: 458.42 | bwd_inner_microstep: 458.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:12:32,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 357.59 | bwd_microstep: 755.62 | bwd_inner_microstep: 755.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:12:33,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.56 | bwd_microstep: 589.41 | bwd_inner_microstep: 589.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:34,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.24 | bwd_microstep: 486.49 | bwd_inner_microstep: 486.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:35,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 332.25 | bwd_microstep: 673.52 | bwd_inner_microstep: 673.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:36,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.91 | bwd_microstep: 592.02 | bwd_inner_microstep: 591.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:38,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.21 | optimizer_gradients: 14.02 | optimizer_step: 23.29
[2024-12-05 15:12:38,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.72 | bwd_microstep: 901.84 | bwd_inner_microstep: 889.64 | bwd_allreduce_microstep: 12.11 | step_microstep: 111.76
[2024-12-05 15:12:38,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4671.28 | bwd: 8987.64 | bwd_inner: 8973.75 | bwd_allreduce: 12.75 | step: 112.96

 81%|████████  | 310/384 [1:29:34<19:57, 16.18s/it]
                                                   
{'loss': 0.1041, 'grad_norm': 1.6198205947875977, 'learning_rate': 9.45010267452495e-08, 'epoch': 1.61}

 81%|████████  | 310/384 [1:29:34<19:57, 16.18s/it][2024-12-05 15:12:39,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.28 | bwd_microstep: 560.16 | bwd_inner_microstep: 560.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:40,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.89 | bwd_microstep: 595.23 | bwd_inner_microstep: 595.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:12:41,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.42 | bwd_microstep: 436.37 | bwd_inner_microstep: 436.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:12:42,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.80 | bwd_microstep: 449.63 | bwd_inner_microstep: 449.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:12:42,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.63 | bwd_microstep: 458.94 | bwd_inner_microstep: 458.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:43,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.14 | bwd_microstep: 612.87 | bwd_inner_microstep: 612.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:44,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.21 | bwd_microstep: 412.61 | bwd_inner_microstep: 412.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:45,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 331.02 | bwd_microstep: 629.93 | bwd_inner_microstep: 629.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:46,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.19 | bwd_microstep: 596.78 | bwd_inner_microstep: 596.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:12:47,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.04 | bwd_microstep: 401.80 | bwd_inner_microstep: 401.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:12:48,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.09 | bwd_microstep: 356.30 | bwd_inner_microstep: 356.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:12:49,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.38 | bwd_microstep: 567.32 | bwd_inner_microstep: 567.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:12:50,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.61 | bwd_microstep: 438.39 | bwd_inner_microstep: 438.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.18
[2024-12-05 15:12:51,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.46 | bwd_microstep: 366.14 | bwd_inner_microstep: 366.01 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 15:12:51,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.69 | bwd_microstep: 433.99 | bwd_inner_microstep: 433.87 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:12:53,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.37 | optimizer_gradients: 14.07 | optimizer_step: 23.26
[2024-12-05 15:12:53,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.65 | bwd_microstep: 1562.44 | bwd_inner_microstep: 1550.31 | bwd_allreduce_microstep: 12.03 | step_microstep: 112.21
[2024-12-05 15:12:53,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4427.58 | bwd: 8878.90 | bwd_inner: 8865.12 | bwd_allreduce: 12.67 | step: 113.43

 81%|████████  | 311/384 [1:29:50<19:32, 16.07s/it]
                                                   
{'loss': 0.1153, 'grad_norm': 1.6569786071777344, 'learning_rate': 9.204510540094095e-08, 'epoch': 1.62}

 81%|████████  | 311/384 [1:29:50<19:32, 16.07s/it][2024-12-05 15:12:54,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.51 | bwd_microstep: 496.63 | bwd_inner_microstep: 496.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:55,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.27 | bwd_microstep: 354.01 | bwd_inner_microstep: 353.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:12:56,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.59 | bwd_microstep: 426.60 | bwd_inner_microstep: 426.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:12:57,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.55 | bwd_microstep: 448.13 | bwd_inner_microstep: 447.99 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:12:58,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.87 | bwd_microstep: 457.11 | bwd_inner_microstep: 456.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 15:12:59,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.83 | bwd_microstep: 623.81 | bwd_inner_microstep: 623.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:13:00,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 343.11 | bwd_microstep: 707.56 | bwd_inner_microstep: 707.42 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 15:13:01,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.09 | bwd_microstep: 681.05 | bwd_inner_microstep: 680.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:13:02,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.07 | bwd_microstep: 401.60 | bwd_inner_microstep: 401.46 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:13:03,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 244.82 | bwd_microstep: 402.05 | bwd_inner_microstep: 401.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:04,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 329.67 | bwd_microstep: 633.53 | bwd_inner_microstep: 633.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:13:05,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.69 | bwd_microstep: 616.54 | bwd_inner_microstep: 616.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:13:06,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.72 | bwd_microstep: 435.56 | bwd_inner_microstep: 435.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:07,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.21 | bwd_microstep: 614.10 | bwd_inner_microstep: 613.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:13:08,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.55 | bwd_microstep: 445.97 | bwd_inner_microstep: 445.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:13:10,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.06 | optimizer_gradients: 14.05 | optimizer_step: 23.27
[2024-12-05 15:13:10,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 343.49 | bwd_microstep: 931.86 | bwd_inner_microstep: 919.74 | bwd_allreduce_microstep: 12.03 | step_microstep: 111.85
[2024-12-05 15:13:10,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4650.15 | bwd: 8676.10 | bwd_inner: 8662.35 | bwd_allreduce: 12.68 | step: 113.02

 81%|████████▏ | 312/384 [1:30:06<19:16, 16.06s/it]
                                                   
{'loss': 0.1195, 'grad_norm': 1.6825376749038696, 'learning_rate': 8.961827939636196e-08, 'epoch': 1.62}

 81%|████████▏ | 312/384 [1:30:06<19:16, 16.06s/it][2024-12-05 15:13:10,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.67 | bwd_microstep: 382.80 | bwd_inner_microstep: 382.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:12,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 485.07 | bwd_microstep: 1201.53 | bwd_inner_microstep: 1201.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:13,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.93 | bwd_microstep: 453.35 | bwd_inner_microstep: 453.17 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 15:13:14,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.53 | bwd_microstep: 484.96 | bwd_inner_microstep: 484.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:13:15,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.68 | bwd_microstep: 441.17 | bwd_inner_microstep: 441.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:13:16,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.32 | bwd_microstep: 409.47 | bwd_inner_microstep: 409.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:17,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.48 | bwd_microstep: 487.78 | bwd_inner_microstep: 487.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:13:17,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.75 | bwd_microstep: 409.46 | bwd_inner_microstep: 409.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:18,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.73 | bwd_microstep: 406.26 | bwd_inner_microstep: 406.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:13:19,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.47 | bwd_microstep: 397.34 | bwd_inner_microstep: 397.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:13:20,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.19 | bwd_microstep: 528.55 | bwd_inner_microstep: 528.40 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:13:21,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.61 | bwd_microstep: 448.86 | bwd_inner_microstep: 448.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:22,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.26 | bwd_microstep: 581.99 | bwd_inner_microstep: 581.75 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:13:23,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.99 | bwd_microstep: 488.25 | bwd_inner_microstep: 488.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:24,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.79 | bwd_microstep: 477.90 | bwd_inner_microstep: 477.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:13:26,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.83 | optimizer_gradients: 14.05 | optimizer_step: 23.10
[2024-12-05 15:13:26,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.65 | bwd_microstep: 1327.76 | bwd_inner_microstep: 1315.77 | bwd_allreduce_microstep: 11.90 | step_microstep: 112.44
[2024-12-05 15:13:26,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4546.18 | bwd: 8927.41 | bwd_inner: 8913.63 | bwd_allreduce: 12.55 | step: 113.62

 82%|████████▏ | 313/384 [1:30:22<18:59, 16.05s/it]
                                                   
{'loss': 0.14, 'grad_norm': 1.6692954301834106, 'learning_rate': 8.722072181272311e-08, 'epoch': 1.63}

 82%|████████▏ | 313/384 [1:30:22<18:59, 16.05s/it][2024-12-05 15:13:26,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.61 | bwd_microstep: 377.81 | bwd_inner_microstep: 377.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:13:27,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.79 | bwd_microstep: 395.49 | bwd_inner_microstep: 395.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:13:28,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.56 | bwd_microstep: 412.02 | bwd_inner_microstep: 411.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:13:29,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.70 | bwd_microstep: 510.02 | bwd_inner_microstep: 509.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:13:30,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.47 | bwd_microstep: 590.66 | bwd_inner_microstep: 590.42 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.12
[2024-12-05 15:13:31,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.46 | bwd_microstep: 434.68 | bwd_inner_microstep: 434.44 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13
[2024-12-05 15:13:32,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 343.35 | bwd_microstep: 684.59 | bwd_inner_microstep: 684.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:33,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.58 | bwd_microstep: 399.77 | bwd_inner_microstep: 399.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:33,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.50 | bwd_microstep: 428.50 | bwd_inner_microstep: 428.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:13:34,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.05 | bwd_microstep: 585.96 | bwd_inner_microstep: 585.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:13:35,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.58 | bwd_microstep: 599.19 | bwd_inner_microstep: 599.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:36,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.22 | bwd_microstep: 446.38 | bwd_inner_microstep: 446.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:13:37,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.84 | bwd_microstep: 474.97 | bwd_inner_microstep: 474.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:13:38,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 363.86 | bwd_microstep: 778.91 | bwd_inner_microstep: 778.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:39,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.08 | bwd_microstep: 687.10 | bwd_inner_microstep: 686.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:41,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.16 | optimizer_gradients: 14.19 | optimizer_step: 23.29
[2024-12-05 15:13:41,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.86 | bwd_microstep: 1155.73 | bwd_inner_microstep: 1143.47 | bwd_allreduce_microstep: 12.16 | step_microstep: 114.11
[2024-12-05 15:13:41,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4615.58 | bwd: 8961.84 | bwd_inner: 8947.60 | bwd_allreduce: 12.84 | step: 115.37

 82%|████████▏ | 314/384 [1:30:37<18:34, 15.93s/it]
                                                   
{'loss': 0.1154, 'grad_norm': 1.86336350440979, 'learning_rate': 8.485260364381186e-08, 'epoch': 1.63}

 82%|████████▏ | 314/384 [1:30:37<18:34, 15.93s/it][2024-12-05 15:13:42,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.62 | bwd_microstep: 427.52 | bwd_inner_microstep: 427.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:43,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 243.57 | bwd_microstep: 379.59 | bwd_inner_microstep: 379.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.17
[2024-12-05 15:13:44,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.50 | bwd_microstep: 426.12 | bwd_inner_microstep: 425.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:13:44,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.82 | bwd_microstep: 489.98 | bwd_inner_microstep: 489.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:45,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.56 | bwd_microstep: 464.39 | bwd_inner_microstep: 464.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:46,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.68 | bwd_microstep: 584.86 | bwd_inner_microstep: 584.70 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08
[2024-12-05 15:13:47,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.64 | bwd_microstep: 410.66 | bwd_inner_microstep: 410.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:48,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.52 | bwd_microstep: 433.86 | bwd_inner_microstep: 433.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:49,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.52 | bwd_microstep: 677.67 | bwd_inner_microstep: 677.53 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:13:50,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.55 | bwd_microstep: 523.54 | bwd_inner_microstep: 523.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:51,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.71 | bwd_microstep: 556.95 | bwd_inner_microstep: 556.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:13:52,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.22 | bwd_microstep: 483.93 | bwd_inner_microstep: 483.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:53,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.52 | bwd_microstep: 436.37 | bwd_inner_microstep: 436.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:13:54,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.71 | bwd_microstep: 608.81 | bwd_inner_microstep: 608.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:55,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.19 | bwd_microstep: 413.35 | bwd_inner_microstep: 413.20 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:13:56,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.40 | optimizer_gradients: 14.08 | optimizer_step: 23.21
[2024-12-05 15:13:56,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.55 | bwd_microstep: 1445.42 | bwd_inner_microstep: 1433.26 | bwd_allreduce_microstep: 12.05 | step_microstep: 112.44
[2024-12-05 15:13:56,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4489.96 | bwd: 8763.01 | bwd_inner: 8749.18 | bwd_allreduce: 12.72 | step: 113.69

 82%|████████▏ | 315/384 [1:30:53<18:06, 15.75s/it]
                                                   
{'loss': 0.1087, 'grad_norm': 1.6375104188919067, 'learning_rate': 8.251409378379637e-08, 'epoch': 1.64}

 82%|████████▏ | 315/384 [1:30:53<18:06, 15.75s/it][2024-12-05 15:13:57,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.94 | bwd_microstep: 454.62 | bwd_inner_microstep: 454.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:58,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.98 | bwd_microstep: 672.28 | bwd_inner_microstep: 672.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:13:59,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.93 | bwd_microstep: 447.77 | bwd_inner_microstep: 447.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:00,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.13 | bwd_microstep: 475.34 | bwd_inner_microstep: 475.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:01,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.36 | bwd_microstep: 538.14 | bwd_inner_microstep: 538.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:02,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.02 | bwd_microstep: 429.74 | bwd_inner_microstep: 429.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:14:03,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.92 | bwd_microstep: 425.11 | bwd_inner_microstep: 424.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:14:04,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.96 | bwd_microstep: 453.66 | bwd_inner_microstep: 453.53 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.20
[2024-12-05 15:14:05,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.66 | bwd_microstep: 519.23 | bwd_inner_microstep: 519.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:14:05,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.07 | bwd_microstep: 435.27 | bwd_inner_microstep: 435.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:07,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 495.23 | bwd_microstep: 1236.13 | bwd_inner_microstep: 1236.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:14:08,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.38 | bwd_microstep: 447.05 | bwd_inner_microstep: 446.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:14:09,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.30 | bwd_microstep: 425.06 | bwd_inner_microstep: 424.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:14:10,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 241.15 | bwd_microstep: 370.90 | bwd_inner_microstep: 370.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:14:11,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.99 | bwd_microstep: 428.54 | bwd_inner_microstep: 428.40 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:14:12,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.62 | optimizer_gradients: 14.02 | optimizer_step: 23.19
[2024-12-05 15:14:12,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.46 | bwd_microstep: 910.91 | bwd_inner_microstep: 898.63 | bwd_allreduce_microstep: 12.16 | step_microstep: 112.12
[2024-12-05 15:14:12,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4567.58 | bwd: 8669.74 | bwd_inner: 8655.83 | bwd_allreduce: 12.81 | step: 113.44

 82%|████████▏ | 316/384 [1:31:08<17:45, 15.67s/it]
                                                   
{'loss': 0.1099, 'grad_norm': 1.6260446310043335, 'learning_rate': 8.02053590151805e-08, 'epoch': 1.64}

 82%|████████▏ | 316/384 [1:31:08<17:45, 15.67s/it][2024-12-05 15:14:13,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 228.33 | bwd_microstep: 314.51 | bwd_inner_microstep: 314.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:14:14,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.08 | bwd_microstep: 557.21 | bwd_inner_microstep: 557.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:14,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.58 | bwd_microstep: 426.04 | bwd_inner_microstep: 425.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:15,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.27 | bwd_microstep: 446.48 | bwd_inner_microstep: 446.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:14:16,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.30 | bwd_microstep: 449.30 | bwd_inner_microstep: 449.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:17,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.78 | bwd_microstep: 356.01 | bwd_inner_microstep: 355.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:18,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.56 | bwd_microstep: 685.43 | bwd_inner_microstep: 685.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:19,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.75 | bwd_microstep: 565.76 | bwd_inner_microstep: 565.61 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:14:20,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.03 | bwd_microstep: 437.40 | bwd_inner_microstep: 437.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:14:21,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.18 | bwd_microstep: 626.92 | bwd_inner_microstep: 626.79 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:14:22,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.90 | bwd_microstep: 399.42 | bwd_inner_microstep: 399.25 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 15:14:23,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.92 | bwd_microstep: 425.23 | bwd_inner_microstep: 424.96 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.08
[2024-12-05 15:14:24,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 400.22 | bwd_microstep: 897.46 | bwd_inner_microstep: 897.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:25,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.35 | bwd_microstep: 431.76 | bwd_inner_microstep: 431.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:26,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.35 | bwd_microstep: 448.66 | bwd_inner_microstep: 448.53 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:14:27,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.51 | optimizer_gradients: 14.03 | optimizer_step: 23.20
[2024-12-05 15:14:27,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.67 | bwd_microstep: 897.00 | bwd_inner_microstep: 884.89 | bwd_allreduce_microstep: 12.03 | step_microstep: 112.38
[2024-12-05 15:14:27,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4465.35 | bwd: 8364.60 | bwd_inner: 8350.63 | bwd_allreduce: 12.83 | step: 113.71

 83%|████████▎ | 317/384 [1:31:24<17:24, 15.60s/it]
                                                   
{'loss': 0.1167, 'grad_norm': 1.7146011590957642, 'learning_rate': 7.792656399690922e-08, 'epoch': 1.65}

 83%|████████▎ | 317/384 [1:31:24<17:24, 15.60s/it][2024-12-05 15:14:28,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.08 | bwd_microstep: 327.16 | bwd_inner_microstep: 327.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:14:29,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.23 | bwd_microstep: 435.18 | bwd_inner_microstep: 435.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:30,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 344.24 | bwd_microstep: 709.22 | bwd_inner_microstep: 708.92 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:14:31,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.26 | bwd_microstep: 478.86 | bwd_inner_microstep: 478.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:32,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.55 | bwd_microstep: 351.87 | bwd_inner_microstep: 351.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:33,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.69 | bwd_microstep: 435.95 | bwd_inner_microstep: 435.80 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:14:33,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.04 | bwd_microstep: 380.88 | bwd_inner_microstep: 380.71 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 15:14:34,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.74 | bwd_microstep: 561.06 | bwd_inner_microstep: 560.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:36,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.70 | bwd_microstep: 585.01 | bwd_inner_microstep: 584.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:36,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.91 | bwd_microstep: 429.56 | bwd_inner_microstep: 429.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:38,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.43 | bwd_microstep: 555.69 | bwd_inner_microstep: 555.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:14:38,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.20 | bwd_microstep: 429.43 | bwd_inner_microstep: 429.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:14:39,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.15 | bwd_microstep: 409.62 | bwd_inner_microstep: 409.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:14:40,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.85 | bwd_microstep: 399.71 | bwd_inner_microstep: 399.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:41,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.34 | bwd_microstep: 689.13 | bwd_inner_microstep: 688.98 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:14:43,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.60 | optimizer_gradients: 14.01 | optimizer_step: 23.15
[2024-12-05 15:14:43,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.53 | bwd_microstep: 1904.53 | bwd_inner_microstep: 1892.37 | bwd_allreduce_microstep: 12.05 | step_microstep: 112.31
[2024-12-05 15:14:43,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4361.00 | bwd: 9082.86 | bwd_inner: 9069.02 | bwd_allreduce: 12.70 | step: 113.50

 83%|████████▎ | 318/384 [1:31:40<17:16, 15.71s/it]
                                                   
{'loss': 0.1171, 'grad_norm': 1.6423556804656982, 'learning_rate': 7.567787125262449e-08, 'epoch': 1.65}

 83%|████████▎ | 318/384 [1:31:40<17:16, 15.71s/it][2024-12-05 15:14:44,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.93 | bwd_microstep: 404.75 | bwd_inner_microstep: 404.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:14:45,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.81 | bwd_microstep: 429.89 | bwd_inner_microstep: 429.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:46,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.31 | bwd_microstep: 456.15 | bwd_inner_microstep: 456.01 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:14:47,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.43 | bwd_microstep: 456.49 | bwd_inner_microstep: 456.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:48,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.49 | bwd_microstep: 413.37 | bwd_inner_microstep: 413.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:49,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.57 | bwd_microstep: 448.46 | bwd_inner_microstep: 448.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:14:49,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.38 | bwd_microstep: 446.05 | bwd_inner_microstep: 445.90 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:14:50,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.08 | bwd_microstep: 431.44 | bwd_inner_microstep: 431.26 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 15:14:51,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.99 | bwd_microstep: 448.05 | bwd_inner_microstep: 447.90 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 15:14:52,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.82 | bwd_microstep: 434.37 | bwd_inner_microstep: 434.21 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 15:14:53,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.78 | bwd_microstep: 460.00 | bwd_inner_microstep: 459.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:14:54,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.66 | bwd_microstep: 440.73 | bwd_inner_microstep: 440.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:14:54,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.55 | bwd_microstep: 361.40 | bwd_inner_microstep: 361.22 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 15:14:55,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.38 | bwd_microstep: 455.63 | bwd_inner_microstep: 455.48 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:14:56,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.52 | bwd_microstep: 520.26 | bwd_inner_microstep: 520.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:14:59,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.28 | optimizer_gradients: 14.08 | optimizer_step: 23.37
[2024-12-05 15:14:59,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.71 | bwd_microstep: 2127.37 | bwd_inner_microstep: 2115.25 | bwd_allreduce_microstep: 12.02 | step_microstep: 112.48
[2024-12-05 15:14:59,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4240.46 | bwd: 8734.40 | bwd_inner: 8720.51 | bwd_allreduce: 12.72 | step: 113.73

 83%|████████▎ | 319/384 [1:31:55<16:56, 15.64s/it]
                                                   
{'loss': 0.1013, 'grad_norm': 1.5510940551757812, 'learning_rate': 7.345944115907421e-08, 'epoch': 1.66}

 83%|████████▎ | 319/384 [1:31:55<16:56, 15.64s/it][2024-12-05 15:15:00,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.31 | bwd_microstep: 433.90 | bwd_inner_microstep: 433.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:15:01,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.10 | bwd_microstep: 484.07 | bwd_inner_microstep: 483.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:15:02,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.35 | bwd_microstep: 497.04 | bwd_inner_microstep: 496.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:15:03,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.12 | bwd_microstep: 679.66 | bwd_inner_microstep: 679.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:15:04,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.09 | bwd_microstep: 614.33 | bwd_inner_microstep: 614.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:05,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.57 | bwd_microstep: 458.56 | bwd_inner_microstep: 458.42 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 15:15:05,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.84 | bwd_microstep: 358.21 | bwd_inner_microstep: 358.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:06,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.25 | bwd_microstep: 491.88 | bwd_inner_microstep: 491.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:07,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.95 | bwd_microstep: 355.65 | bwd_inner_microstep: 355.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:08,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.64 | bwd_microstep: 445.97 | bwd_inner_microstep: 445.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:09,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.39 | bwd_microstep: 457.47 | bwd_inner_microstep: 457.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:10,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.42 | bwd_microstep: 452.88 | bwd_inner_microstep: 452.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:15:11,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.34 | bwd_microstep: 448.88 | bwd_inner_microstep: 448.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:12,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.75 | bwd_microstep: 412.54 | bwd_inner_microstep: 412.27 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:15:12,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.19 | bwd_microstep: 488.26 | bwd_inner_microstep: 488.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:15:15,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.51 | optimizer_gradients: 14.03 | optimizer_step: 23.12
[2024-12-05 15:15:15,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.44 | bwd_microstep: 1701.55 | bwd_inner_microstep: 1689.41 | bwd_allreduce_microstep: 12.03 | step_microstep: 113.65
[2024-12-05 15:15:15,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4322.83 | bwd: 8780.84 | bwd_inner: 8767.09 | bwd_allreduce: 12.66 | step: 114.85

 83%|████████▎ | 320/384 [1:32:11<16:45, 15.71s/it]
                                                   
{'loss': 0.1423, 'grad_norm': 1.863503336906433, 'learning_rate': 7.127143193467445e-08, 'epoch': 1.66}

 83%|████████▎ | 320/384 [1:32:11<16:45, 15.71s/it][2024-12-05 15:15:16,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.97 | bwd_microstep: 417.50 | bwd_inner_microstep: 417.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:15:16,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.94 | bwd_microstep: 431.89 | bwd_inner_microstep: 431.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:17,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.84 | bwd_microstep: 555.48 | bwd_inner_microstep: 555.35 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 15:15:18,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.48 | bwd_microstep: 456.66 | bwd_inner_microstep: 456.54 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:15:19,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.07 | bwd_microstep: 488.19 | bwd_inner_microstep: 488.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:20,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.67 | bwd_microstep: 449.92 | bwd_inner_microstep: 449.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:15:21,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.97 | bwd_microstep: 585.69 | bwd_inner_microstep: 585.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:22,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.72 | bwd_microstep: 540.70 | bwd_inner_microstep: 540.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:15:23,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 345.22 | bwd_microstep: 702.75 | bwd_inner_microstep: 702.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:15:24,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.84 | bwd_microstep: 557.70 | bwd_inner_microstep: 557.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:15:25,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.07 | bwd_microstep: 586.83 | bwd_inner_microstep: 586.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:26,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.34 | bwd_microstep: 421.17 | bwd_inner_microstep: 420.99 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 15:15:27,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.11 | bwd_microstep: 348.12 | bwd_inner_microstep: 347.95 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 15:15:28,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.30 | bwd_microstep: 441.48 | bwd_inner_microstep: 441.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.23
[2024-12-05 15:15:29,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 351.57 | bwd_microstep: 716.39 | bwd_inner_microstep: 716.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:15:30,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.63 | optimizer_gradients: 14.06 | optimizer_step: 23.10
[2024-12-05 15:15:30,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 241.54 | bwd_microstep: 902.19 | bwd_inner_microstep: 889.83 | bwd_allreduce_microstep: 12.05 | step_microstep: 112.24
[2024-12-05 15:15:30,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4536.66 | bwd: 8602.66 | bwd_inner: 8588.77 | bwd_allreduce: 12.71 | step: 113.58

 84%|████████▎ | 321/384 [1:32:27<16:30, 15.72s/it]
                                                   
{'loss': 0.1112, 'grad_norm': 1.7284936904907227, 'learning_rate': 6.911399962822518e-08, 'epoch': 1.67}

 84%|████████▎ | 321/384 [1:32:27<16:30, 15.72s/it][2024-12-05 15:15:31,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.39 | bwd_microstep: 417.35 | bwd_inner_microstep: 417.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:32,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.27 | bwd_microstep: 618.52 | bwd_inner_microstep: 618.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:15:33,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.28 | bwd_microstep: 439.78 | bwd_inner_microstep: 439.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:15:34,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.43 | bwd_microstep: 475.78 | bwd_inner_microstep: 475.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:15:35,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.89 | bwd_microstep: 611.07 | bwd_inner_microstep: 610.93 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:15:36,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.48 | bwd_microstep: 403.06 | bwd_inner_microstep: 402.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:37,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.39 | bwd_microstep: 555.93 | bwd_inner_microstep: 555.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:38,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.94 | bwd_microstep: 400.01 | bwd_inner_microstep: 399.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:39,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.98 | bwd_microstep: 411.66 | bwd_inner_microstep: 411.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:40,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.42 | bwd_microstep: 413.61 | bwd_inner_microstep: 413.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:15:40,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.93 | bwd_microstep: 420.55 | bwd_inner_microstep: 420.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:15:42,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.94 | bwd_microstep: 622.85 | bwd_inner_microstep: 622.71 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:15:42,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.79 | bwd_microstep: 427.79 | bwd_inner_microstep: 427.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:15:43,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.82 | bwd_microstep: 480.05 | bwd_inner_microstep: 479.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:15:44,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.79 | bwd_microstep: 414.10 | bwd_inner_microstep: 413.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:15:46,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.66 | optimizer_gradients: 14.20 | optimizer_step: 23.21
[2024-12-05 15:15:46,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.59 | bwd_microstep: 1127.06 | bwd_inner_microstep: 1114.44 | bwd_allreduce_microstep: 12.52 | step_microstep: 113.82
[2024-12-05 15:15:46,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4388.42 | bwd: 8239.18 | bwd_inner: 8224.92 | bwd_allreduce: 13.16 | step: 114.91

 84%|████████▍ | 322/384 [1:32:42<16:08, 15.62s/it]
                                                   
{'loss': 0.136, 'grad_norm': 1.7205549478530884, 'learning_rate': 6.698729810778064e-08, 'epoch': 1.67}

 84%|████████▍ | 322/384 [1:32:42<16:08, 15.62s/it][2024-12-05 15:15:47,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.27 | bwd_microstep: 371.73 | bwd_inner_microstep: 371.59 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:15:48,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 283.77 | bwd_microstep: 479.43 | bwd_inner_microstep: 479.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:49,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.98 | bwd_microstep: 590.82 | bwd_inner_microstep: 590.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:49,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.61 | bwd_microstep: 452.40 | bwd_inner_microstep: 452.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:50,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.23 | bwd_microstep: 455.92 | bwd_inner_microstep: 455.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:52,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.13 | bwd_microstep: 626.17 | bwd_inner_microstep: 626.03 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:15:52,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.98 | bwd_microstep: 409.75 | bwd_inner_microstep: 409.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:15:53,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.27 | bwd_microstep: 401.87 | bwd_inner_microstep: 401.73 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:15:54,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.36 | bwd_microstep: 491.12 | bwd_inner_microstep: 490.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:15:55,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.06 | bwd_microstep: 434.76 | bwd_inner_microstep: 434.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:15:56,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.29 | bwd_microstep: 433.10 | bwd_inner_microstep: 432.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:15:56,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.30 | bwd_microstep: 417.88 | bwd_inner_microstep: 417.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:57,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.96 | bwd_microstep: 484.99 | bwd_inner_microstep: 484.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:58,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.47 | bwd_microstep: 486.12 | bwd_inner_microstep: 485.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:15:59,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.62 | bwd_microstep: 525.57 | bwd_inner_microstep: 525.43 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:16:02,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.91 | optimizer_gradients: 14.28 | optimizer_step: 23.49
[2024-12-05 15:16:02,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.18 | bwd_microstep: 2178.45 | bwd_inner_microstep: 2165.64 | bwd_allreduce_microstep: 12.42 | step_microstep: 113.87
[2024-12-05 15:16:02,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4380.57 | bwd: 9240.07 | bwd_inner: 9225.89 | bwd_allreduce: 13.09 | step: 115.02

 84%|████████▍ | 323/384 [1:32:58<15:58, 15.72s/it]
                                                   
{'loss': 0.1144, 'grad_norm': 1.4762628078460693, 'learning_rate': 6.48914790496759e-08, 'epoch': 1.68}

 84%|████████▍ | 323/384 [1:32:58<15:58, 15.72s/it][2024-12-05 15:16:03,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.45 | bwd_microstep: 580.43 | bwd_inner_microstep: 580.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:04,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.49 | bwd_microstep: 534.97 | bwd_inner_microstep: 534.83 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:16:05,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.07 | bwd_microstep: 481.26 | bwd_inner_microstep: 480.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:16:06,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.66 | bwd_microstep: 558.83 | bwd_inner_microstep: 558.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:16:07,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.73 | bwd_microstep: 422.21 | bwd_inner_microstep: 422.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:16:07,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.69 | bwd_microstep: 451.90 | bwd_inner_microstep: 451.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:16:08,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.10 | bwd_microstep: 518.96 | bwd_inner_microstep: 518.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:16:09,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 332.69 | bwd_microstep: 675.13 | bwd_inner_microstep: 675.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:16:10,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.81 | bwd_microstep: 490.71 | bwd_inner_microstep: 490.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:16:11,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.44 | bwd_microstep: 546.53 | bwd_inner_microstep: 546.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:12,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.06 | bwd_microstep: 415.90 | bwd_inner_microstep: 415.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:13,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.21 | bwd_microstep: 401.20 | bwd_inner_microstep: 401.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:16:14,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.58 | bwd_microstep: 608.99 | bwd_inner_microstep: 608.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:15,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.35 | bwd_microstep: 454.45 | bwd_inner_microstep: 454.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:16,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.48 | bwd_microstep: 480.01 | bwd_inner_microstep: 479.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:18,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.70 | optimizer_gradients: 14.02 | optimizer_step: 23.11
[2024-12-05 15:16:18,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.72 | bwd_microstep: 784.80 | bwd_inner_microstep: 772.62 | bwd_allreduce_microstep: 12.08 | step_microstep: 112.31
[2024-12-05 15:16:18,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4550.58 | bwd: 8406.28 | bwd_inner: 8392.43 | bwd_allreduce: 12.73 | step: 113.49

 84%|████████▍ | 324/384 [1:33:14<15:45, 15.75s/it]
                                                   
{'loss': 0.1328, 'grad_norm': 1.8362925052642822, 'learning_rate': 6.282669192770895e-08, 'epoch': 1.68}

 84%|████████▍ | 324/384 [1:33:14<15:45, 15.75s/it][2024-12-05 15:16:19,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.24 | bwd_microstep: 525.16 | bwd_inner_microstep: 525.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:20,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.67 | bwd_microstep: 449.27 | bwd_inner_microstep: 449.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:16:21,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.15 | bwd_microstep: 425.77 | bwd_inner_microstep: 425.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:16:22,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.53 | bwd_microstep: 587.41 | bwd_inner_microstep: 587.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:22,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.63 | bwd_microstep: 356.67 | bwd_inner_microstep: 356.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:23,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.88 | bwd_microstep: 431.10 | bwd_inner_microstep: 430.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:24,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.08 | bwd_microstep: 445.09 | bwd_inner_microstep: 444.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:25,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.96 | bwd_microstep: 497.86 | bwd_inner_microstep: 497.72 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:16:27,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 344.23 | bwd_microstep: 710.61 | bwd_inner_microstep: 710.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:27,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.35 | bwd_microstep: 439.21 | bwd_inner_microstep: 439.09 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:28,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.61 | bwd_microstep: 406.01 | bwd_inner_microstep: 405.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:30,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.28 | bwd_microstep: 590.50 | bwd_inner_microstep: 590.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:31,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.48 | bwd_microstep: 352.94 | bwd_inner_microstep: 352.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:16:32,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.18 | bwd_microstep: 440.39 | bwd_inner_microstep: 440.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:32,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.89 | bwd_microstep: 435.05 | bwd_inner_microstep: 434.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:16:35,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.30 | optimizer_gradients: 14.05 | optimizer_step: 23.09
[2024-12-05 15:16:35,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.08 | bwd_microstep: 1440.78 | bwd_inner_microstep: 1428.78 | bwd_allreduce_microstep: 11.91 | step_microstep: 111.90
[2024-12-05 15:16:35,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4385.31 | bwd: 8533.80 | bwd_inner: 8520.15 | bwd_allreduce: 12.53 | step: 113.07

 85%|████████▍ | 325/384 [1:33:31<15:50, 16.11s/it]
                                                   
{'loss': 0.1055, 'grad_norm': 1.4527249336242676, 'learning_rate': 6.079308400248029e-08, 'epoch': 1.69}

 85%|████████▍ | 325/384 [1:33:31<15:50, 16.11s/it][2024-12-05 15:16:35,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 229.04 | bwd_microstep: 325.34 | bwd_inner_microstep: 325.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:16:36,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.13 | bwd_microstep: 436.99 | bwd_inner_microstep: 436.84 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 15:16:37,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.08 | bwd_microstep: 531.64 | bwd_inner_microstep: 531.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:38,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.53 | bwd_microstep: 436.49 | bwd_inner_microstep: 436.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:40,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 345.62 | bwd_microstep: 717.11 | bwd_inner_microstep: 716.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:16:40,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.44 | bwd_microstep: 450.14 | bwd_inner_microstep: 450.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:16:41,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.41 | bwd_microstep: 444.18 | bwd_inner_microstep: 444.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:42,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.70 | bwd_microstep: 490.15 | bwd_inner_microstep: 490.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:43,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.81 | bwd_microstep: 467.22 | bwd_inner_microstep: 467.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:16:44,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.45 | bwd_microstep: 383.51 | bwd_inner_microstep: 383.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:45,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.35 | bwd_microstep: 410.98 | bwd_inner_microstep: 410.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:16:46,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.57 | bwd_microstep: 415.57 | bwd_inner_microstep: 415.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:16:47,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.77 | bwd_microstep: 436.35 | bwd_inner_microstep: 436.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:16:48,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.01 | bwd_microstep: 488.85 | bwd_inner_microstep: 488.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:16:49,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.80 | bwd_microstep: 432.05 | bwd_inner_microstep: 431.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:16:53,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.18 | optimizer_gradients: 14.08 | optimizer_step: 23.35
[2024-12-05 15:16:53,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.02 | bwd_microstep: 2971.94 | bwd_inner_microstep: 2959.76 | bwd_allreduce_microstep: 12.09 | step_microstep: 113.00
[2024-12-05 15:16:53,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4275.80 | bwd: 9838.52 | bwd_inner: 9824.70 | bwd_allreduce: 12.73 | step: 114.13

 85%|████████▍ | 326/384 [1:33:49<16:13, 16.78s/it]
                                                   
{'loss': 0.1451, 'grad_norm': 1.7803164720535278, 'learning_rate': 5.8790800310890456e-08, 'epoch': 1.69}

 85%|████████▍ | 326/384 [1:33:49<16:13, 16.78s/it][2024-12-05 15:16:54,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.86 | bwd_microstep: 494.75 | bwd_inner_microstep: 494.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:55,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.08 | bwd_microstep: 475.55 | bwd_inner_microstep: 475.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:56,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.45 | bwd_microstep: 456.69 | bwd_inner_microstep: 456.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:16:58,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.55 | bwd_microstep: 459.09 | bwd_inner_microstep: 458.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:16:59,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.56 | bwd_microstep: 435.76 | bwd_inner_microstep: 435.62 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:17:00,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.77 | bwd_microstep: 554.65 | bwd_inner_microstep: 554.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:17:01,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.67 | bwd_microstep: 430.63 | bwd_inner_microstep: 430.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:17:02,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.35 | bwd_microstep: 372.92 | bwd_inner_microstep: 372.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:17:03,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.34 | bwd_microstep: 591.76 | bwd_inner_microstep: 591.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:17:05,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.98 | bwd_microstep: 482.73 | bwd_inner_microstep: 482.59 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:17:06,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.80 | bwd_microstep: 401.91 | bwd_inner_microstep: 401.77 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:17:07,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.72 | bwd_microstep: 430.16 | bwd_inner_microstep: 430.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:17:08,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.78 | bwd_microstep: 526.37 | bwd_inner_microstep: 526.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.15
[2024-12-05 15:17:09,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.38 | bwd_microstep: 449.20 | bwd_inner_microstep: 449.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:17:10,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.81 | bwd_microstep: 444.60 | bwd_inner_microstep: 444.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:17:14,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.94 | optimizer_gradients: 14.04 | optimizer_step: 23.21
[2024-12-05 15:17:14,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.13 | bwd_microstep: 2224.41 | bwd_inner_microstep: 2212.25 | bwd_allreduce_microstep: 12.07 | step_microstep: 115.04
[2024-12-05 15:17:14,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4314.28 | bwd: 9231.16 | bwd_inner: 9217.37 | bwd_allreduce: 12.72 | step: 116.31

 85%|████████▌ | 327/384 [1:34:10<17:02, 17.94s/it]
                                                   
{'loss': 0.1458, 'grad_norm': 2.059589147567749, 'learning_rate': 5.6819983655795936e-08, 'epoch': 1.7}

 85%|████████▌ | 327/384 [1:34:10<17:02, 17.94s/it][2024-12-05 15:17:15,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.38 | bwd_microstep: 450.11 | bwd_inner_microstep: 449.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:17:16,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.49 | bwd_microstep: 612.78 | bwd_inner_microstep: 612.64 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:17:17,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.05 | bwd_microstep: 433.05 | bwd_inner_microstep: 432.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:17:18,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.94 | bwd_microstep: 378.62 | bwd_inner_microstep: 378.47 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 15:17:20,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.51 | bwd_microstep: 433.25 | bwd_inner_microstep: 433.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:17:21,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.96 | bwd_microstep: 676.69 | bwd_inner_microstep: 676.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:17:22,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.26 | bwd_microstep: 609.75 | bwd_inner_microstep: 609.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:17:24,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.85 | bwd_microstep: 447.93 | bwd_inner_microstep: 447.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:17:25,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.72 | bwd_microstep: 480.41 | bwd_inner_microstep: 480.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:17:26,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.49 | bwd_microstep: 478.83 | bwd_inner_microstep: 478.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:17:27,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.95 | bwd_microstep: 427.00 | bwd_inner_microstep: 426.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:17:29,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.80 | bwd_microstep: 432.76 | bwd_inner_microstep: 432.62 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:17:30,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.49 | bwd_microstep: 421.60 | bwd_inner_microstep: 421.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:17:31,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.70 | bwd_microstep: 389.89 | bwd_inner_microstep: 389.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:17:32,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.30 | bwd_microstep: 580.14 | bwd_inner_microstep: 580.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:17:35,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.92 | optimizer_gradients: 14.01 | optimizer_step: 23.08
[2024-12-05 15:17:35,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 344.14 | bwd_microstep: 1723.43 | bwd_inner_microstep: 1711.51 | bwd_allreduce_microstep: 11.83 | step_microstep: 113.73
[2024-12-05 15:17:35,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4470.09 | bwd: 8976.22 | bwd_inner: 8962.63 | bwd_allreduce: 12.48 | step: 114.83

 85%|████████▌ | 328/384 [1:34:31<17:39, 18.92s/it]
                                                   
{'loss': 0.1257, 'grad_norm': 1.7053784132003784, 'learning_rate': 5.4880774595824245e-08, 'epoch': 1.71}

 85%|████████▌ | 328/384 [1:34:31<17:39, 18.92s/it][2024-12-05 15:17:36,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.05 | bwd_microstep: 576.02 | bwd_inner_microstep: 575.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:17:38,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.41 | bwd_microstep: 586.41 | bwd_inner_microstep: 586.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:17:39,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.22 | bwd_microstep: 461.45 | bwd_inner_microstep: 461.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:17:40,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.30 | bwd_microstep: 529.20 | bwd_inner_microstep: 529.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:17:41,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.23 | bwd_microstep: 427.13 | bwd_inner_microstep: 427.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:17:42,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.03 | bwd_microstep: 463.43 | bwd_inner_microstep: 463.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:17:43,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.41 | bwd_microstep: 489.41 | bwd_inner_microstep: 489.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:17:45,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.01 | bwd_microstep: 591.84 | bwd_inner_microstep: 591.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:17:46,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.24 | bwd_microstep: 565.49 | bwd_inner_microstep: 565.35 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:17:47,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.68 | bwd_microstep: 447.77 | bwd_inner_microstep: 447.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:17:48,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.16 | bwd_microstep: 359.82 | bwd_inner_microstep: 359.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:17:49,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.69 | bwd_microstep: 455.91 | bwd_inner_microstep: 455.75 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:17:50,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.13 | bwd_microstep: 414.21 | bwd_inner_microstep: 414.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:17:51,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.96 | bwd_microstep: 443.73 | bwd_inner_microstep: 443.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:17:53,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.78 | bwd_microstep: 407.30 | bwd_inner_microstep: 407.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:17:54,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.71 | optimizer_gradients: 14.05 | optimizer_step: 23.12
[2024-12-05 15:17:54,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.19 | bwd_microstep: 867.49 | bwd_inner_microstep: 854.99 | bwd_allreduce_microstep: 12.22 | step_microstep: 114.20
[2024-12-05 15:17:54,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4354.55 | bwd: 8086.61 | bwd_inner: 8072.60 | bwd_allreduce: 12.90 | step: 115.34

 86%|████████▌ | 329/384 [1:34:50<17:27, 19.05s/it]
                                                   
{'loss': 0.1381, 'grad_norm': 1.97287118434906, 'learning_rate': 5.297331143534972e-08, 'epoch': 1.71}

 86%|████████▌ | 329/384 [1:34:50<17:27, 19.05s/it][2024-12-05 15:17:55,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.54 | bwd_microstep: 459.68 | bwd_inner_microstep: 459.53 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:17:57,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 242.30 | bwd_microstep: 378.28 | bwd_inner_microstep: 378.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:17:58,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.90 | bwd_microstep: 618.98 | bwd_inner_microstep: 618.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:17:59,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.97 | bwd_microstep: 530.45 | bwd_inner_microstep: 530.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:18:00,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.69 | bwd_microstep: 452.90 | bwd_inner_microstep: 452.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:18:01,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.99 | bwd_microstep: 429.80 | bwd_inner_microstep: 429.64 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 15:18:02,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.52 | bwd_microstep: 555.37 | bwd_inner_microstep: 555.24 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:18:03,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.24 | bwd_microstep: 431.90 | bwd_inner_microstep: 431.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:18:05,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.29 | bwd_microstep: 531.12 | bwd_inner_microstep: 530.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:18:06,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.71 | bwd_microstep: 437.81 | bwd_inner_microstep: 437.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:18:07,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.91 | bwd_microstep: 447.89 | bwd_inner_microstep: 447.75 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:18:08,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.39 | bwd_microstep: 397.99 | bwd_inner_microstep: 397.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:18:09,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.62 | bwd_microstep: 477.46 | bwd_inner_microstep: 477.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:18:10,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.33 | bwd_microstep: 482.86 | bwd_inner_microstep: 482.70 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.11
[2024-12-05 15:18:12,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.63 | bwd_microstep: 592.78 | bwd_inner_microstep: 592.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:18:15,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.19 | optimizer_gradients: 14.11 | optimizer_step: 23.26
[2024-12-05 15:18:15,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.02 | bwd_microstep: 2030.46 | bwd_inner_microstep: 2018.29 | bwd_allreduce_microstep: 12.07 | step_microstep: 112.54
[2024-12-05 15:18:15,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4385.11 | bwd: 9255.73 | bwd_inner: 9241.85 | bwd_allreduce: 12.75 | step: 113.81

 86%|████████▌ | 330/384 [1:35:11<17:31, 19.46s/it]
                                                   
{'loss': 0.1155, 'grad_norm': 1.6465667486190796, 'learning_rate': 5.109773021462921e-08, 'epoch': 1.72}

 86%|████████▌ | 330/384 [1:35:11<17:31, 19.46s/it][2024-12-05 15:18:16,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.55 | bwd_microstep: 420.79 | bwd_inner_microstep: 420.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:18:17,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.63 | bwd_microstep: 405.07 | bwd_inner_microstep: 404.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:18:18,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 355.07 | bwd_microstep: 696.35 | bwd_inner_microstep: 696.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:18:19,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.50 | bwd_microstep: 490.73 | bwd_inner_microstep: 490.58 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:18:20,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 342.20 | bwd_microstep: 668.87 | bwd_inner_microstep: 668.70 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 15:18:21,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.44 | bwd_microstep: 436.87 | bwd_inner_microstep: 436.71 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:18:23,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.54 | bwd_microstep: 618.45 | bwd_inner_microstep: 618.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:18:24,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.31 | bwd_microstep: 443.09 | bwd_inner_microstep: 442.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:18:25,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.32 | bwd_microstep: 592.66 | bwd_inner_microstep: 592.53 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:18:26,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.17 | bwd_microstep: 452.98 | bwd_inner_microstep: 452.83 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 15:18:27,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.55 | bwd_microstep: 538.61 | bwd_inner_microstep: 538.46 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:18:28,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.92 | bwd_microstep: 461.44 | bwd_inner_microstep: 461.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:18:29,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.73 | bwd_microstep: 489.85 | bwd_inner_microstep: 489.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.25
[2024-12-05 15:18:30,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.34 | bwd_microstep: 440.77 | bwd_inner_microstep: 440.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:18:31,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.59 | bwd_microstep: 428.08 | bwd_inner_microstep: 427.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:18:33,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.22 | optimizer_gradients: 14.04 | optimizer_step: 23.21
[2024-12-05 15:18:33,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.07 | bwd_microstep: 1247.89 | bwd_inner_microstep: 1235.81 | bwd_allreduce_microstep: 11.99 | step_microstep: 113.93
[2024-12-05 15:18:33,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4537.01 | bwd: 8832.48 | bwd_inner: 8818.67 | bwd_allreduce: 12.69 | step: 115.48

 86%|████████▌ | 331/384 [1:35:29<16:57, 19.20s/it]
                                                   
{'loss': 0.1224, 'grad_norm': 1.6381739377975464, 'learning_rate': 4.925416470009991e-08, 'epoch': 1.72}

 86%|████████▌ | 331/384 [1:35:29<16:57, 19.20s/it][2024-12-05 15:18:34,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.01 | bwd_microstep: 403.18 | bwd_inner_microstep: 403.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:18:35,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.62 | bwd_microstep: 479.90 | bwd_inner_microstep: 479.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:18:37,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 346.30 | bwd_microstep: 718.41 | bwd_inner_microstep: 718.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:18:37,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.25 | bwd_microstep: 405.64 | bwd_inner_microstep: 405.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:18:39,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.80 | bwd_microstep: 569.50 | bwd_inner_microstep: 569.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:18:40,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.81 | bwd_microstep: 492.65 | bwd_inner_microstep: 492.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:18:41,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.65 | bwd_microstep: 484.66 | bwd_inner_microstep: 484.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:18:42,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.05 | bwd_microstep: 411.59 | bwd_inner_microstep: 411.42 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:18:43,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.19 | bwd_microstep: 447.11 | bwd_inner_microstep: 446.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:18:44,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.57 | bwd_microstep: 591.04 | bwd_inner_microstep: 590.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:18:45,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.68 | bwd_microstep: 396.58 | bwd_inner_microstep: 396.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:18:46,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.21 | bwd_microstep: 565.60 | bwd_inner_microstep: 565.43 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 15:18:48,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 346.02 | bwd_microstep: 680.14 | bwd_inner_microstep: 680.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:18:49,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.67 | bwd_microstep: 436.40 | bwd_inner_microstep: 436.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:18:50,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.35 | bwd_microstep: 541.11 | bwd_inner_microstep: 540.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:18:52,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.40 | optimizer_gradients: 14.01 | optimizer_step: 23.17
[2024-12-05 15:18:52,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 327.25 | bwd_microstep: 1424.69 | bwd_inner_microstep: 1412.71 | bwd_allreduce_microstep: 11.90 | step_microstep: 111.96
[2024-12-05 15:18:52,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4615.49 | bwd: 9048.23 | bwd_inner: 9034.51 | bwd_allreduce: 12.55 | step: 113.06

 86%|████████▋ | 332/384 [1:35:48<16:33, 19.11s/it]
                                                   
{'loss': 0.1203, 'grad_norm': 1.5278061628341675, 'learning_rate': 4.744274637483936e-08, 'epoch': 1.73}

 86%|████████▋ | 332/384 [1:35:48<16:33, 19.11s/it][2024-12-05 15:18:53,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.27 | bwd_microstep: 521.94 | bwd_inner_microstep: 521.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:18:54,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.01 | bwd_microstep: 479.63 | bwd_inner_microstep: 479.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:18:55,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.75 | bwd_microstep: 426.54 | bwd_inner_microstep: 426.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:18:56,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.78 | bwd_microstep: 407.87 | bwd_inner_microstep: 407.73 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 15:18:58,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 330.11 | bwd_microstep: 623.55 | bwd_inner_microstep: 623.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:18:59,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.63 | bwd_microstep: 683.61 | bwd_inner_microstep: 683.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:19:00,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.27 | bwd_microstep: 586.98 | bwd_inner_microstep: 586.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:19:01,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.50 | bwd_microstep: 358.47 | bwd_inner_microstep: 358.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.28
[2024-12-05 15:19:02,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.82 | bwd_microstep: 457.35 | bwd_inner_microstep: 457.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:19:03,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.52 | bwd_microstep: 408.06 | bwd_inner_microstep: 407.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:19:04,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.96 | bwd_microstep: 405.34 | bwd_inner_microstep: 405.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.16
[2024-12-05 15:19:04,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.56 | bwd_microstep: 406.32 | bwd_inner_microstep: 406.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:19:06,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.42 | bwd_microstep: 456.79 | bwd_inner_microstep: 456.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:19:07,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.33 | bwd_microstep: 602.16 | bwd_inner_microstep: 602.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:19:08,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.99 | bwd_microstep: 499.22 | bwd_inner_microstep: 499.08 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:19:09,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.53 | optimizer_gradients: 14.03 | optimizer_step: 23.14
[2024-12-05 15:19:09,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.34 | bwd_microstep: 919.86 | bwd_inner_microstep: 907.71 | bwd_allreduce_microstep: 12.05 | step_microstep: 112.21
[2024-12-05 15:19:09,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4454.32 | bwd: 8243.66 | bwd_inner: 8229.90 | bwd_allreduce: 12.69 | step: 113.63

 87%|████████▋ | 333/384 [1:36:06<15:47, 18.57s/it]
                                                   
{'loss': 0.1334, 'grad_norm': 1.7712794542312622, 'learning_rate': 4.566360442918754e-08, 'epoch': 1.73}

 87%|████████▋ | 333/384 [1:36:06<15:47, 18.57s/it][2024-12-05 15:19:10,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.76 | bwd_microstep: 370.84 | bwd_inner_microstep: 370.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:19:12,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 367.70 | bwd_microstep: 783.51 | bwd_inner_microstep: 783.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:19:12,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.02 | bwd_microstep: 356.98 | bwd_inner_microstep: 356.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:19:13,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.12 | bwd_microstep: 426.42 | bwd_inner_microstep: 426.29 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:19:14,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.99 | bwd_microstep: 460.25 | bwd_inner_microstep: 460.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:19:15,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.97 | bwd_microstep: 438.23 | bwd_inner_microstep: 438.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:19:16,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.20 | bwd_microstep: 415.53 | bwd_inner_microstep: 415.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.23
[2024-12-05 15:19:17,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.20 | bwd_microstep: 483.72 | bwd_inner_microstep: 483.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:19:18,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.80 | bwd_microstep: 433.53 | bwd_inner_microstep: 433.26 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.08
[2024-12-05 15:19:19,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.46 | bwd_microstep: 412.14 | bwd_inner_microstep: 412.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:19:19,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.49 | bwd_microstep: 356.29 | bwd_inner_microstep: 356.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:19:20,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.54 | bwd_microstep: 586.82 | bwd_inner_microstep: 586.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:19:21,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.44 | bwd_microstep: 524.18 | bwd_inner_microstep: 524.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:19:22,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.24 | bwd_microstep: 430.01 | bwd_inner_microstep: 429.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:19:23,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.74 | bwd_microstep: 437.04 | bwd_inner_microstep: 436.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:19:26,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.71 | optimizer_gradients: 14.13 | optimizer_step: 23.05
[2024-12-05 15:19:26,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.97 | bwd_microstep: 2224.76 | bwd_inner_microstep: 2212.55 | bwd_allreduce_microstep: 12.13 | step_microstep: 114.43
[2024-12-05 15:19:26,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4282.70 | bwd: 9140.25 | bwd_inner: 9126.24 | bwd_allreduce: 12.91 | step: 115.81

 87%|████████▋ | 334/384 [1:36:23<15:03, 18.08s/it]
                                                   
{'loss': 0.1475, 'grad_norm': 1.912316083908081, 'learning_rate': 4.3916865751533306e-08, 'epoch': 1.74}

 87%|████████▋ | 334/384 [1:36:23<15:03, 18.08s/it][2024-12-05 15:19:27,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.26 | bwd_microstep: 325.40 | bwd_inner_microstep: 325.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:19:28,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.40 | bwd_microstep: 478.53 | bwd_inner_microstep: 478.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:19:29,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.78 | bwd_microstep: 408.90 | bwd_inner_microstep: 408.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:19:30,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.84 | bwd_microstep: 481.78 | bwd_inner_microstep: 481.64 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:19:31,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.53 | bwd_microstep: 428.23 | bwd_inner_microstep: 428.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:19:32,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.20 | bwd_microstep: 447.33 | bwd_inner_microstep: 447.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:19:33,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.24 | bwd_microstep: 399.41 | bwd_inner_microstep: 399.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:19:34,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.66 | bwd_microstep: 577.59 | bwd_inner_microstep: 577.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:19:35,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.82 | bwd_microstep: 549.64 | bwd_inner_microstep: 549.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:19:36,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.88 | bwd_microstep: 355.51 | bwd_inner_microstep: 355.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:19:37,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.28 | bwd_microstep: 564.70 | bwd_inner_microstep: 564.53 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.36
[2024-12-05 15:19:38,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.37 | bwd_microstep: 481.92 | bwd_inner_microstep: 481.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:19:39,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.88 | bwd_microstep: 484.86 | bwd_inner_microstep: 484.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:19:40,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.59 | bwd_microstep: 347.71 | bwd_inner_microstep: 347.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:19:41,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.99 | bwd_microstep: 403.99 | bwd_inner_microstep: 403.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.28
[2024-12-05 15:19:44,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.58 | optimizer_gradients: 14.06 | optimizer_step: 23.17
[2024-12-05 15:19:44,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.78 | bwd_microstep: 2237.31 | bwd_inner_microstep: 2225.30 | bwd_allreduce_microstep: 11.91 | step_microstep: 112.43
[2024-12-05 15:19:44,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4311.58 | bwd: 8972.83 | bwd_inner: 8959.15 | bwd_allreduce: 12.56 | step: 114.04

 87%|████████▋ | 335/384 [1:36:40<14:38, 17.93s/it]
                                                   
{'loss': 0.1165, 'grad_norm': 1.7500931024551392, 'learning_rate': 4.220265491926489e-08, 'epoch': 1.74}

 87%|████████▋ | 335/384 [1:36:40<14:38, 17.93s/it][2024-12-05 15:19:45,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.77 | bwd_microstep: 380.48 | bwd_inner_microstep: 380.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:19:46,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.30 | bwd_microstep: 399.76 | bwd_inner_microstep: 399.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:19:47,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.47 | bwd_microstep: 446.22 | bwd_inner_microstep: 445.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:19:48,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.44 | bwd_microstep: 402.61 | bwd_inner_microstep: 402.43 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 15:19:49,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.63 | bwd_microstep: 600.32 | bwd_inner_microstep: 600.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:19:50,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.16 | bwd_microstep: 454.85 | bwd_inner_microstep: 454.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.18
[2024-12-05 15:19:51,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.12 | bwd_microstep: 437.10 | bwd_inner_microstep: 436.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:19:51,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.25 | bwd_microstep: 356.08 | bwd_inner_microstep: 355.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:19:52,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.19 | bwd_microstep: 436.42 | bwd_inner_microstep: 436.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:19:53,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.27 | bwd_microstep: 579.69 | bwd_inner_microstep: 579.55 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:19:54,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.33 | bwd_microstep: 420.56 | bwd_inner_microstep: 420.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:19:55,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.38 | bwd_microstep: 438.67 | bwd_inner_microstep: 438.49 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 15:19:57,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.60 | bwd_microstep: 616.82 | bwd_inner_microstep: 616.67 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:19:58,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.54 | bwd_microstep: 566.80 | bwd_inner_microstep: 566.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:19:59,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.39 | bwd_microstep: 425.16 | bwd_inner_microstep: 425.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.20
[2024-12-05 15:20:01,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.44 | optimizer_gradients: 14.11 | optimizer_step: 23.37
[2024-12-05 15:20:01,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.36 | bwd_microstep: 2093.41 | bwd_inner_microstep: 2081.40 | bwd_allreduce_microstep: 11.92 | step_microstep: 113.23
[2024-12-05 15:20:01,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4314.27 | bwd: 9054.92 | bwd_inner: 9041.15 | bwd_allreduce: 12.60 | step: 114.64

 88%|████████▊ | 336/384 [1:36:58<14:15, 17.81s/it]
                                                   
{'loss': 0.119, 'grad_norm': 1.7582532167434692, 'learning_rate': 4.0521094189884696e-08, 'epoch': 1.75}

 88%|████████▊ | 336/384 [1:36:58<14:15, 17.81s/it][2024-12-05 15:20:03,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.58 | bwd_microstep: 653.50 | bwd_inner_microstep: 653.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:20:04,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.15 | bwd_microstep: 479.67 | bwd_inner_microstep: 479.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:20:05,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.01 | bwd_microstep: 565.69 | bwd_inner_microstep: 565.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:07,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 369.13 | bwd_microstep: 788.90 | bwd_inner_microstep: 788.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:20:08,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.74 | bwd_microstep: 567.09 | bwd_inner_microstep: 566.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:20:09,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 356.88 | bwd_microstep: 741.73 | bwd_inner_microstep: 741.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:10,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.34 | bwd_microstep: 434.71 | bwd_inner_microstep: 434.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:11,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 326.22 | bwd_microstep: 619.07 | bwd_inner_microstep: 618.93 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:20:12,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.44 | bwd_microstep: 411.28 | bwd_inner_microstep: 411.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:20:13,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.73 | bwd_microstep: 382.10 | bwd_inner_microstep: 381.93 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:20:14,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 242.30 | bwd_microstep: 350.20 | bwd_inner_microstep: 350.08 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:20:15,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.27 | bwd_microstep: 475.57 | bwd_inner_microstep: 475.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:16,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.85 | bwd_microstep: 354.11 | bwd_inner_microstep: 353.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:20:17,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.33 | bwd_microstep: 402.79 | bwd_inner_microstep: 402.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:18,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.16 | bwd_microstep: 457.21 | bwd_inner_microstep: 457.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:20,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.48 | optimizer_gradients: 14.02 | optimizer_step: 23.27
[2024-12-05 15:20:20,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.12 | bwd_microstep: 902.49 | bwd_inner_microstep: 890.34 | bwd_allreduce_microstep: 12.07 | step_microstep: 113.36
[2024-12-05 15:20:20,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4593.31 | bwd: 8586.13 | bwd_inner: 8572.31 | bwd_allreduce: 12.71 | step: 114.46

 88%|████████▊ | 337/384 [1:37:16<14:01, 17.91s/it]
                                                   
{'loss': 0.1093, 'grad_norm': 1.6794097423553467, 'learning_rate': 3.887230349229015e-08, 'epoch': 1.75}

 88%|████████▊ | 337/384 [1:37:16<14:01, 17.91s/it][2024-12-05 15:20:21,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.75 | bwd_microstep: 534.97 | bwd_inner_microstep: 534.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:20:22,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.54 | bwd_microstep: 414.39 | bwd_inner_microstep: 414.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:23,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.63 | bwd_microstep: 539.96 | bwd_inner_microstep: 539.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:20:24,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.55 | bwd_microstep: 434.06 | bwd_inner_microstep: 433.91 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 15:20:25,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.45 | bwd_microstep: 425.23 | bwd_inner_microstep: 425.10 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:20:26,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.10 | bwd_microstep: 492.59 | bwd_inner_microstep: 492.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:27,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.08 | bwd_microstep: 416.34 | bwd_inner_microstep: 416.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:28,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.90 | bwd_microstep: 438.15 | bwd_inner_microstep: 438.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:29,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.79 | bwd_microstep: 407.24 | bwd_inner_microstep: 407.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:20:31,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.51 | bwd_microstep: 592.83 | bwd_inner_microstep: 592.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:32,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.98 | bwd_microstep: 432.41 | bwd_inner_microstep: 432.25 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 15:20:33,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.60 | bwd_microstep: 450.43 | bwd_inner_microstep: 450.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:34,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.30 | bwd_microstep: 484.82 | bwd_inner_microstep: 484.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:20:35,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.86 | bwd_microstep: 589.17 | bwd_inner_microstep: 588.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:36,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.95 | bwd_microstep: 427.21 | bwd_inner_microstep: 427.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:38,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.33 | optimizer_gradients: 14.03 | optimizer_step: 23.14
[2024-12-05 15:20:38,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.71 | bwd_microstep: 1584.20 | bwd_inner_microstep: 1571.99 | bwd_allreduce_microstep: 12.12 | step_microstep: 111.84
[2024-12-05 15:20:38,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4348.75 | bwd: 8663.98 | bwd_inner: 8650.01 | bwd_allreduce: 12.86 | step: 113.03

 88%|████████▊ | 338/384 [1:37:34<13:54, 18.14s/it]
                                                   
{'loss': 0.1235, 'grad_norm': 1.797913908958435, 'learning_rate': 3.7256400418220256e-08, 'epoch': 1.76}

 88%|████████▊ | 338/384 [1:37:34<13:54, 18.14s/it][2024-12-05 15:20:39,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.69 | bwd_microstep: 365.05 | bwd_inner_microstep: 364.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:20:40,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.69 | bwd_microstep: 620.80 | bwd_inner_microstep: 620.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:20:41,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.42 | bwd_microstep: 560.91 | bwd_inner_microstep: 560.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:42,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.84 | bwd_microstep: 478.58 | bwd_inner_microstep: 478.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:44,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.78 | bwd_microstep: 612.08 | bwd_inner_microstep: 611.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:20:45,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.17 | bwd_microstep: 681.02 | bwd_inner_microstep: 680.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:46,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.68 | bwd_microstep: 487.16 | bwd_inner_microstep: 487.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:20:47,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.39 | bwd_microstep: 487.50 | bwd_inner_microstep: 487.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:20:48,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.27 | bwd_microstep: 407.78 | bwd_inner_microstep: 407.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:20:49,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 380.95 | bwd_microstep: 840.23 | bwd_inner_microstep: 840.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:50,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.22 | bwd_microstep: 413.71 | bwd_inner_microstep: 413.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:51,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.73 | bwd_microstep: 488.96 | bwd_inner_microstep: 488.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:52,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.48 | bwd_microstep: 355.53 | bwd_inner_microstep: 355.38 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:20:53,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.82 | bwd_microstep: 410.58 | bwd_inner_microstep: 410.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:20:54,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.83 | bwd_microstep: 502.13 | bwd_inner_microstep: 502.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:56,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.44 | optimizer_gradients: 14.04 | optimizer_step: 23.13
[2024-12-05 15:20:56,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.88 | bwd_microstep: 928.86 | bwd_inner_microstep: 916.63 | bwd_allreduce_microstep: 12.14 | step_microstep: 112.09
[2024-12-05 15:20:56,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4589.91 | bwd: 8640.89 | bwd_inner: 8626.96 | bwd_allreduce: 12.80 | step: 113.27

 88%|████████▊ | 339/384 [1:37:52<13:27, 17.94s/it]
                                                   
{'loss': 0.096, 'grad_norm': 1.6042097806930542, 'learning_rate': 3.567350021386895e-08, 'epoch': 1.76}

 88%|████████▊ | 339/384 [1:37:52<13:27, 17.94s/it][2024-12-05 15:20:57,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.03 | bwd_microstep: 522.52 | bwd_inner_microstep: 522.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:20:58,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.09 | bwd_microstep: 686.48 | bwd_inner_microstep: 686.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:20:59,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 348.11 | bwd_microstep: 481.86 | bwd_inner_microstep: 481.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:00,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.62 | bwd_microstep: 426.40 | bwd_inner_microstep: 426.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:01,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.42 | bwd_microstep: 425.94 | bwd_inner_microstep: 425.77 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08
[2024-12-05 15:21:02,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.29 | bwd_microstep: 417.03 | bwd_inner_microstep: 416.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:21:03,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.37 | bwd_microstep: 561.74 | bwd_inner_microstep: 561.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:21:04,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.69 | bwd_microstep: 478.47 | bwd_inner_microstep: 478.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:05,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.06 | bwd_microstep: 422.57 | bwd_inner_microstep: 422.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:21:06,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.21 | bwd_microstep: 483.76 | bwd_inner_microstep: 483.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:21:08,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 401.42 | bwd_microstep: 894.33 | bwd_inner_microstep: 894.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:21:09,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.70 | bwd_microstep: 427.97 | bwd_inner_microstep: 427.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:10,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.88 | bwd_microstep: 476.28 | bwd_inner_microstep: 476.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:21:10,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.19 | bwd_microstep: 446.53 | bwd_inner_microstep: 446.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:21:12,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.52 | bwd_microstep: 677.52 | bwd_inner_microstep: 677.34 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 15:21:13,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.74 | optimizer_gradients: 14.29 | optimizer_step: 23.74
[2024-12-05 15:21:13,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 383.80 | bwd_microstep: 1052.76 | bwd_inner_microstep: 1040.44 | bwd_allreduce_microstep: 12.20 | step_microstep: 114.26
[2024-12-05 15:21:13,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4790.45 | bwd: 8882.32 | bwd_inner: 8868.07 | bwd_allreduce: 12.87 | step: 115.47

 89%|████████▊ | 340/384 [1:38:10<13:05, 17.85s/it]
                                                   
{'loss': 0.1153, 'grad_norm': 1.7623355388641357, 'learning_rate': 3.412371577166578e-08, 'epoch': 1.77}

 89%|████████▊ | 340/384 [1:38:10<13:05, 17.85s/it][2024-12-05 15:21:14,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 271.64 | bwd_microstep: 377.85 | bwd_inner_microstep: 377.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:21:15,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.30 | bwd_microstep: 410.86 | bwd_inner_microstep: 410.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:21:16,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.90 | bwd_microstep: 555.73 | bwd_inner_microstep: 555.59 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:21:17,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 270.34 | bwd_microstep: 454.29 | bwd_inner_microstep: 454.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:18,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.87 | bwd_microstep: 400.33 | bwd_inner_microstep: 400.19 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:21:18,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.61 | bwd_microstep: 412.26 | bwd_inner_microstep: 412.13 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:21:19,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.54 | bwd_microstep: 399.89 | bwd_inner_microstep: 399.72 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:21:20,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.13 | bwd_microstep: 562.39 | bwd_inner_microstep: 562.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:21,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.83 | bwd_microstep: 411.74 | bwd_inner_microstep: 411.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:21:22,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.09 | bwd_microstep: 405.39 | bwd_inner_microstep: 405.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:23,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.00 | bwd_microstep: 402.04 | bwd_inner_microstep: 401.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:24,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.06 | bwd_microstep: 583.56 | bwd_inner_microstep: 583.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:25,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.23 | bwd_microstep: 454.72 | bwd_inner_microstep: 454.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:21:26,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.49 | bwd_microstep: 445.35 | bwd_inner_microstep: 445.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:21:26,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.54 | bwd_microstep: 440.09 | bwd_inner_microstep: 439.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:21:28,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.10 | optimizer_gradients: 14.05 | optimizer_step: 23.11
[2024-12-05 15:21:28,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.44 | bwd_microstep: 988.15 | bwd_inner_microstep: 975.98 | bwd_allreduce_microstep: 12.07 | step_microstep: 112.89
[2024-12-05 15:21:28,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4286.08 | bwd: 7704.61 | bwd_inner: 7690.75 | bwd_allreduce: 12.71 | step: 113.96

 89%|████████▉ | 341/384 [1:38:24<12:04, 16.85s/it]
                                                   
{'loss': 0.1327, 'grad_norm': 1.737052321434021, 'learning_rate': 3.260715762222449e-08, 'epoch': 1.77}

 89%|████████▉ | 341/384 [1:38:24<12:04, 16.85s/it][2024-12-05 15:21:29,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.21 | bwd_microstep: 524.31 | bwd_inner_microstep: 524.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:21:30,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 344.26 | bwd_microstep: 721.44 | bwd_inner_microstep: 721.31 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:21:31,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.58 | bwd_microstep: 451.20 | bwd_inner_microstep: 451.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:32,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.89 | bwd_microstep: 436.26 | bwd_inner_microstep: 436.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:32,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.31 | bwd_microstep: 437.37 | bwd_inner_microstep: 437.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:33,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 229.39 | bwd_microstep: 348.97 | bwd_inner_microstep: 348.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:21:34,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.49 | bwd_microstep: 412.76 | bwd_inner_microstep: 412.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:35,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.27 | bwd_microstep: 543.86 | bwd_inner_microstep: 543.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:35,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.71 | bwd_microstep: 412.73 | bwd_inner_microstep: 412.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:36,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.20 | bwd_microstep: 430.57 | bwd_inner_microstep: 430.43 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:21:37,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.45 | bwd_microstep: 452.25 | bwd_inner_microstep: 452.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:21:38,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.11 | bwd_microstep: 413.55 | bwd_inner_microstep: 413.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:21:39,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 332.73 | bwd_microstep: 683.85 | bwd_inner_microstep: 683.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:40,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.45 | bwd_microstep: 425.81 | bwd_inner_microstep: 425.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:21:41,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.44 | bwd_microstep: 609.97 | bwd_inner_microstep: 609.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:42,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.57 | optimizer_gradients: 14.10 | optimizer_step: 23.20
[2024-12-05 15:21:42,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.91 | bwd_microstep: 1027.45 | bwd_inner_microstep: 1015.42 | bwd_allreduce_microstep: 11.94 | step_microstep: 112.68
[2024-12-05 15:21:42,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4424.46 | bwd: 8332.34 | bwd_inner: 8318.67 | bwd_allreduce: 12.57 | step: 113.84

 89%|████████▉ | 342/384 [1:38:39<11:17, 16.14s/it]
                                                   
{'loss': 0.1213, 'grad_norm': 1.61103355884552, 'learning_rate': 3.1123933926459845e-08, 'epoch': 1.78}

 89%|████████▉ | 342/384 [1:38:39<11:17, 16.14s/it][2024-12-05 15:21:43,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 346.92 | bwd_microstep: 631.39 | bwd_inner_microstep: 631.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:21:44,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.24 | bwd_microstep: 527.31 | bwd_inner_microstep: 527.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:21:45,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.36 | bwd_microstep: 584.20 | bwd_inner_microstep: 584.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:21:46,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.56 | bwd_microstep: 565.84 | bwd_inner_microstep: 565.71 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 15:21:47,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.58 | bwd_microstep: 686.80 | bwd_inner_microstep: 686.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:48,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.92 | bwd_microstep: 566.91 | bwd_inner_microstep: 566.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:21:49,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.81 | bwd_microstep: 456.43 | bwd_inner_microstep: 456.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:50,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 243.41 | bwd_microstep: 371.90 | bwd_inner_microstep: 371.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:21:51,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.61 | bwd_microstep: 596.47 | bwd_inner_microstep: 596.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:21:52,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.60 | bwd_microstep: 442.96 | bwd_inner_microstep: 442.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:21:53,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.44 | bwd_microstep: 452.68 | bwd_inner_microstep: 452.45 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08
[2024-12-05 15:21:53,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.82 | bwd_microstep: 401.06 | bwd_inner_microstep: 400.90 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.11
[2024-12-05 15:21:54,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.81 | bwd_microstep: 431.91 | bwd_inner_microstep: 431.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:21:55,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.68 | bwd_microstep: 434.03 | bwd_inner_microstep: 433.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:21:56,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.36 | bwd_microstep: 347.95 | bwd_inner_microstep: 347.78 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 15:21:58,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.18 | optimizer_gradients: 14.02 | optimizer_step: 23.12
[2024-12-05 15:21:58,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.85 | bwd_microstep: 1849.78 | bwd_inner_microstep: 1837.51 | bwd_allreduce_microstep: 12.17 | step_microstep: 111.71
[2024-12-05 15:21:58,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4571.05 | bwd: 9347.61 | bwd_inner: 9333.51 | bwd_allreduce: 12.83 | step: 112.92

 89%|████████▉ | 343/384 [1:38:54<10:57, 16.03s/it]
                                                   
{'loss': 0.1262, 'grad_norm': 1.78020441532135, 'learning_rate': 2.9674150467873527e-08, 'epoch': 1.78}

 89%|████████▉ | 343/384 [1:38:54<10:57, 16.03s/it][2024-12-05 15:21:59,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 355.58 | bwd_microstep: 719.20 | bwd_inner_microstep: 719.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:22:00,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.34 | bwd_microstep: 431.94 | bwd_inner_microstep: 431.76 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 15:22:01,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 347.92 | bwd_microstep: 691.78 | bwd_inner_microstep: 691.64 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:22:02,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.95 | bwd_microstep: 411.38 | bwd_inner_microstep: 411.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:22:03,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.31 | bwd_microstep: 411.81 | bwd_inner_microstep: 411.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:04,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.47 | bwd_microstep: 427.54 | bwd_inner_microstep: 427.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:22:05,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 345.52 | bwd_microstep: 718.25 | bwd_inner_microstep: 718.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:06,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.84 | bwd_microstep: 458.29 | bwd_inner_microstep: 458.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:06,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.74 | bwd_microstep: 499.57 | bwd_inner_microstep: 499.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:07,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 329.57 | bwd_microstep: 618.24 | bwd_inner_microstep: 618.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:22:08,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.27 | bwd_microstep: 450.22 | bwd_inner_microstep: 450.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:22:09,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.57 | bwd_microstep: 445.57 | bwd_inner_microstep: 445.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:10,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.49 | bwd_microstep: 583.52 | bwd_inner_microstep: 583.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:22:11,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.19 | bwd_microstep: 447.15 | bwd_inner_microstep: 447.01 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:22:12,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.42 | bwd_microstep: 357.24 | bwd_inner_microstep: 357.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:13,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.11 | optimizer_gradients: 13.99 | optimizer_step: 23.14
[2024-12-05 15:22:13,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.44 | bwd_microstep: 1289.56 | bwd_inner_microstep: 1277.32 | bwd_allreduce_microstep: 12.14 | step_microstep: 111.63
[2024-12-05 15:22:13,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4585.71 | bwd: 8961.34 | bwd_inner: 8947.31 | bwd_allreduce: 12.80 | step: 112.83

 90%|████████▉ | 344/384 [1:39:10<10:33, 15.84s/it]
                                                   
{'loss': 0.1465, 'grad_norm': 1.7625352144241333, 'learning_rate': 2.825791064500993e-08, 'epoch': 1.79}

 90%|████████▉ | 344/384 [1:39:10<10:33, 15.84s/it][2024-12-05 15:22:14,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.46 | bwd_microstep: 361.45 | bwd_inner_microstep: 361.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:15,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.77 | bwd_microstep: 487.37 | bwd_inner_microstep: 487.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:16,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.45 | bwd_microstep: 444.59 | bwd_inner_microstep: 444.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:22:17,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.73 | bwd_microstep: 580.13 | bwd_inner_microstep: 580.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:18,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.84 | bwd_microstep: 402.51 | bwd_inner_microstep: 402.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:19,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.18 | bwd_microstep: 459.75 | bwd_inner_microstep: 459.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:22:19,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.15 | bwd_microstep: 418.43 | bwd_inner_microstep: 418.29 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 15:22:20,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.10 | bwd_microstep: 483.01 | bwd_inner_microstep: 482.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:21,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.91 | bwd_microstep: 442.75 | bwd_inner_microstep: 442.61 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:22:22,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 344.42 | bwd_microstep: 682.44 | bwd_inner_microstep: 682.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:23,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.35 | bwd_microstep: 477.73 | bwd_inner_microstep: 477.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:24,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.40 | bwd_microstep: 407.20 | bwd_inner_microstep: 407.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:22:25,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.34 | bwd_microstep: 561.04 | bwd_inner_microstep: 560.68 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:22:26,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.71 | bwd_microstep: 449.61 | bwd_inner_microstep: 449.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:26,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.37 | bwd_microstep: 427.09 | bwd_inner_microstep: 426.95 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:22:29,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.55 | optimizer_gradients: 14.01 | optimizer_step: 23.12
[2024-12-05 15:22:29,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 328.30 | bwd_microstep: 2198.49 | bwd_inner_microstep: 2186.35 | bwd_allreduce_microstep: 12.05 | step_microstep: 112.09
[2024-12-05 15:22:29,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4436.55 | bwd: 9283.59 | bwd_inner: 9269.57 | bwd_allreduce: 12.70 | step: 113.31

 90%|████████▉ | 345/384 [1:39:25<10:15, 15.79s/it]
                                                   
{'loss': 0.1289, 'grad_norm': 1.7474111318588257, 'learning_rate': 2.6875315464081562e-08, 'epoch': 1.79}

 90%|████████▉ | 345/384 [1:39:25<10:15, 15.79s/it][2024-12-05 15:22:30,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.52 | bwd_microstep: 552.38 | bwd_inner_microstep: 552.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:31,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.38 | bwd_microstep: 464.07 | bwd_inner_microstep: 463.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:32,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 244.61 | bwd_microstep: 383.57 | bwd_inner_microstep: 383.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:33,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.03 | bwd_microstep: 528.82 | bwd_inner_microstep: 528.58 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:22:34,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.41 | bwd_microstep: 618.18 | bwd_inner_microstep: 618.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:35,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.61 | bwd_microstep: 428.03 | bwd_inner_microstep: 427.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:22:35,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.98 | bwd_microstep: 442.16 | bwd_inner_microstep: 442.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:36,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.96 | bwd_microstep: 440.83 | bwd_inner_microstep: 440.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:22:37,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.40 | bwd_microstep: 428.10 | bwd_inner_microstep: 427.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:38,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.11 | bwd_microstep: 449.88 | bwd_inner_microstep: 449.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:39,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.53 | bwd_microstep: 553.21 | bwd_inner_microstep: 553.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:39,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.88 | bwd_microstep: 399.40 | bwd_inner_microstep: 399.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:22:40,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.52 | bwd_microstep: 448.45 | bwd_inner_microstep: 448.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:41,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 345.40 | bwd_microstep: 721.11 | bwd_inner_microstep: 720.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:42,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.66 | bwd_microstep: 584.03 | bwd_inner_microstep: 583.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:44,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.60 | optimizer_gradients: 14.03 | optimizer_step: 23.04
[2024-12-05 15:22:44,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.40 | bwd_microstep: 1358.46 | bwd_inner_microstep: 1346.34 | bwd_allreduce_microstep: 12.03 | step_microstep: 112.19
[2024-12-05 15:22:44,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4461.45 | bwd: 8800.70 | bwd_inner: 8786.78 | bwd_allreduce: 12.78 | step: 113.37

 90%|█████████ | 346/384 [1:39:41<09:52, 15.59s/it]
                                                   
{'loss': 0.1219, 'grad_norm': 1.7776579856872559, 'learning_rate': 2.5526463531765463e-08, 'epoch': 1.8}

 90%|█████████ | 346/384 [1:39:41<09:52, 15.59s/it][2024-12-05 15:22:45,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.52 | bwd_microstep: 590.99 | bwd_inner_microstep: 590.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:46,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.21 | bwd_microstep: 398.96 | bwd_inner_microstep: 398.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:47,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.53 | bwd_microstep: 493.71 | bwd_inner_microstep: 493.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:48,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.44 | bwd_microstep: 358.28 | bwd_inner_microstep: 358.14 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:22:48,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.52 | bwd_microstep: 429.12 | bwd_inner_microstep: 428.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:22:49,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.27 | bwd_microstep: 406.36 | bwd_inner_microstep: 406.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:22:50,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.57 | bwd_microstep: 402.37 | bwd_inner_microstep: 402.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:22:51,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.46 | bwd_microstep: 453.06 | bwd_inner_microstep: 452.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:51,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 236.04 | bwd_microstep: 342.18 | bwd_inner_microstep: 342.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:52,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.69 | bwd_microstep: 352.90 | bwd_inner_microstep: 352.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:22:53,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 353.26 | bwd_microstep: 732.41 | bwd_inner_microstep: 732.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:22:54,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.60 | bwd_microstep: 409.85 | bwd_inner_microstep: 409.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:22:55,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.98 | bwd_microstep: 469.61 | bwd_inner_microstep: 469.46 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:22:56,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.71 | bwd_microstep: 458.91 | bwd_inner_microstep: 458.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:22:56,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 227.57 | bwd_microstep: 338.37 | bwd_inner_microstep: 338.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:00,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.33 | optimizer_gradients: 14.03 | optimizer_step: 23.36
[2024-12-05 15:23:00,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.14 | bwd_microstep: 3364.68 | bwd_inner_microstep: 3352.41 | bwd_allreduce_microstep: 12.17 | step_microstep: 112.38
[2024-12-05 15:23:00,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4206.57 | bwd: 10001.73 | bwd_inner: 9987.88 | bwd_allreduce: 12.80 | step: 113.57

 90%|█████████ | 347/384 [1:39:56<09:39, 15.65s/it]
                                                   
{'loss': 0.1361, 'grad_norm': 1.7461159229278564, 'learning_rate': 2.4211451048170296e-08, 'epoch': 1.8}

 90%|█████████ | 347/384 [1:39:56<09:39, 15.65s/it][2024-12-05 15:23:01,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.84 | bwd_microstep: 421.80 | bwd_inner_microstep: 421.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:23:02,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.69 | bwd_microstep: 569.78 | bwd_inner_microstep: 569.50 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.08
[2024-12-05 15:23:03,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.18 | bwd_microstep: 410.02 | bwd_inner_microstep: 409.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:03,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 243.76 | bwd_microstep: 380.24 | bwd_inner_microstep: 380.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:04,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.04 | bwd_microstep: 552.63 | bwd_inner_microstep: 552.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:05,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.11 | bwd_microstep: 553.92 | bwd_inner_microstep: 553.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:23:06,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.90 | bwd_microstep: 450.43 | bwd_inner_microstep: 450.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:07,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.03 | bwd_microstep: 419.54 | bwd_inner_microstep: 419.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:08,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.11 | bwd_microstep: 403.42 | bwd_inner_microstep: 403.29 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:23:09,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.53 | bwd_microstep: 561.17 | bwd_inner_microstep: 561.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:10,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.58 | bwd_microstep: 487.38 | bwd_inner_microstep: 487.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:10,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.87 | bwd_microstep: 484.59 | bwd_inner_microstep: 484.46 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:23:11,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.48 | bwd_microstep: 454.08 | bwd_inner_microstep: 453.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:12,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.74 | bwd_microstep: 456.58 | bwd_inner_microstep: 456.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:13,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.23 | bwd_microstep: 488.66 | bwd_inner_microstep: 488.53 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:23:15,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.86 | optimizer_gradients: 14.01 | optimizer_step: 23.30
[2024-12-05 15:23:15,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.53 | bwd_microstep: 1697.47 | bwd_inner_microstep: 1685.33 | bwd_allreduce_microstep: 12.07 | step_microstep: 112.60
[2024-12-05 15:23:15,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4395.69 | bwd: 8791.68 | bwd_inner: 8777.74 | bwd_allreduce: 12.88 | step: 113.79

 91%|█████████ | 348/384 [1:40:11<09:15, 15.44s/it]
                                                   
{'loss': 0.1463, 'grad_norm': 1.701060175895691, 'learning_rate': 2.293037179997559e-08, 'epoch': 1.81}

 91%|█████████ | 348/384 [1:40:11<09:15, 15.44s/it][2024-12-05 15:23:16,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.72 | bwd_microstep: 456.04 | bwd_inner_microstep: 455.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:23:17,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.17 | bwd_microstep: 613.94 | bwd_inner_microstep: 613.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:23:18,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 241.99 | bwd_microstep: 378.27 | bwd_inner_microstep: 378.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:23:19,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 343.16 | bwd_microstep: 710.38 | bwd_inner_microstep: 710.21 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08
[2024-12-05 15:23:20,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.25 | bwd_microstep: 600.03 | bwd_inner_microstep: 599.89 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:23:20,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.79 | bwd_microstep: 472.47 | bwd_inner_microstep: 472.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:22,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.83 | bwd_microstep: 646.30 | bwd_inner_microstep: 646.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:22,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.15 | bwd_microstep: 486.29 | bwd_inner_microstep: 486.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:23,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.91 | bwd_microstep: 454.43 | bwd_inner_microstep: 454.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:23:24,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.47 | bwd_microstep: 622.20 | bwd_inner_microstep: 622.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:23:25,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.73 | bwd_microstep: 427.87 | bwd_inner_microstep: 427.60 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.07
[2024-12-05 15:23:26,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.82 | bwd_microstep: 544.57 | bwd_inner_microstep: 544.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:27,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.26 | bwd_microstep: 381.24 | bwd_inner_microstep: 381.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:23:28,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.06 | bwd_microstep: 554.17 | bwd_inner_microstep: 554.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:29,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 286.17 | bwd_microstep: 506.29 | bwd_inner_microstep: 506.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:30,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.24 | optimizer_gradients: 14.07 | optimizer_step: 23.86
[2024-12-05 15:23:30,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.43 | bwd_microstep: 946.85 | bwd_inner_microstep: 934.68 | bwd_allreduce_microstep: 12.08 | step_microstep: 111.59
[2024-12-05 15:23:30,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4571.95 | bwd: 8801.34 | bwd_inner: 8787.30 | bwd_allreduce: 12.76 | step: 112.75

 91%|█████████ | 349/384 [1:40:26<08:55, 15.31s/it]
                                                   
{'loss': 0.14, 'grad_norm': 1.7974237203598022, 'learning_rate': 2.1683317153742775e-08, 'epoch': 1.81}

 91%|█████████ | 349/384 [1:40:26<08:55, 15.31s/it][2024-12-05 15:23:31,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.05 | bwd_microstep: 455.29 | bwd_inner_microstep: 455.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:23:32,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.03 | bwd_microstep: 469.50 | bwd_inner_microstep: 469.38 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:32,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.06 | bwd_microstep: 436.51 | bwd_inner_microstep: 436.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:33,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.49 | bwd_microstep: 452.58 | bwd_inner_microstep: 452.44 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:23:34,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.27 | bwd_microstep: 429.23 | bwd_inner_microstep: 429.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:23:35,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.24 | bwd_microstep: 414.46 | bwd_inner_microstep: 414.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:23:36,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.89 | bwd_microstep: 538.68 | bwd_inner_microstep: 538.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:23:37,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.60 | bwd_microstep: 458.95 | bwd_inner_microstep: 458.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:37,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 240.81 | bwd_microstep: 350.27 | bwd_inner_microstep: 350.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:23:38,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.25 | bwd_microstep: 481.78 | bwd_inner_microstep: 481.48 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.08
[2024-12-05 15:23:39,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.25 | bwd_microstep: 436.62 | bwd_inner_microstep: 436.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:40,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.82 | bwd_microstep: 675.98 | bwd_inner_microstep: 675.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.22
[2024-12-05 15:23:41,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.58 | bwd_microstep: 427.83 | bwd_inner_microstep: 427.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:41,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.75 | bwd_microstep: 401.68 | bwd_inner_microstep: 401.51 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:23:42,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.45 | bwd_microstep: 590.48 | bwd_inner_microstep: 590.34 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:23:45,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.50 | optimizer_gradients: 14.02 | optimizer_step: 23.89
[2024-12-05 15:23:45,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.18 | bwd_microstep: 1931.66 | bwd_inner_microstep: 1919.41 | bwd_allreduce_microstep: 12.15 | step_microstep: 111.94
[2024-12-05 15:23:45,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4323.77 | bwd: 8951.48 | bwd_inner: 8937.39 | bwd_allreduce: 12.97 | step: 113.23

 91%|█████████ | 350/384 [1:40:41<08:35, 15.16s/it]
                                                   
{'loss': 0.1234, 'grad_norm': 1.5575218200683594, 'learning_rate': 2.047037604939894e-08, 'epoch': 1.82}

 91%|█████████ | 350/384 [1:40:41<08:35, 15.16s/it][2024-12-05 15:23:46,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 343.80 | bwd_microstep: 678.18 | bwd_inner_microstep: 678.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:47,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.95 | bwd_microstep: 464.26 | bwd_inner_microstep: 464.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:48,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.74 | bwd_microstep: 405.36 | bwd_inner_microstep: 405.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:48,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 229.60 | bwd_microstep: 336.49 | bwd_inner_microstep: 336.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:49,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 360.45 | bwd_microstep: 741.88 | bwd_inner_microstep: 741.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:50,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.47 | bwd_microstep: 412.14 | bwd_inner_microstep: 412.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:51,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.21 | bwd_microstep: 540.61 | bwd_inner_microstep: 540.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:52,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 244.33 | bwd_microstep: 372.03 | bwd_inner_microstep: 371.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:53,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.61 | bwd_microstep: 446.50 | bwd_inner_microstep: 446.26 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08
[2024-12-05 15:23:54,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.85 | bwd_microstep: 451.10 | bwd_inner_microstep: 450.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:55,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.76 | bwd_microstep: 519.44 | bwd_inner_microstep: 519.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:55,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.57 | bwd_microstep: 433.61 | bwd_inner_microstep: 433.45 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 15:23:56,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.47 | bwd_microstep: 492.28 | bwd_inner_microstep: 492.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:57,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.00 | bwd_microstep: 432.53 | bwd_inner_microstep: 432.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:23:58,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.19 | bwd_microstep: 494.81 | bwd_inner_microstep: 494.67 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:24:00,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.31 | optimizer_gradients: 14.04 | optimizer_step: 23.49
[2024-12-05 15:24:00,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.04 | bwd_microstep: 1596.31 | bwd_inner_microstep: 1584.09 | bwd_allreduce_microstep: 12.11 | step_microstep: 112.41
[2024-12-05 15:24:00,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4372.10 | bwd: 8817.53 | bwd_inner: 8803.56 | bwd_allreduce: 12.87 | step: 113.62

 91%|█████████▏| 351/384 [1:40:56<08:20, 15.18s/it]
                                                   
{'loss': 0.1119, 'grad_norm': 1.6900442838668823, 'learning_rate': 1.92916349938938e-08, 'epoch': 1.82}

 91%|█████████▏| 351/384 [1:40:56<08:20, 15.18s/it][2024-12-05 15:24:01,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.23 | bwd_microstep: 420.90 | bwd_inner_microstep: 420.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:02,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 243.46 | bwd_microstep: 378.61 | bwd_inner_microstep: 378.47 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:24:02,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.84 | bwd_microstep: 440.14 | bwd_inner_microstep: 440.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:03,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.10 | bwd_microstep: 403.70 | bwd_inner_microstep: 403.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:04,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 368.92 | bwd_microstep: 783.67 | bwd_inner_microstep: 783.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:24:05,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.73 | bwd_microstep: 601.42 | bwd_inner_microstep: 601.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:06,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.76 | bwd_microstep: 426.65 | bwd_inner_microstep: 426.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:07,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.68 | bwd_microstep: 490.49 | bwd_inner_microstep: 490.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:24:08,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.41 | bwd_microstep: 403.29 | bwd_inner_microstep: 403.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:09,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.98 | bwd_microstep: 410.72 | bwd_inner_microstep: 410.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:09,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.63 | bwd_microstep: 442.12 | bwd_inner_microstep: 441.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:10,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.16 | bwd_microstep: 485.05 | bwd_inner_microstep: 484.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:24:11,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.94 | bwd_microstep: 559.52 | bwd_inner_microstep: 559.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:24:12,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.69 | bwd_microstep: 424.09 | bwd_inner_microstep: 423.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:24:13,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.70 | bwd_microstep: 448.64 | bwd_inner_microstep: 448.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:24:15,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.33 | optimizer_gradients: 14.06 | optimizer_step: 23.23
[2024-12-05 15:24:15,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.75 | bwd_microstep: 1798.08 | bwd_inner_microstep: 1785.95 | bwd_allreduce_microstep: 12.03 | step_microstep: 112.38
[2024-12-05 15:24:15,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4404.05 | bwd: 8917.06 | bwd_inner: 8903.31 | bwd_allreduce: 12.65 | step: 113.51

 92%|█████████▏| 352/384 [1:41:11<08:03, 15.12s/it]
                                                   
{'loss': 0.1456, 'grad_norm': 1.7725763320922852, 'learning_rate': 1.8147178055029577e-08, 'epoch': 1.83}

 92%|█████████▏| 352/384 [1:41:11<08:03, 15.12s/it][2024-12-05 15:24:16,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.58 | bwd_microstep: 581.08 | bwd_inner_microstep: 580.91 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 15:24:17,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.55 | bwd_microstep: 455.11 | bwd_inner_microstep: 454.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:24:18,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.34 | bwd_microstep: 432.94 | bwd_inner_microstep: 432.80 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:24:19,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.90 | bwd_microstep: 433.71 | bwd_inner_microstep: 433.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:24:20,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.50 | bwd_microstep: 564.50 | bwd_inner_microstep: 564.36 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:24:21,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.03 | bwd_microstep: 406.40 | bwd_inner_microstep: 406.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:24:21,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.71 | bwd_microstep: 469.00 | bwd_inner_microstep: 468.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:24:22,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.14 | bwd_microstep: 446.61 | bwd_inner_microstep: 446.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:23,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.15 | bwd_microstep: 429.48 | bwd_inner_microstep: 429.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:24,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.58 | bwd_microstep: 678.79 | bwd_inner_microstep: 678.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:24:25,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.71 | bwd_microstep: 426.14 | bwd_inner_microstep: 426.03 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:26,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.05 | bwd_microstep: 462.84 | bwd_inner_microstep: 462.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:27,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.56 | bwd_microstep: 621.16 | bwd_inner_microstep: 620.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:28,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.28 | bwd_microstep: 451.99 | bwd_inner_microstep: 451.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:29,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.81 | bwd_microstep: 555.60 | bwd_inner_microstep: 555.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:30,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.49 | optimizer_gradients: 14.03 | optimizer_step: 23.11
[2024-12-05 15:24:30,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.65 | bwd_microstep: 1302.59 | bwd_inner_microstep: 1290.46 | bwd_allreduce_microstep: 12.04 | step_microstep: 112.12
[2024-12-05 15:24:30,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4473.60 | bwd: 8718.15 | bwd_inner: 8704.16 | bwd_allreduce: 12.71 | step: 113.32

 92%|█████████▏| 353/384 [1:41:27<07:50, 15.18s/it]
                                                   
{'loss': 0.1041, 'grad_norm': 1.677262544631958, 'learning_rate': 1.7037086855465898e-08, 'epoch': 1.83}

 92%|█████████▏| 353/384 [1:41:27<07:50, 15.18s/it][2024-12-05 15:24:31,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 244.94 | bwd_microstep: 351.99 | bwd_inner_microstep: 351.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:32,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 344.24 | bwd_microstep: 664.79 | bwd_inner_microstep: 664.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:24:33,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 363.90 | bwd_microstep: 776.45 | bwd_inner_microstep: 776.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:34,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.92 | bwd_microstep: 568.26 | bwd_inner_microstep: 568.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:24:35,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.17 | bwd_microstep: 450.89 | bwd_inner_microstep: 450.75 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:24:36,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.95 | bwd_microstep: 584.36 | bwd_inner_microstep: 584.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:24:37,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.34 | bwd_microstep: 565.96 | bwd_inner_microstep: 565.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:24:38,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.33 | bwd_microstep: 403.87 | bwd_inner_microstep: 403.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:24:39,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.74 | bwd_microstep: 459.18 | bwd_inner_microstep: 459.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:24:39,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.76 | bwd_microstep: 477.86 | bwd_inner_microstep: 477.73 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:24:40,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.08 | bwd_microstep: 411.05 | bwd_inner_microstep: 410.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:24:41,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.51 | bwd_microstep: 361.43 | bwd_inner_microstep: 361.29 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:24:42,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.17 | bwd_microstep: 527.09 | bwd_inner_microstep: 526.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:24:43,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.84 | bwd_microstep: 435.89 | bwd_inner_microstep: 435.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:24:44,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.98 | bwd_microstep: 587.33 | bwd_inner_microstep: 587.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:46,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.33 | optimizer_gradients: 14.02 | optimizer_step: 23.13
[2024-12-05 15:24:46,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.68 | bwd_microstep: 1491.98 | bwd_inner_microstep: 1479.75 | bwd_allreduce_microstep: 12.13 | step_microstep: 113.03
[2024-12-05 15:24:46,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4526.63 | bwd: 9118.37 | bwd_inner: 9104.44 | bwd_allreduce: 12.80 | step: 114.20

 92%|█████████▏| 354/384 [1:41:42<07:36, 15.22s/it]
                                                   
{'loss': 0.1149, 'grad_norm': 1.5990703105926514, 'learning_rate': 1.596144056689791e-08, 'epoch': 1.84}

 92%|█████████▏| 354/384 [1:41:42<07:36, 15.22s/it][2024-12-05 15:24:46,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.67 | bwd_microstep: 432.80 | bwd_inner_microstep: 432.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:47,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 227.51 | bwd_microstep: 334.03 | bwd_inner_microstep: 333.86 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 15:24:48,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.38 | bwd_microstep: 407.59 | bwd_inner_microstep: 407.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:49,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.13 | bwd_microstep: 430.08 | bwd_inner_microstep: 429.95 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:24:49,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.95 | bwd_microstep: 415.22 | bwd_inner_microstep: 415.07 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:24:50,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.52 | bwd_microstep: 493.58 | bwd_inner_microstep: 493.16 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.19
[2024-12-05 15:24:51,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.57 | bwd_microstep: 429.86 | bwd_inner_microstep: 429.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:24:52,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.95 | bwd_microstep: 451.46 | bwd_inner_microstep: 451.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:53,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.03 | bwd_microstep: 439.65 | bwd_inner_microstep: 439.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:53,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 240.61 | bwd_microstep: 374.29 | bwd_inner_microstep: 374.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:24:54,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.41 | bwd_microstep: 492.36 | bwd_inner_microstep: 492.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:24:55,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 229.87 | bwd_microstep: 347.56 | bwd_inner_microstep: 347.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:24:56,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.97 | bwd_microstep: 450.92 | bwd_inner_microstep: 450.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:57,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.31 | bwd_microstep: 558.18 | bwd_inner_microstep: 558.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:24:57,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.64 | bwd_microstep: 433.32 | bwd_inner_microstep: 433.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:25:01,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.49 | optimizer_gradients: 14.07 | optimizer_step: 23.29
[2024-12-05 15:25:01,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.20 | bwd_microstep: 3274.10 | bwd_inner_microstep: 3261.95 | bwd_allreduce_microstep: 12.05 | step_microstep: 113.69
[2024-12-05 15:25:01,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4178.78 | bwd: 9765.02 | bwd_inner: 9750.82 | bwd_allreduce: 12.83 | step: 115.01

 92%|█████████▏| 355/384 [1:41:57<07:23, 15.29s/it]
                                                   
{'loss': 0.1375, 'grad_norm': 2.083341598510742, 'learning_rate': 1.4920315904410064e-08, 'epoch': 1.85}

 92%|█████████▏| 355/384 [1:41:57<07:23, 15.29s/it][2024-12-05 15:25:02,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.29 | bwd_microstep: 350.18 | bwd_inner_microstep: 350.04 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:25:03,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.31 | bwd_microstep: 448.45 | bwd_inner_microstep: 448.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:03,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.54 | bwd_microstep: 414.44 | bwd_inner_microstep: 414.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:25:04,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.88 | bwd_microstep: 432.15 | bwd_inner_microstep: 432.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:25:05,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.02 | bwd_microstep: 435.01 | bwd_inner_microstep: 434.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:25:06,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.35 | bwd_microstep: 438.76 | bwd_inner_microstep: 438.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:25:07,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.77 | bwd_microstep: 458.24 | bwd_inner_microstep: 458.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:25:07,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.00 | bwd_microstep: 428.48 | bwd_inner_microstep: 428.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:25:08,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.09 | bwd_microstep: 482.55 | bwd_inner_microstep: 482.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:25:10,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 393.49 | bwd_microstep: 870.41 | bwd_inner_microstep: 870.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:10,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.08 | bwd_microstep: 403.42 | bwd_inner_microstep: 403.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:25:12,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 361.86 | bwd_microstep: 731.58 | bwd_inner_microstep: 731.44 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:25:12,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.77 | bwd_microstep: 452.61 | bwd_inner_microstep: 452.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:25:13,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.00 | bwd_microstep: 468.02 | bwd_inner_microstep: 467.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:14,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.28 | bwd_microstep: 544.27 | bwd_inner_microstep: 544.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:16,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.58 | optimizer_gradients: 14.05 | optimizer_step: 23.14
[2024-12-05 15:25:16,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.43 | bwd_microstep: 1408.26 | bwd_inner_microstep: 1396.17 | bwd_allreduce_microstep: 12.02 | step_microstep: 112.27
[2024-12-05 15:25:16,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4467.21 | bwd: 8766.84 | bwd_inner: 8753.09 | bwd_allreduce: 12.66 | step: 113.41

 93%|█████████▎| 356/384 [1:42:12<07:04, 15.15s/it]
                                                   
{'loss': 0.1365, 'grad_norm': 1.7379205226898193, 'learning_rate': 1.3913787121004716e-08, 'epoch': 1.85}

 93%|█████████▎| 356/384 [1:42:12<07:04, 15.15s/it][2024-12-05 15:25:17,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.39 | bwd_microstep: 406.71 | bwd_inner_microstep: 406.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:25:18,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 345.20 | bwd_microstep: 713.67 | bwd_inner_microstep: 713.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:19,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.96 | bwd_microstep: 436.31 | bwd_inner_microstep: 436.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:19,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.70 | bwd_microstep: 417.06 | bwd_inner_microstep: 416.92 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:25:20,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.48 | bwd_microstep: 362.97 | bwd_inner_microstep: 362.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:25:21,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.64 | bwd_microstep: 441.15 | bwd_inner_microstep: 441.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:22,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.96 | bwd_microstep: 409.02 | bwd_inner_microstep: 408.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:25:22,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.59 | bwd_microstep: 449.49 | bwd_inner_microstep: 449.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:25:23,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.78 | bwd_microstep: 437.06 | bwd_inner_microstep: 436.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:25:24,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.71 | bwd_microstep: 480.86 | bwd_inner_microstep: 480.72 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:25:25,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 346.97 | bwd_microstep: 720.28 | bwd_inner_microstep: 720.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:26,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.48 | bwd_microstep: 433.47 | bwd_inner_microstep: 433.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:27,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.71 | bwd_microstep: 438.07 | bwd_inner_microstep: 437.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:25:28,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.15 | bwd_microstep: 688.33 | bwd_inner_microstep: 688.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:29,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.10 | bwd_microstep: 406.26 | bwd_inner_microstep: 406.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:31,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.80 | optimizer_gradients: 14.07 | optimizer_step: 23.12
[2024-12-05 15:25:31,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 227.02 | bwd_microstep: 2209.15 | bwd_inner_microstep: 2197.01 | bwd_allreduce_microstep: 12.07 | step_microstep: 112.82
[2024-12-05 15:25:31,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4347.90 | bwd: 9449.86 | bwd_inner: 9436.06 | bwd_allreduce: 12.69 | step: 113.97

 93%|█████████▎| 357/384 [1:42:28<06:50, 15.21s/it]
                                                   
{'loss': 0.1348, 'grad_norm': 1.6175756454467773, 'learning_rate': 1.2941926002306536e-08, 'epoch': 1.86}

 93%|█████████▎| 357/384 [1:42:28<06:50, 15.21s/it][2024-12-05 15:25:32,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.51 | bwd_microstep: 360.57 | bwd_inner_microstep: 360.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:33,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.73 | bwd_microstep: 636.87 | bwd_inner_microstep: 636.72 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:25:34,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 242.99 | bwd_microstep: 373.73 | bwd_inner_microstep: 373.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:25:35,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.41 | bwd_microstep: 410.41 | bwd_inner_microstep: 410.27 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:25:36,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.41 | bwd_microstep: 587.17 | bwd_inner_microstep: 587.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:25:36,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.34 | bwd_microstep: 481.32 | bwd_inner_microstep: 481.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:37,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.24 | bwd_microstep: 433.59 | bwd_inner_microstep: 433.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:25:38,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.59 | bwd_microstep: 486.07 | bwd_inner_microstep: 485.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:25:39,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.27 | bwd_microstep: 480.85 | bwd_inner_microstep: 480.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:40,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 397.64 | bwd_microstep: 540.55 | bwd_inner_microstep: 540.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:41,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.39 | bwd_microstep: 483.33 | bwd_inner_microstep: 483.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:25:42,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.73 | bwd_microstep: 543.08 | bwd_inner_microstep: 542.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:43,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.63 | bwd_microstep: 456.35 | bwd_inner_microstep: 456.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:44,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.38 | bwd_microstep: 688.57 | bwd_inner_microstep: 688.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:45,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 244.51 | bwd_microstep: 398.04 | bwd_inner_microstep: 397.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:46,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.24 | optimizer_gradients: 14.03 | optimizer_step: 23.17
[2024-12-05 15:25:46,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.27 | bwd_microstep: 1096.82 | bwd_inner_microstep: 1084.55 | bwd_allreduce_microstep: 12.16 | step_microstep: 112.11
[2024-12-05 15:25:46,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4588.11 | bwd: 8457.31 | bwd_inner: 8443.43 | bwd_allreduce: 12.82 | step: 113.28

 93%|█████████▎| 358/384 [1:42:42<06:33, 15.12s/it]
                                                   
{'loss': 0.1232, 'grad_norm': 1.7680426836013794, 'learning_rate': 1.200480186144237e-08, 'epoch': 1.86}

 93%|█████████▎| 358/384 [1:42:42<06:33, 15.12s/it][2024-12-05 15:25:47,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.38 | bwd_microstep: 374.41 | bwd_inner_microstep: 374.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:25:48,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.27 | bwd_microstep: 406.65 | bwd_inner_microstep: 406.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:25:49,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.94 | bwd_microstep: 485.46 | bwd_inner_microstep: 485.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:49,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 229.32 | bwd_microstep: 328.93 | bwd_inner_microstep: 328.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:50,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.04 | bwd_microstep: 454.38 | bwd_inner_microstep: 454.25 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:25:51,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.66 | bwd_microstep: 354.49 | bwd_inner_microstep: 354.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.28
[2024-12-05 15:25:52,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.01 | bwd_microstep: 680.06 | bwd_inner_microstep: 679.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:25:53,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.15 | bwd_microstep: 433.24 | bwd_inner_microstep: 433.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:25:53,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.30 | bwd_microstep: 425.15 | bwd_inner_microstep: 425.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:25:54,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.15 | bwd_microstep: 394.04 | bwd_inner_microstep: 393.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:25:55,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.45 | bwd_microstep: 427.30 | bwd_inner_microstep: 427.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:25:56,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.68 | bwd_microstep: 674.14 | bwd_inner_microstep: 674.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:25:57,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.45 | bwd_microstep: 676.19 | bwd_inner_microstep: 676.05 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:25:58,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 241.15 | bwd_microstep: 370.96 | bwd_inner_microstep: 370.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:25:59,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.92 | bwd_microstep: 493.78 | bwd_inner_microstep: 493.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:01,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.58 | optimizer_gradients: 14.14 | optimizer_step: 23.99
[2024-12-05 15:26:01,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.18 | bwd_microstep: 2010.27 | bwd_inner_microstep: 1997.99 | bwd_allreduce_microstep: 12.18 | step_microstep: 112.53
[2024-12-05 15:26:01,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4364.10 | bwd: 8989.44 | bwd_inner: 8975.55 | bwd_allreduce: 12.81 | step: 113.83

 93%|█████████▎| 359/384 [1:42:58<06:18, 15.15s/it]
                                                   
{'loss': 0.0966, 'grad_norm': 1.3798372745513916, 'learning_rate': 1.1102481534098374e-08, 'epoch': 1.87}

 93%|█████████▎| 359/384 [1:42:58<06:18, 15.15s/it][2024-12-05 15:26:02,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.70 | bwd_microstep: 402.50 | bwd_inner_microstep: 402.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:26:03,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.53 | bwd_microstep: 454.47 | bwd_inner_microstep: 454.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:26:04,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.74 | bwd_microstep: 447.56 | bwd_inner_microstep: 447.42 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:26:05,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.08 | bwd_microstep: 589.64 | bwd_inner_microstep: 589.50 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:26:06,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.87 | bwd_microstep: 590.67 | bwd_inner_microstep: 590.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:26:07,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.05 | bwd_microstep: 435.40 | bwd_inner_microstep: 435.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:26:08,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.05 | bwd_microstep: 561.00 | bwd_inner_microstep: 560.77 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.08
[2024-12-05 15:26:09,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 332.60 | bwd_microstep: 672.52 | bwd_inner_microstep: 672.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:09,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 226.23 | bwd_microstep: 336.96 | bwd_inner_microstep: 336.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:10,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.49 | bwd_microstep: 449.27 | bwd_inner_microstep: 449.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:26:11,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.72 | bwd_microstep: 476.64 | bwd_inner_microstep: 476.46 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:26:12,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 380.58 | bwd_microstep: 841.90 | bwd_inner_microstep: 841.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:13,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.52 | bwd_microstep: 355.38 | bwd_inner_microstep: 355.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:14,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.55 | bwd_microstep: 433.81 | bwd_inner_microstep: 433.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:26:15,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.21 | bwd_microstep: 674.87 | bwd_inner_microstep: 674.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:17,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.17 | optimizer_gradients: 14.04 | optimizer_step: 23.35
[2024-12-05 15:26:17,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.80 | bwd_microstep: 1657.60 | bwd_inner_microstep: 1645.45 | bwd_allreduce_microstep: 12.06 | step_microstep: 113.94
[2024-12-05 15:26:17,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4582.77 | bwd: 9380.18 | bwd_inner: 9366.25 | bwd_allreduce: 12.70 | step: 115.07

 94%|█████████▍| 360/384 [1:43:13<06:06, 15.27s/it]
                                                   
{'loss': 0.1082, 'grad_norm': 1.584373950958252, 'learning_rate': 1.0235029373752757e-08, 'epoch': 1.87}

 94%|█████████▍| 360/384 [1:43:13<06:06, 15.27s/it][2024-12-05 15:26:18,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.75 | bwd_microstep: 550.83 | bwd_inner_microstep: 550.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:19,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.15 | bwd_microstep: 431.01 | bwd_inner_microstep: 430.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:26:20,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.51 | bwd_microstep: 444.11 | bwd_inner_microstep: 443.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:26:20,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.19 | bwd_microstep: 501.44 | bwd_inner_microstep: 501.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:22,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 378.42 | bwd_microstep: 834.55 | bwd_inner_microstep: 834.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:23,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.83 | bwd_microstep: 448.97 | bwd_inner_microstep: 448.79 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07
[2024-12-05 15:26:23,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.22 | bwd_microstep: 446.06 | bwd_inner_microstep: 445.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:26:24,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 241.46 | bwd_microstep: 369.88 | bwd_inner_microstep: 369.74 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:26:25,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.01 | bwd_microstep: 685.67 | bwd_inner_microstep: 685.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:26:26,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.17 | bwd_microstep: 449.63 | bwd_inner_microstep: 449.49 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07
[2024-12-05 15:26:27,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.80 | bwd_microstep: 412.00 | bwd_inner_microstep: 411.86 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:26:28,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.71 | bwd_microstep: 477.98 | bwd_inner_microstep: 477.83 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:26:29,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.91 | bwd_microstep: 537.74 | bwd_inner_microstep: 537.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:29,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.21 | bwd_microstep: 456.76 | bwd_inner_microstep: 456.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:30,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.24 | bwd_microstep: 404.00 | bwd_inner_microstep: 403.84 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 15:26:33,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.70 | optimizer_gradients: 14.08 | optimizer_step: 23.08
[2024-12-05 15:26:33,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.05 | bwd_microstep: 1928.57 | bwd_inner_microstep: 1916.46 | bwd_allreduce_microstep: 12.00 | step_microstep: 114.32
[2024-12-05 15:26:33,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4449.69 | bwd: 9379.19 | bwd_inner: 9365.33 | bwd_allreduce: 12.69 | step: 115.48

 94%|█████████▍| 361/384 [1:43:29<05:53, 15.36s/it]
                                                   
{'loss': 0.1348, 'grad_norm': 1.7235833406448364, 'learning_rate': 9.402507247086578e-09, 'epoch': 1.88}

 94%|█████████▍| 361/384 [1:43:29<05:53, 15.36s/it][2024-12-05 15:26:33,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.03 | bwd_microstep: 324.50 | bwd_inner_microstep: 324.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:34,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 273.87 | bwd_microstep: 477.20 | bwd_inner_microstep: 477.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:26:35,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.27 | bwd_microstep: 479.30 | bwd_inner_microstep: 479.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:36,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.32 | bwd_microstep: 480.46 | bwd_inner_microstep: 480.35 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:26:37,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.31 | bwd_microstep: 537.15 | bwd_inner_microstep: 537.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:26:38,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.47 | bwd_microstep: 555.11 | bwd_inner_microstep: 554.97 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:26:38,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 225.11 | bwd_microstep: 331.78 | bwd_inner_microstep: 331.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:26:39,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.83 | bwd_microstep: 445.11 | bwd_inner_microstep: 444.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:40,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.07 | bwd_microstep: 391.28 | bwd_inner_microstep: 391.12 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:26:41,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.04 | bwd_microstep: 413.70 | bwd_inner_microstep: 413.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:26:42,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.80 | bwd_microstep: 550.79 | bwd_inner_microstep: 550.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:26:42,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.91 | bwd_microstep: 435.60 | bwd_inner_microstep: 435.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:26:43,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.69 | bwd_microstep: 356.71 | bwd_inner_microstep: 356.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:26:44,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.75 | bwd_microstep: 423.63 | bwd_inner_microstep: 423.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:26:45,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.85 | bwd_microstep: 435.27 | bwd_inner_microstep: 435.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:26:49,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.50 | optimizer_gradients: 14.02 | optimizer_step: 23.13
[2024-12-05 15:26:49,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.07 | bwd_microstep: 3920.10 | bwd_inner_microstep: 3908.13 | bwd_allreduce_microstep: 11.88 | step_microstep: 112.13
[2024-12-05 15:26:49,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4264.46 | bwd: 10557.69 | bwd_inner: 10544.04 | bwd_allreduce: 12.52 | step: 113.21

 94%|█████████▍| 362/384 [1:43:45<05:45, 15.69s/it]
                                                   
{'loss': 0.1234, 'grad_norm': 1.6077182292938232, 'learning_rate': 8.60497452957104e-09, 'epoch': 1.88}

 94%|█████████▍| 362/384 [1:43:45<05:45, 15.69s/it][2024-12-05 15:26:50,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.09 | bwd_microstep: 416.79 | bwd_inner_microstep: 416.64 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:26:51,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.86 | bwd_microstep: 448.45 | bwd_inner_microstep: 448.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:26:51,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.55 | bwd_microstep: 405.99 | bwd_inner_microstep: 405.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:26:52,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.85 | bwd_microstep: 427.03 | bwd_inner_microstep: 426.86 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 15:26:53,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.41 | bwd_microstep: 453.76 | bwd_inner_microstep: 453.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:54,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.60 | bwd_microstep: 409.05 | bwd_inner_microstep: 408.91 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:26:55,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.75 | bwd_microstep: 674.26 | bwd_inner_microstep: 674.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:56,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.95 | bwd_microstep: 449.47 | bwd_inner_microstep: 449.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:57,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.03 | bwd_microstep: 439.63 | bwd_inner_microstep: 439.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:57,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.83 | bwd_microstep: 434.37 | bwd_inner_microstep: 434.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:58,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.44 | bwd_microstep: 552.12 | bwd_inner_microstep: 552.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:26:59,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.44 | bwd_microstep: 413.23 | bwd_inner_microstep: 413.05 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 15:27:00,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.37 | bwd_microstep: 385.56 | bwd_inner_microstep: 385.42 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:27:01,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.75 | bwd_microstep: 479.79 | bwd_inner_microstep: 479.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:27:01,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.65 | bwd_microstep: 362.74 | bwd_inner_microstep: 362.60 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:27:05,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.85 | optimizer_gradients: 14.05 | optimizer_step: 23.15
[2024-12-05 15:27:05,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.97 | bwd_microstep: 2697.93 | bwd_inner_microstep: 2685.73 | bwd_allreduce_microstep: 12.10 | step_microstep: 112.75
[2024-12-05 15:27:05,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4342.65 | bwd: 9450.16 | bwd_inner: 9436.01 | bwd_allreduce: 12.77 | step: 113.94

 95%|█████████▍| 363/384 [1:44:01<05:28, 15.65s/it]
                                                   
{'loss': 0.1177, 'grad_norm': 1.5776941776275635, 'learning_rate': 7.842488101232891e-09, 'epoch': 1.89}

 95%|█████████▍| 363/384 [1:44:01<05:28, 15.65s/it][2024-12-05 15:27:06,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.10 | bwd_microstep: 553.97 | bwd_inner_microstep: 553.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:06,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.18 | bwd_microstep: 539.33 | bwd_inner_microstep: 539.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:27:07,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.37 | bwd_microstep: 538.98 | bwd_inner_microstep: 538.84 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:27:08,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.19 | bwd_microstep: 408.32 | bwd_inner_microstep: 408.18 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:27:09,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.60 | bwd_microstep: 448.31 | bwd_inner_microstep: 448.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:27:10,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 354.81 | bwd_microstep: 746.67 | bwd_inner_microstep: 746.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:11,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.60 | bwd_microstep: 446.65 | bwd_inner_microstep: 446.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:12,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.46 | bwd_microstep: 412.24 | bwd_inner_microstep: 412.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:13,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.46 | bwd_microstep: 437.07 | bwd_inner_microstep: 436.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:27:14,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.55 | bwd_microstep: 437.65 | bwd_inner_microstep: 437.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:14,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.16 | bwd_microstep: 461.19 | bwd_inner_microstep: 461.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:15,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.72 | bwd_microstep: 401.10 | bwd_inner_microstep: 400.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:16,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.47 | bwd_microstep: 455.50 | bwd_inner_microstep: 455.29 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07
[2024-12-05 15:27:17,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.93 | bwd_microstep: 537.14 | bwd_inner_microstep: 537.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:18,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.48 | bwd_microstep: 481.79 | bwd_inner_microstep: 481.66 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:27:19,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.46 | optimizer_gradients: 14.04 | optimizer_step: 23.12
[2024-12-05 15:27:19,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.23 | bwd_microstep: 1093.77 | bwd_inner_microstep: 1081.63 | bwd_allreduce_microstep: 12.03 | step_microstep: 112.13
[2024-12-05 15:27:19,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4437.36 | bwd: 8399.67 | bwd_inner: 8385.79 | bwd_allreduce: 12.71 | step: 113.28

 95%|█████████▍| 364/384 [1:44:15<05:06, 15.35s/it]
                                                   
{'loss': 0.1133, 'grad_norm': 1.5909714698791504, 'learning_rate': 7.115102342598101e-09, 'epoch': 1.89}

 95%|█████████▍| 364/384 [1:44:15<05:06, 15.35s/it][2024-12-05 15:27:20,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.77 | bwd_microstep: 394.44 | bwd_inner_microstep: 394.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:21,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.98 | bwd_microstep: 410.19 | bwd_inner_microstep: 410.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:22,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.50 | bwd_microstep: 493.57 | bwd_inner_microstep: 493.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:27:22,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.44 | bwd_microstep: 484.67 | bwd_inner_microstep: 484.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:23,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.43 | bwd_microstep: 400.93 | bwd_inner_microstep: 400.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:27:24,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.89 | bwd_microstep: 398.88 | bwd_inner_microstep: 398.57 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.08
[2024-12-05 15:27:25,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.32 | bwd_microstep: 469.44 | bwd_inner_microstep: 469.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:27:26,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.29 | bwd_microstep: 436.54 | bwd_inner_microstep: 436.38 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 15:27:26,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.56 | bwd_microstep: 449.32 | bwd_inner_microstep: 449.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:27,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.27 | bwd_microstep: 406.79 | bwd_inner_microstep: 406.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:28,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.47 | bwd_microstep: 448.13 | bwd_inner_microstep: 448.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:29,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.91 | bwd_microstep: 539.82 | bwd_inner_microstep: 539.68 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:27:30,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.87 | bwd_microstep: 431.66 | bwd_inner_microstep: 431.52 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:27:30,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.33 | bwd_microstep: 413.03 | bwd_inner_microstep: 412.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:31,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.60 | bwd_microstep: 435.80 | bwd_inner_microstep: 435.66 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:27:34,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.47 | optimizer_gradients: 14.03 | optimizer_step: 23.12
[2024-12-05 15:27:34,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.68 | bwd_microstep: 2208.87 | bwd_inner_microstep: 2196.66 | bwd_allreduce_microstep: 12.12 | step_microstep: 112.20
[2024-12-05 15:27:34,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4213.36 | bwd: 8822.08 | bwd_inner: 8808.06 | bwd_allreduce: 12.94 | step: 113.43

 95%|█████████▌| 365/384 [1:44:30<04:47, 15.12s/it]
                                                   
{'loss': 0.1269, 'grad_norm': 1.7370619773864746, 'learning_rate': 6.422869130812913e-09, 'epoch': 1.9}

 95%|█████████▌| 365/384 [1:44:30<04:47, 15.12s/it][2024-12-05 15:27:35,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.56 | bwd_microstep: 587.21 | bwd_inner_microstep: 587.02 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08
[2024-12-05 15:27:36,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 331.71 | bwd_microstep: 673.09 | bwd_inner_microstep: 672.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:27:37,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.96 | bwd_microstep: 542.25 | bwd_inner_microstep: 542.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:27:38,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.74 | bwd_microstep: 541.00 | bwd_inner_microstep: 540.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:39,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.73 | bwd_microstep: 489.24 | bwd_inner_microstep: 489.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:40,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.53 | bwd_microstep: 567.10 | bwd_inner_microstep: 566.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:40,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.69 | bwd_microstep: 395.57 | bwd_inner_microstep: 395.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:41,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.52 | bwd_microstep: 436.23 | bwd_inner_microstep: 436.04 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.14
[2024-12-05 15:27:42,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.40 | bwd_microstep: 404.78 | bwd_inner_microstep: 404.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:43,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.02 | bwd_microstep: 454.15 | bwd_inner_microstep: 454.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:43,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.53 | bwd_microstep: 450.70 | bwd_inner_microstep: 450.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:44,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.83 | bwd_microstep: 435.74 | bwd_inner_microstep: 435.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:45,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 367.52 | bwd_microstep: 616.72 | bwd_inner_microstep: 616.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:46,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.45 | bwd_microstep: 571.82 | bwd_inner_microstep: 571.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:47,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.97 | bwd_microstep: 436.68 | bwd_inner_microstep: 436.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:27:49,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.55 | optimizer_gradients: 14.00 | optimizer_step: 24.07
[2024-12-05 15:27:49,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.12 | bwd_microstep: 1481.51 | bwd_inner_microstep: 1469.26 | bwd_allreduce_microstep: 12.16 | step_microstep: 112.09
[2024-12-05 15:27:49,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4639.35 | bwd: 9083.80 | bwd_inner: 9069.72 | bwd_allreduce: 12.89 | step: 113.36

 95%|█████████▌| 366/384 [1:44:45<04:32, 15.13s/it]
                                                   
{'loss': 0.1307, 'grad_norm': 1.7405683994293213, 'learning_rate': 5.765837835944309e-09, 'epoch': 1.9}

 95%|█████████▌| 366/384 [1:44:45<04:32, 15.13s/it][2024-12-05 15:27:50,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.46 | bwd_microstep: 402.66 | bwd_inner_microstep: 402.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:50,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.74 | bwd_microstep: 440.46 | bwd_inner_microstep: 440.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:27:51,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.23 | bwd_microstep: 482.13 | bwd_inner_microstep: 481.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:27:52,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.42 | bwd_microstep: 430.63 | bwd_inner_microstep: 430.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:27:53,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.11 | bwd_microstep: 430.92 | bwd_inner_microstep: 430.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:27:54,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.00 | bwd_microstep: 448.70 | bwd_inner_microstep: 448.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:55,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.88 | bwd_microstep: 518.61 | bwd_inner_microstep: 518.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:56,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 316.99 | bwd_microstep: 611.51 | bwd_inner_microstep: 611.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:57,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.84 | bwd_microstep: 427.98 | bwd_inner_microstep: 427.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:27:57,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.12 | bwd_microstep: 411.40 | bwd_inner_microstep: 411.21 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11
[2024-12-05 15:27:58,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.06 | bwd_microstep: 382.86 | bwd_inner_microstep: 382.68 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10
[2024-12-05 15:27:59,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 379.26 | bwd_microstep: 741.64 | bwd_inner_microstep: 741.47 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 15:28:00,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.89 | bwd_microstep: 469.68 | bwd_inner_microstep: 469.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:01,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.09 | bwd_microstep: 412.88 | bwd_inner_microstep: 412.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:28:02,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.14 | bwd_microstep: 447.08 | bwd_inner_microstep: 446.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:04,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.21 | optimizer_gradients: 14.03 | optimizer_step: 23.38
[2024-12-05 15:28:04,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.53 | bwd_microstep: 1222.64 | bwd_inner_microstep: 1210.47 | bwd_allreduce_microstep: 12.09 | step_microstep: 111.97
[2024-12-05 15:28:04,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4429.81 | bwd: 8281.78 | bwd_inner: 8267.85 | bwd_allreduce: 12.77 | step: 113.17

 96%|█████████▌| 367/384 [1:45:00<04:14, 14.96s/it]
                                                   
{'loss': 0.1546, 'grad_norm': 1.9631272554397583, 'learning_rate': 5.144055317458817e-09, 'epoch': 1.91}

 96%|█████████▌| 367/384 [1:45:00<04:14, 14.96s/it][2024-12-05 15:28:04,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.45 | bwd_microstep: 585.18 | bwd_inner_microstep: 585.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:05,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.09 | bwd_microstep: 540.27 | bwd_inner_microstep: 540.13 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 15:28:07,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.14 | bwd_microstep: 687.09 | bwd_inner_microstep: 686.95 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:28:07,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.73 | bwd_microstep: 459.34 | bwd_inner_microstep: 459.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:28:08,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.07 | bwd_microstep: 433.13 | bwd_inner_microstep: 433.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:09,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 331.94 | bwd_microstep: 674.28 | bwd_inner_microstep: 674.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:10,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.89 | bwd_microstep: 446.39 | bwd_inner_microstep: 446.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:28:11,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.00 | bwd_microstep: 449.10 | bwd_inner_microstep: 448.97 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:28:12,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.52 | bwd_microstep: 692.59 | bwd_inner_microstep: 692.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:13,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.03 | bwd_microstep: 561.50 | bwd_inner_microstep: 561.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:14,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 345.78 | bwd_microstep: 708.93 | bwd_inner_microstep: 708.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:15,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.50 | bwd_microstep: 482.33 | bwd_inner_microstep: 482.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:16,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.45 | bwd_microstep: 687.23 | bwd_inner_microstep: 687.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:17,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.43 | bwd_microstep: 413.13 | bwd_inner_microstep: 413.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:18,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.86 | bwd_microstep: 525.72 | bwd_inner_microstep: 525.58 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:28:19,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.64 | optimizer_gradients: 14.01 | optimizer_step: 23.28
[2024-12-05 15:28:19,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.20 | bwd_microstep: 871.01 | bwd_inner_microstep: 858.94 | bwd_allreduce_microstep: 11.97 | step_microstep: 112.43
[2024-12-05 15:28:19,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4733.15 | bwd: 9217.24 | bwd_inner: 9203.47 | bwd_allreduce: 12.54 | step: 113.61

 96%|█████████▌| 368/384 [1:45:15<04:01, 15.11s/it]
                                                   
{'loss': 0.1249, 'grad_norm': 1.6291249990463257, 'learning_rate': 4.55756592088058e-09, 'epoch': 1.91}

 96%|█████████▌| 368/384 [1:45:15<04:01, 15.11s/it][2024-12-05 15:28:20,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.09 | bwd_microstep: 511.50 | bwd_inner_microstep: 511.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:28:21,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.13 | bwd_microstep: 485.57 | bwd_inner_microstep: 485.22 | bwd_allreduce_microstep: 0.27 | step_microstep: 0.07
[2024-12-05 15:28:22,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.99 | bwd_microstep: 441.63 | bwd_inner_microstep: 441.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:28:22,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.61 | bwd_microstep: 414.56 | bwd_inner_microstep: 414.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:28:23,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.15 | bwd_microstep: 673.93 | bwd_inner_microstep: 673.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:28:24,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 284.18 | bwd_microstep: 477.33 | bwd_inner_microstep: 477.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:28:25,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.79 | bwd_microstep: 438.76 | bwd_inner_microstep: 438.62 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.06
[2024-12-05 15:28:26,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.24 | bwd_microstep: 407.01 | bwd_inner_microstep: 406.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:27,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.36 | bwd_microstep: 456.35 | bwd_inner_microstep: 456.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:28:27,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.23 | bwd_microstep: 436.81 | bwd_inner_microstep: 436.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:28,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.04 | bwd_microstep: 532.85 | bwd_inner_microstep: 532.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:28:29,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.40 | bwd_microstep: 416.04 | bwd_inner_microstep: 415.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:28:30,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.51 | bwd_microstep: 478.49 | bwd_inner_microstep: 478.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:28:31,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.84 | bwd_microstep: 567.11 | bwd_inner_microstep: 566.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:28:32,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.02 | bwd_microstep: 688.37 | bwd_inner_microstep: 688.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:28:34,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.15 | optimizer_gradients: 14.13 | optimizer_step: 23.10
[2024-12-05 15:28:34,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.27 | bwd_microstep: 1738.24 | bwd_inner_microstep: 1726.04 | bwd_allreduce_microstep: 12.10 | step_microstep: 112.05
[2024-12-05 15:28:34,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4500.93 | bwd: 9164.53 | bwd_inner: 9150.43 | bwd_allreduce: 12.96 | step: 113.09

 96%|█████████▌| 369/384 [1:45:31<03:47, 15.17s/it]
                                                   
{'loss': 0.1027, 'grad_norm': 1.528748869895935, 'learning_rate': 4.0064114746284905e-09, 'epoch': 1.92}

 96%|█████████▌| 369/384 [1:45:31<03:47, 15.17s/it][2024-12-05 15:28:35,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.80 | bwd_microstep: 408.93 | bwd_inner_microstep: 408.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:28:36,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 225.71 | bwd_microstep: 330.99 | bwd_inner_microstep: 330.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:36,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.41 | bwd_microstep: 364.57 | bwd_inner_microstep: 364.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:37,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.34 | bwd_microstep: 370.39 | bwd_inner_microstep: 370.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:38,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 327.29 | bwd_microstep: 621.34 | bwd_inner_microstep: 621.19 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:28:39,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.76 | bwd_microstep: 606.73 | bwd_inner_microstep: 606.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:28:40,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.36 | bwd_microstep: 547.13 | bwd_inner_microstep: 547.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:41,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.87 | bwd_microstep: 409.46 | bwd_inner_microstep: 409.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:28:41,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.91 | bwd_microstep: 359.68 | bwd_inner_microstep: 359.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:42,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.63 | bwd_microstep: 428.04 | bwd_inner_microstep: 427.89 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:28:43,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.06 | bwd_microstep: 594.13 | bwd_inner_microstep: 594.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:44,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.45 | bwd_microstep: 401.29 | bwd_inner_microstep: 401.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:28:45,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 305.51 | bwd_microstep: 585.80 | bwd_inner_microstep: 585.52 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.07
[2024-12-05 15:28:46,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.01 | bwd_microstep: 593.74 | bwd_inner_microstep: 593.61 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:47,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 368.34 | bwd_microstep: 783.24 | bwd_inner_microstep: 783.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:50,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.80 | optimizer_gradients: 14.02 | optimizer_step: 23.33
[2024-12-05 15:28:50,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.84 | bwd_microstep: 2267.09 | bwd_inner_microstep: 2254.89 | bwd_allreduce_microstep: 12.11 | step_microstep: 113.05
[2024-12-05 15:28:50,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4438.34 | bwd: 9672.55 | bwd_inner: 9658.56 | bwd_allreduce: 12.89 | step: 114.26

 96%|█████████▋| 370/384 [1:45:46<03:34, 15.34s/it]
                                                   
{'loss': 0.1128, 'grad_norm': 1.8311703205108643, 'learning_rate': 3.4906312870331965e-09, 'epoch': 1.92}

 96%|█████████▋| 370/384 [1:45:46<03:34, 15.34s/it][2024-12-05 15:28:51,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.53 | bwd_microstep: 453.71 | bwd_inner_microstep: 453.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:28:52,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.63 | bwd_microstep: 450.29 | bwd_inner_microstep: 450.15 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:28:52,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 248.14 | bwd_microstep: 403.34 | bwd_inner_microstep: 403.18 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:28:53,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 231.46 | bwd_microstep: 354.88 | bwd_inner_microstep: 354.74 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:28:54,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.93 | bwd_microstep: 448.13 | bwd_inner_microstep: 447.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:28:55,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.64 | bwd_microstep: 428.02 | bwd_inner_microstep: 427.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:28:55,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.20 | bwd_microstep: 488.40 | bwd_inner_microstep: 488.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:28:57,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 364.18 | bwd_microstep: 772.81 | bwd_inner_microstep: 772.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.18
[2024-12-05 15:28:58,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 355.06 | bwd_microstep: 745.72 | bwd_inner_microstep: 745.56 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08
[2024-12-05 15:28:59,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.48 | bwd_microstep: 556.94 | bwd_inner_microstep: 556.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:29:00,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.13 | bwd_microstep: 431.88 | bwd_inner_microstep: 431.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:29:00,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.11 | bwd_microstep: 434.34 | bwd_inner_microstep: 434.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:29:01,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 233.57 | bwd_microstep: 360.50 | bwd_inner_microstep: 360.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.10
[2024-12-05 15:29:02,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.34 | bwd_microstep: 619.40 | bwd_inner_microstep: 619.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.17
[2024-12-05 15:29:03,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 371.66 | bwd_microstep: 777.95 | bwd_inner_microstep: 777.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:05,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.01 | optimizer_gradients: 14.04 | optimizer_step: 23.30
[2024-12-05 15:29:05,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.54 | bwd_microstep: 887.52 | bwd_inner_microstep: 875.31 | bwd_allreduce_microstep: 12.11 | step_microstep: 111.81
[2024-12-05 15:29:05,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4556.67 | bwd: 8613.82 | bwd_inner: 8599.91 | bwd_allreduce: 12.80 | step: 113.16

 97%|█████████▋| 371/384 [1:46:01<03:17, 15.18s/it]
                                                   
{'loss': 0.1441, 'grad_norm': 1.8863190412521362, 'learning_rate': 3.010262143533393e-09, 'epoch': 1.93}

 97%|█████████▋| 371/384 [1:46:01<03:17, 15.18s/it][2024-12-05 15:29:06,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.24 | bwd_microstep: 525.34 | bwd_inner_microstep: 525.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:07,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.30 | bwd_microstep: 454.14 | bwd_inner_microstep: 453.95 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 15:29:07,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.91 | bwd_microstep: 434.43 | bwd_inner_microstep: 434.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:08,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.31 | bwd_microstep: 480.13 | bwd_inner_microstep: 480.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:29:09,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.98 | bwd_microstep: 591.19 | bwd_inner_microstep: 591.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:29:10,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 275.79 | bwd_microstep: 482.98 | bwd_inner_microstep: 482.85 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:11,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.78 | bwd_microstep: 429.37 | bwd_inner_microstep: 429.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:12,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.63 | bwd_microstep: 412.95 | bwd_inner_microstep: 412.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:12,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.34 | bwd_microstep: 405.84 | bwd_inner_microstep: 405.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:13,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.64 | bwd_microstep: 440.93 | bwd_inner_microstep: 440.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:29:14,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 285.25 | bwd_microstep: 519.02 | bwd_inner_microstep: 518.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:15,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.12 | bwd_microstep: 437.85 | bwd_inner_microstep: 437.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:29:16,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.00 | bwd_microstep: 445.36 | bwd_inner_microstep: 445.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:29:16,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.18 | bwd_microstep: 445.31 | bwd_inner_microstep: 445.17 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:29:17,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.03 | bwd_microstep: 457.64 | bwd_inner_microstep: 457.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:29:20,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.62 | optimizer_gradients: 14.05 | optimizer_step: 23.08
[2024-12-05 15:29:20,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.71 | bwd_microstep: 2312.98 | bwd_inner_microstep: 2300.58 | bwd_allreduce_microstep: 12.31 | step_microstep: 112.41
[2024-12-05 15:29:20,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4328.29 | bwd: 9275.45 | bwd_inner: 9261.40 | bwd_allreduce: 12.97 | step: 113.60

 97%|█████████▋| 372/384 [1:46:16<03:01, 15.16s/it]
                                                   
{'loss': 0.1198, 'grad_norm': 1.5785506963729858, 'learning_rate': 2.5653383040524224e-09, 'epoch': 1.93}

 97%|█████████▋| 372/384 [1:46:16<03:01, 15.16s/it][2024-12-05 15:29:21,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.69 | bwd_microstep: 398.82 | bwd_inner_microstep: 398.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:21,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.12 | bwd_microstep: 448.06 | bwd_inner_microstep: 447.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:29:22,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.55 | bwd_microstep: 431.01 | bwd_inner_microstep: 430.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:29:23,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.39 | bwd_microstep: 483.88 | bwd_inner_microstep: 483.75 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:24,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.85 | bwd_microstep: 431.00 | bwd_inner_microstep: 430.87 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:25,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.41 | bwd_microstep: 594.03 | bwd_inner_microstep: 593.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:26,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.97 | bwd_microstep: 426.14 | bwd_inner_microstep: 425.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:26,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.19 | bwd_microstep: 413.03 | bwd_inner_microstep: 412.87 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 15:29:27,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.60 | bwd_microstep: 410.90 | bwd_inner_microstep: 410.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:28,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.38 | bwd_microstep: 599.49 | bwd_inner_microstep: 599.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:29,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.41 | bwd_microstep: 585.90 | bwd_inner_microstep: 585.75 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:29:30,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.83 | bwd_microstep: 630.27 | bwd_inner_microstep: 630.13 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:29:31,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 355.61 | bwd_microstep: 760.54 | bwd_inner_microstep: 760.39 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:29:32,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.76 | bwd_microstep: 454.08 | bwd_inner_microstep: 453.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:29:33,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.55 | bwd_microstep: 396.44 | bwd_inner_microstep: 396.04 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.20
[2024-12-05 15:29:36,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.38 | optimizer_gradients: 14.05 | optimizer_step: 23.31
[2024-12-05 15:29:36,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.43 | bwd_microstep: 2357.97 | bwd_inner_microstep: 2345.80 | bwd_allreduce_microstep: 12.07 | step_microstep: 112.50
[2024-12-05 15:29:36,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4535.78 | bwd: 9821.61 | bwd_inner: 9807.31 | bwd_allreduce: 12.83 | step: 113.83

 97%|█████████▋| 373/384 [1:46:32<02:49, 15.42s/it]
                                                   
{'loss': 0.1224, 'grad_norm': 1.6531542539596558, 'learning_rate': 2.155891500554896e-09, 'epoch': 1.94}

 97%|█████████▋| 373/384 [1:46:32<02:49, 15.42s/it][2024-12-05 15:29:37,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.02 | bwd_microstep: 655.63 | bwd_inner_microstep: 655.49 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:29:38,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.01 | bwd_microstep: 585.85 | bwd_inner_microstep: 585.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:39,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.74 | bwd_microstep: 428.58 | bwd_inner_microstep: 428.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:40,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.90 | bwd_microstep: 474.06 | bwd_inner_microstep: 473.91 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.23
[2024-12-05 15:29:41,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.88 | bwd_microstep: 584.85 | bwd_inner_microstep: 584.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:42,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.31 | bwd_microstep: 545.50 | bwd_inner_microstep: 545.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:43,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 342.94 | bwd_microstep: 715.64 | bwd_inner_microstep: 715.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:44,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.89 | bwd_microstep: 483.26 | bwd_inner_microstep: 483.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:29:45,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.48 | bwd_microstep: 357.67 | bwd_inner_microstep: 357.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:46,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.96 | bwd_microstep: 543.26 | bwd_inner_microstep: 543.13 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:29:46,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.74 | bwd_microstep: 433.72 | bwd_inner_microstep: 433.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:47,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 306.37 | bwd_microstep: 581.08 | bwd_inner_microstep: 580.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:29:48,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.02 | bwd_microstep: 411.54 | bwd_inner_microstep: 411.43 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:49,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 346.88 | bwd_microstep: 718.01 | bwd_inner_microstep: 717.88 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:50,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 269.16 | bwd_microstep: 446.31 | bwd_inner_microstep: 446.18 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:52,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.29 | optimizer_gradients: 14.01 | optimizer_step: 23.15
[2024-12-05 15:29:52,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.36 | bwd_microstep: 991.74 | bwd_inner_microstep: 979.45 | bwd_allreduce_microstep: 12.17 | step_microstep: 111.97
[2024-12-05 15:29:52,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4710.73 | bwd: 8956.69 | bwd_inner: 8942.78 | bwd_allreduce: 12.83 | step: 113.31

 97%|█████████▋| 374/384 [1:46:48<02:35, 15.52s/it]
                                                   
{'loss': 0.1149, 'grad_norm': 1.5542500019073486, 'learning_rate': 1.7819509347835049e-09, 'epoch': 1.94}

 97%|█████████▋| 374/384 [1:46:48<02:35, 15.52s/it][2024-12-05 15:29:53,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 391.81 | bwd_microstep: 833.66 | bwd_inner_microstep: 833.39 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.08
[2024-12-05 15:29:54,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.50 | bwd_microstep: 434.14 | bwd_inner_microstep: 433.99 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:29:55,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.50 | bwd_microstep: 539.46 | bwd_inner_microstep: 539.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:29:56,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.02 | bwd_microstep: 449.99 | bwd_inner_microstep: 449.84 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:29:56,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.53 | bwd_microstep: 429.98 | bwd_inner_microstep: 429.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:29:58,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 343.27 | bwd_microstep: 708.83 | bwd_inner_microstep: 708.64 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:29:59,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.74 | bwd_microstep: 572.84 | bwd_inner_microstep: 572.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:30:00,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.29 | bwd_microstep: 532.53 | bwd_inner_microstep: 532.38 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.11
[2024-12-05 15:30:00,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.18 | bwd_microstep: 433.59 | bwd_inner_microstep: 433.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:30:01,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.79 | bwd_microstep: 584.71 | bwd_inner_microstep: 584.54 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 15:30:02,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.49 | bwd_microstep: 536.03 | bwd_inner_microstep: 535.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:30:03,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.98 | bwd_microstep: 408.37 | bwd_inner_microstep: 408.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:30:04,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.75 | bwd_microstep: 493.74 | bwd_inner_microstep: 493.59 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:30:05,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.23 | bwd_microstep: 698.25 | bwd_inner_microstep: 698.11 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10
[2024-12-05 15:30:06,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.14 | bwd_microstep: 570.73 | bwd_inner_microstep: 570.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:30:07,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.52 | optimizer_gradients: 14.03 | optimizer_step: 23.26
[2024-12-05 15:30:07,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 253.28 | bwd_microstep: 871.13 | bwd_inner_microstep: 858.92 | bwd_allreduce_microstep: 12.11 | step_microstep: 112.40
[2024-12-05 15:30:07,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4669.55 | bwd: 9097.98 | bwd_inner: 9083.80 | bwd_allreduce: 12.94 | step: 113.66

 98%|█████████▊| 375/384 [1:47:04<02:20, 15.58s/it]
                                                   
{'loss': 0.1061, 'grad_norm': 1.5082885026931763, 'learning_rate': 1.4435432761762955e-09, 'epoch': 1.95}

 98%|█████████▊| 375/384 [1:47:04<02:20, 15.58s/it][2024-12-05 15:30:08,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.37 | bwd_microstep: 396.41 | bwd_inner_microstep: 396.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:30:09,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.29 | bwd_microstep: 446.59 | bwd_inner_microstep: 446.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:30:10,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 344.85 | bwd_microstep: 709.36 | bwd_inner_microstep: 709.18 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 15:30:11,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.04 | bwd_microstep: 357.02 | bwd_inner_microstep: 356.83 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.34
[2024-12-05 15:30:12,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.62 | bwd_microstep: 411.32 | bwd_inner_microstep: 411.17 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:30:13,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.08 | bwd_microstep: 492.57 | bwd_inner_microstep: 492.44 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:30:13,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.40 | bwd_microstep: 433.09 | bwd_inner_microstep: 432.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:30:14,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.18 | bwd_microstep: 358.44 | bwd_inner_microstep: 358.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:30:15,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 252.58 | bwd_microstep: 410.67 | bwd_inner_microstep: 410.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:30:16,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.61 | bwd_microstep: 403.86 | bwd_inner_microstep: 403.73 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:30:17,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.53 | bwd_microstep: 560.86 | bwd_inner_microstep: 560.72 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:30:17,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.93 | bwd_microstep: 409.94 | bwd_inner_microstep: 409.81 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:30:18,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.70 | bwd_microstep: 428.68 | bwd_inner_microstep: 428.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:30:19,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 330.79 | bwd_microstep: 676.32 | bwd_inner_microstep: 676.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:30:20,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.83 | bwd_microstep: 451.78 | bwd_inner_microstep: 451.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:30:22,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.11 | optimizer_gradients: 14.03 | optimizer_step: 23.13
[2024-12-05 15:30:22,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.25 | bwd_microstep: 1643.38 | bwd_inner_microstep: 1631.37 | bwd_allreduce_microstep: 11.92 | step_microstep: 114.01
[2024-12-05 15:30:22,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4357.09 | bwd: 8590.29 | bwd_inner: 8576.56 | bwd_allreduce: 12.56 | step: 115.44

 98%|█████████▊| 376/384 [1:47:19<02:02, 15.37s/it]
                                                   
{'loss': 0.1327, 'grad_norm': 1.5856765508651733, 'learning_rate': 1.1406926599646372e-09, 'epoch': 1.95}

 98%|█████████▊| 376/384 [1:47:19<02:02, 15.37s/it][2024-12-05 15:30:23,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.06 | bwd_microstep: 398.52 | bwd_inner_microstep: 398.39 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:30:24,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.73 | bwd_microstep: 411.96 | bwd_inner_microstep: 411.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:30:25,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.90 | bwd_microstep: 450.20 | bwd_inner_microstep: 450.05 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08
[2024-12-05 15:30:25,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 251.81 | bwd_microstep: 413.21 | bwd_inner_microstep: 413.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:30:26,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.60 | bwd_microstep: 479.76 | bwd_inner_microstep: 479.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:30:27,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 257.72 | bwd_microstep: 417.04 | bwd_inner_microstep: 416.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:30:28,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.70 | bwd_microstep: 447.07 | bwd_inner_microstep: 446.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:30:29,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.94 | bwd_microstep: 537.33 | bwd_inner_microstep: 537.17 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:30:30,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.97 | bwd_microstep: 428.84 | bwd_inner_microstep: 428.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:30:30,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.21 | bwd_microstep: 438.97 | bwd_inner_microstep: 438.76 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.13
[2024-12-05 15:30:31,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.58 | bwd_microstep: 444.30 | bwd_inner_microstep: 444.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:30:32,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 336.83 | bwd_microstep: 683.65 | bwd_inner_microstep: 683.51 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:30:33,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.79 | bwd_microstep: 357.27 | bwd_inner_microstep: 357.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:30:34,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.44 | bwd_microstep: 612.95 | bwd_inner_microstep: 612.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:30:35,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.67 | bwd_microstep: 407.11 | bwd_inner_microstep: 407.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:30:38,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.97 | optimizer_gradients: 14.08 | optimizer_step: 23.36
[2024-12-05 15:30:38,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.25 | bwd_microstep: 2266.12 | bwd_inner_microstep: 2254.09 | bwd_allreduce_microstep: 11.93 | step_microstep: 111.97
[2024-12-05 15:30:38,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4325.28 | bwd: 9194.32 | bwd_inner: 9180.51 | bwd_allreduce: 12.61 | step: 113.13

 98%|█████████▊| 377/384 [1:47:34<01:47, 15.34s/it]
                                                   
{'loss': 0.1105, 'grad_norm': 1.6566141843795776, 'learning_rate': 8.73420685452042e-10, 'epoch': 1.96}

 98%|█████████▊| 377/384 [1:47:34<01:47, 15.34s/it][2024-12-05 15:30:38,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.67 | bwd_microstep: 453.34 | bwd_inner_microstep: 453.21 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:30:39,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 228.34 | bwd_microstep: 333.39 | bwd_inner_microstep: 333.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:30:40,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.62 | bwd_microstep: 487.66 | bwd_inner_microstep: 487.37 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:30:41,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.22 | bwd_microstep: 432.07 | bwd_inner_microstep: 431.94 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:30:42,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.59 | bwd_microstep: 401.78 | bwd_inner_microstep: 401.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:30:43,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 353.77 | bwd_microstep: 757.39 | bwd_inner_microstep: 757.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:30:44,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.43 | bwd_microstep: 432.42 | bwd_inner_microstep: 432.29 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:30:45,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 288.34 | bwd_microstep: 530.81 | bwd_inner_microstep: 530.67 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:30:45,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.96 | bwd_microstep: 437.69 | bwd_inner_microstep: 437.55 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:30:46,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.37 | bwd_microstep: 598.44 | bwd_inner_microstep: 598.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:30:47,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.87 | bwd_microstep: 449.81 | bwd_inner_microstep: 449.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:30:48,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.71 | bwd_microstep: 532.65 | bwd_inner_microstep: 532.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:30:49,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 280.38 | bwd_microstep: 483.49 | bwd_inner_microstep: 483.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.20
[2024-12-05 15:30:50,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 319.70 | bwd_microstep: 612.22 | bwd_inner_microstep: 612.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:30:51,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 317.66 | bwd_microstep: 612.37 | bwd_inner_microstep: 612.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:30:53,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.40 | optimizer_gradients: 14.03 | optimizer_step: 23.14
[2024-12-05 15:30:53,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 313.71 | bwd_microstep: 1376.83 | bwd_inner_microstep: 1364.83 | bwd_allreduce_microstep: 11.90 | step_microstep: 112.09
[2024-12-05 15:30:53,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4560.42 | bwd: 8932.36 | bwd_inner: 8918.54 | bwd_allreduce: 12.54 | step: 113.35

 98%|█████████▊| 378/384 [1:47:49<01:32, 15.34s/it]
                                                   
{'loss': 0.1301, 'grad_norm': 1.7047754526138306, 'learning_rate': 6.417464144736207e-10, 'epoch': 1.96}

 98%|█████████▊| 378/384 [1:47:49<01:32, 15.34s/it][2024-12-05 15:30:54,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.94 | bwd_microstep: 369.48 | bwd_inner_microstep: 369.36 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:30:54,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 256.22 | bwd_microstep: 403.98 | bwd_inner_microstep: 403.83 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:30:55,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.94 | bwd_microstep: 378.26 | bwd_inner_microstep: 378.13 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:30:56,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.50 | bwd_microstep: 399.68 | bwd_inner_microstep: 399.54 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:30:57,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.77 | bwd_microstep: 437.91 | bwd_inner_microstep: 437.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:30:57,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.97 | bwd_microstep: 430.66 | bwd_inner_microstep: 430.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:30:58,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.27 | bwd_microstep: 622.67 | bwd_inner_microstep: 622.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:30:59,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.16 | bwd_microstep: 345.83 | bwd_inner_microstep: 345.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:31:00,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.23 | bwd_microstep: 477.70 | bwd_inner_microstep: 477.57 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:31:01,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.08 | bwd_microstep: 445.13 | bwd_inner_microstep: 444.99 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:31:02,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 320.08 | bwd_microstep: 627.08 | bwd_inner_microstep: 626.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:31:03,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 367.67 | bwd_microstep: 772.69 | bwd_inner_microstep: 772.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:31:04,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.34 | bwd_microstep: 438.41 | bwd_inner_microstep: 438.28 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:31:05,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 281.80 | bwd_microstep: 459.35 | bwd_inner_microstep: 459.17 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.12
[2024-12-05 15:31:06,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.38 | bwd_microstep: 528.12 | bwd_inner_microstep: 527.98 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:31:08,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.15 | optimizer_gradients: 14.20 | optimizer_step: 23.29
[2024-12-05 15:31:08,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.71 | bwd_microstep: 1865.71 | bwd_inner_microstep: 1853.16 | bwd_allreduce_microstep: 12.42 | step_microstep: 114.78
[2024-12-05 15:31:08,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4428.12 | bwd: 9002.67 | bwd_inner: 8988.40 | bwd_allreduce: 13.09 | step: 115.79

 99%|█████████▊| 379/384 [1:48:04<01:16, 15.23s/it]
                                                   
{'loss': 0.1412, 'grad_norm': 1.7699244022369385, 'learning_rate': 4.4568637003633556e-10, 'epoch': 1.97}

 99%|█████████▊| 379/384 [1:48:04<01:16, 15.23s/it][2024-12-05 15:31:09,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.62 | bwd_microstep: 377.63 | bwd_inner_microstep: 377.51 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:31:09,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 229.76 | bwd_microstep: 333.40 | bwd_inner_microstep: 333.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:31:10,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.04 | bwd_microstep: 585.47 | bwd_inner_microstep: 585.34 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:31:11,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 250.77 | bwd_microstep: 403.45 | bwd_inner_microstep: 403.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:31:12,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.72 | bwd_microstep: 432.63 | bwd_inner_microstep: 432.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:31:13,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.23 | bwd_microstep: 435.18 | bwd_inner_microstep: 435.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:31:14,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.20 | bwd_microstep: 560.39 | bwd_inner_microstep: 560.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:31:14,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 262.43 | bwd_microstep: 403.26 | bwd_inner_microstep: 403.14 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:31:15,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 266.07 | bwd_microstep: 446.71 | bwd_inner_microstep: 446.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:31:16,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.56 | bwd_microstep: 554.95 | bwd_inner_microstep: 554.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:31:17,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.88 | bwd_microstep: 524.59 | bwd_inner_microstep: 524.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:31:18,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 235.59 | bwd_microstep: 356.83 | bwd_inner_microstep: 356.70 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:31:19,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.21 | bwd_microstep: 429.69 | bwd_inner_microstep: 429.55 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:31:19,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.83 | bwd_microstep: 400.72 | bwd_inner_microstep: 400.59 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:31:20,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 249.32 | bwd_microstep: 412.76 | bwd_inner_microstep: 412.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:31:24,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.13 | optimizer_gradients: 14.03 | optimizer_step: 23.13
[2024-12-05 15:31:24,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.93 | bwd_microstep: 3042.43 | bwd_inner_microstep: 3030.43 | bwd_allreduce_microstep: 11.92 | step_microstep: 111.66
[2024-12-05 15:31:24,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4241.23 | bwd: 9700.08 | bwd_inner: 9686.46 | bwd_allreduce: 12.55 | step: 112.79

 99%|█████████▉| 380/384 [1:48:20<01:01, 15.40s/it]
                                                   
{'loss': 0.1571, 'grad_norm': 1.8177380561828613, 'learning_rate': 2.852545351409996e-10, 'epoch': 1.98}

 99%|█████████▉| 380/384 [1:48:20<01:01, 15.40s/it][2024-12-05 15:31:24,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 245.08 | bwd_microstep: 343.92 | bwd_inner_microstep: 343.79 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:31:25,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 255.94 | bwd_microstep: 399.62 | bwd_inner_microstep: 399.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:31:26,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.78 | bwd_microstep: 564.40 | bwd_inner_microstep: 564.26 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:31:27,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 391.58 | bwd_microstep: 861.32 | bwd_inner_microstep: 861.19 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:31:28,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.05 | bwd_microstep: 428.21 | bwd_inner_microstep: 428.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:31:29,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.67 | bwd_microstep: 581.92 | bwd_inner_microstep: 581.78 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:31:30,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.61 | bwd_microstep: 356.10 | bwd_inner_microstep: 355.97 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:31:31,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.11 | bwd_microstep: 434.67 | bwd_inner_microstep: 434.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:31:32,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 278.51 | bwd_microstep: 482.46 | bwd_inner_microstep: 482.33 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:31:33,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.13 | bwd_microstep: 482.47 | bwd_inner_microstep: 482.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:31:34,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 334.23 | bwd_microstep: 640.66 | bwd_inner_microstep: 640.52 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:31:35,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.90 | bwd_microstep: 535.58 | bwd_inner_microstep: 535.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:31:35,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.84 | bwd_microstep: 526.63 | bwd_inner_microstep: 526.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:31:36,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.51 | bwd_microstep: 559.97 | bwd_inner_microstep: 559.84 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:31:37,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.84 | bwd_microstep: 587.84 | bwd_inner_microstep: 587.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:31:39,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.11 | optimizer_gradients: 14.00 | optimizer_step: 23.34
[2024-12-05 15:31:39,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 309.21 | bwd_microstep: 829.34 | bwd_inner_microstep: 817.34 | bwd_allreduce_microstep: 11.90 | step_microstep: 113.32
[2024-12-05 15:31:39,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4688.05 | bwd: 8615.12 | bwd_inner: 8601.43 | bwd_allreduce: 12.54 | step: 114.30

 99%|█████████▉| 381/384 [1:48:35<00:45, 15.29s/it]
                                                   
{'loss': 0.104, 'grad_norm': 1.5334748029708862, 'learning_rate': 1.6046235178474033e-10, 'epoch': 1.98}

 99%|█████████▉| 381/384 [1:48:35<00:45, 15.29s/it][2024-12-05 15:31:40,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 279.98 | bwd_microstep: 456.72 | bwd_inner_microstep: 456.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:31:40,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 259.61 | bwd_microstep: 424.63 | bwd_inner_microstep: 424.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:31:41,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.06 | bwd_microstep: 525.84 | bwd_inner_microstep: 525.69 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:31:42,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 315.52 | bwd_microstep: 612.06 | bwd_inner_microstep: 611.92 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:31:43,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.99 | bwd_microstep: 455.21 | bwd_inner_microstep: 455.06 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:31:44,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.71 | bwd_microstep: 450.07 | bwd_inner_microstep: 449.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:31:45,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.59 | bwd_microstep: 437.30 | bwd_inner_microstep: 437.16 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:31:46,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.99 | bwd_microstep: 552.22 | bwd_inner_microstep: 552.08 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:31:46,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.98 | bwd_microstep: 458.04 | bwd_inner_microstep: 457.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:31:47,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.43 | bwd_microstep: 444.15 | bwd_inner_microstep: 444.00 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 15:31:48,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 292.97 | bwd_microstep: 535.77 | bwd_inner_microstep: 535.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:31:49,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.58 | bwd_microstep: 440.54 | bwd_inner_microstep: 440.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:31:50,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 246.78 | bwd_microstep: 400.06 | bwd_inner_microstep: 399.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:31:51,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.72 | bwd_microstep: 451.31 | bwd_inner_microstep: 451.01 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.08
[2024-12-05 15:31:51,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 258.06 | bwd_microstep: 428.76 | bwd_inner_microstep: 428.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:31:53,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.34 | optimizer_gradients: 14.05 | optimizer_step: 23.17
[2024-12-05 15:31:53,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.70 | bwd_microstep: 1550.34 | bwd_inner_microstep: 1538.17 | bwd_allreduce_microstep: 12.08 | step_microstep: 113.64
[2024-12-05 15:31:53,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4348.73 | bwd: 8623.00 | bwd_inner: 8609.00 | bwd_allreduce: 12.90 | step: 114.78

 99%|█████████▉| 382/384 [1:48:50<00:30, 15.11s/it]
                                                   
{'loss': 0.1263, 'grad_norm': 1.7260781526565552, 'learning_rate': 7.13187201450971e-11, 'epoch': 1.99}

 99%|█████████▉| 382/384 [1:48:50<00:30, 15.11s/it][2024-12-05 15:31:54,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 277.81 | bwd_microstep: 460.62 | bwd_inner_microstep: 460.48 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:31:55,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.06 | bwd_microstep: 453.14 | bwd_inner_microstep: 452.99 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09
[2024-12-05 15:31:56,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.24 | bwd_microstep: 487.02 | bwd_inner_microstep: 486.84 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10
[2024-12-05 15:31:57,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 268.49 | bwd_microstep: 435.09 | bwd_inner_microstep: 434.96 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:31:58,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 318.10 | bwd_microstep: 520.19 | bwd_inner_microstep: 520.03 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.11
[2024-12-05 15:31:59,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 240.98 | bwd_microstep: 378.96 | bwd_inner_microstep: 378.77 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.30
[2024-12-05 15:31:59,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 265.69 | bwd_microstep: 433.06 | bwd_inner_microstep: 432.93 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09
[2024-12-05 15:32:00,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.34 | bwd_microstep: 579.14 | bwd_inner_microstep: 579.01 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:32:01,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 232.81 | bwd_microstep: 358.73 | bwd_inner_microstep: 358.58 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09
[2024-12-05 15:32:02,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.38 | bwd_microstep: 478.53 | bwd_inner_microstep: 478.40 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:32:03,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 243.58 | bwd_microstep: 369.63 | bwd_inner_microstep: 369.52 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:32:04,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 263.20 | bwd_microstep: 448.34 | bwd_inner_microstep: 448.20 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:32:04,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 247.80 | bwd_microstep: 397.16 | bwd_inner_microstep: 397.02 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:32:06,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 354.33 | bwd_microstep: 739.60 | bwd_inner_microstep: 739.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:32:06,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 264.54 | bwd_microstep: 459.79 | bwd_inner_microstep: 459.66 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:32:09,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.38 | optimizer_gradients: 14.03 | optimizer_step: 23.31
[2024-12-05 15:32:09,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 303.34 | bwd_microstep: 2109.87 | bwd_inner_microstep: 2097.77 | bwd_allreduce_microstep: 12.03 | step_microstep: 114.04
[2024-12-05 15:32:09,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4416.70 | bwd: 9108.91 | bwd_inner: 9094.99 | bwd_allreduce: 12.75 | step: 115.33

100%|█████████▉| 383/384 [1:49:05<00:15, 15.24s/it]
                                                   
{'loss': 0.1135, 'grad_norm': 1.6607623100280762, 'learning_rate': 1.7829997945084662e-11, 'epoch': 1.99}

100%|█████████▉| 383/384 [1:49:05<00:15, 15.24s/it][2024-12-05 15:32:10,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 234.22 | bwd_microstep: 324.35 | bwd_inner_microstep: 324.22 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:32:11,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 282.47 | bwd_microstep: 482.89 | bwd_inner_microstep: 482.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:32:11,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 260.77 | bwd_microstep: 442.09 | bwd_inner_microstep: 441.95 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:32:13,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.53 | bwd_microstep: 684.92 | bwd_inner_microstep: 684.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:32:13,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 261.27 | bwd_microstep: 432.92 | bwd_inner_microstep: 432.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:32:14,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 267.13 | bwd_microstep: 436.77 | bwd_inner_microstep: 436.63 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07
[2024-12-05 15:32:15,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 276.50 | bwd_microstep: 478.25 | bwd_inner_microstep: 478.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:32:16,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.85 | bwd_microstep: 590.43 | bwd_inner_microstep: 590.30 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06
[2024-12-05 15:32:17,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 254.48 | bwd_microstep: 414.44 | bwd_inner_microstep: 414.30 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:32:18,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.06 | bwd_microstep: 528.54 | bwd_inner_microstep: 528.41 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07
[2024-12-05 15:32:19,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 272.21 | bwd_microstep: 441.87 | bwd_inner_microstep: 441.76 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06
[2024-12-05 15:32:19,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 237.92 | bwd_microstep: 356.97 | bwd_inner_microstep: 356.84 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:32:20,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 230.55 | bwd_microstep: 353.91 | bwd_inner_microstep: 353.80 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:32:21,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 274.24 | bwd_microstep: 484.82 | bwd_inner_microstep: 484.68 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08
[2024-12-05 15:32:22,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.06 | bwd_microstep: 417.59 | bwd_inner_microstep: 417.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08
[2024-12-05 15:32:25,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.27 | optimizer_gradients: 14.09 | optimizer_step: 23.18
[2024-12-05 15:32:25,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 229.47 | bwd_microstep: 2890.38 | bwd_inner_microstep: 2878.27 | bwd_allreduce_microstep: 12.03 | step_microstep: 114.70
[2024-12-05 15:32:25,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4360.81 | bwd: 9761.14 | bwd_inner: 9747.39 | bwd_allreduce: 12.68 | step: 115.77

100%|██████████| 384/384 [1:49:22<00:00, 15.56s/it]
                                                   
{'loss': 0.1315, 'grad_norm': 1.7004939317703247, 'learning_rate': 0.0, 'epoch': 2.0}

100%|██████████| 384/384 [1:49:22<00:00, 15.56s/it][INFO|trainer.py:3705] 2024-12-05 15:32:31,871 >> Saving model checkpoint to /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025/checkpoint-384
[INFO|configuration_utils.py:407] 2024-12-05 15:32:31,937 >> Configuration saved in /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025/checkpoint-384/config.json
[INFO|configuration_utils.py:868] 2024-12-05 15:32:31,972 >> Configuration saved in /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025/checkpoint-384/generation_config.json
[INFO|modeling_utils.py:2838] 2024-12-05 15:33:11,815 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 4 checkpoint shards. You can find where each parameters has been saved in the index located at /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025/checkpoint-384/model.safetensors.index.json.
[INFO|tokenization_utils_base.py:2649] 2024-12-05 15:33:11,856 >> tokenizer config file saved in /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025/checkpoint-384/tokenizer_config.json
[INFO|tokenization_utils_base.py:2658] 2024-12-05 15:33:11,895 >> Special tokens file saved in /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025/checkpoint-384/special_tokens_map.json
[2024-12-05 15:33:12,401] [INFO] [logging.py:128:log_dist] [Rank 0] [Torch] Checkpoint global_step384 is about to be saved!
[2024-12-05 15:33:12,431] [INFO] [logging.py:128:log_dist] [Rank 0] Saving model checkpoint: /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025/checkpoint-384/global_step384/mp_rank_00_model_states.pt
[2024-12-05 15:33:12,431] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025/checkpoint-384/global_step384/mp_rank_00_model_states.pt...
[2024-12-05 15:33:52,818] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025/checkpoint-384/global_step384/mp_rank_00_model_states.pt.
[2024-12-05 15:33:53,188] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025/checkpoint-384/global_step384/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2024-12-05 15:36:48,940] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025/checkpoint-384/global_step384/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2024-12-05 15:36:49,848] [INFO] [engine.py:3536:_save_zero_checkpoint] zero checkpoint saved /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025/checkpoint-384/global_step384/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
[2024-12-05 15:36:49,848] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step384 is ready now!
[INFO|trainer.py:2505] 2024-12-05 15:36:52,089 >> 

Training completed. Do not forget to share your model on huggingface.co/models =)



                                                   
{'train_runtime': 6828.3197, 'train_samples_per_second': 7.211, 'train_steps_per_second': 0.056, 'train_loss': 0.19105235013800362, 'epoch': 2.0}

100%|██████████| 384/384 [1:53:48<00:00, 15.56s/it]
100%|██████████| 384/384 [1:53:48<00:00, 17.78s/it]
[INFO|image_processing_base.py:258] 2024-12-05 15:36:52,231 >> Image processor saved in /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025/preprocessor_config.json
[INFO|trainer.py:3705] 2024-12-05 15:36:57,061 >> Saving model checkpoint to /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025
[INFO|configuration_utils.py:407] 2024-12-05 15:36:57,118 >> Configuration saved in /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025/config.json
[INFO|configuration_utils.py:868] 2024-12-05 15:36:57,153 >> Configuration saved in /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025/generation_config.json
[INFO|modeling_utils.py:2838] 2024-12-05 15:37:37,945 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 4 checkpoint shards. You can find where each parameters has been saved in the index located at /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025/model.safetensors.index.json.
[INFO|tokenization_utils_base.py:2649] 2024-12-05 15:37:38,009 >> tokenizer config file saved in /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025/tokenizer_config.json
[INFO|tokenization_utils_base.py:2658] 2024-12-05 15:37:38,079 >> Special tokens file saved in /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025/special_tokens_map.json
***** train metrics *****
  epoch                    =       1.9961
  total_flos               = 2845225360GF
  train_loss               =       0.1911
  train_runtime            =   1:53:48.31
  train_samples_per_second =        7.211
  train_steps_per_second   =        0.056
Figure saved at: /nas/shared/NLP_A100/wuzhenyu/ckpt/qwen2vl_7b_sim_24k_1025/training_loss.png
12/05/2024 15:37:39 - WARNING - llamafactory.extras.ploting - No metric eval_loss to plot.
12/05/2024 15:37:39 - WARNING - llamafactory.extras.ploting - No metric eval_accuracy to plot.
[INFO|modelcard.py:449] 2024-12-05 15:37:39,381 >> Dropping the following result as it does not have all the necessary fields:
{'task': {'name': 'Causal Language Modeling', 'type': 'text-generation'}}