

TITLE
DeepSpeed: There is no obvious benefit from increasing the number of GPU nodes

COMMENTS
12

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
I tested the performance of DeepSpeed on a GPU cluster, all GPUs are V100 16GB.
"2x4" means 2 node with 4 gpu each

my setup:

HF launch scripts:
```bash
deepspeed \
    --hostfile=hostfile.txt \
    run_clm.py \
    --deepspeed ds_config.json \
    --model_name_or_path /huggingface_with_deepspeed/gpt2-large/ \
    --preprocessing_num_workers $(nproc) \
    --do_train \
    --train_file /data/train_mini.txt \
    --fp16 true \
    --output_dir output \
    --overwrite_output_dir true \
    --per_device_train_batch_size 1
```

DeepSpeed config:
```json
{
    "train_micro_batch_size_per_gpu": "auto",
    "fp16": {
        "enabled": true,
        "loss_scale": 0
    },

    "flops_profiler": {
        "enabled": true,
        "profile_step": 10,
        "module_depth": -1,
        "top_modules": 1,
        "detailed": true,
        "output_file": "./tmp.log"
      },

    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true
        },
        "offload_param":{
            "device": "cpu",
            "pin_memory": true
        },
        "allgather_partitions": true,
        "allgather_bucket_size": 2e8,
        "reduce_scatter": true,
        "reduce_bucket_size": 2e8,
        "overlap_comm": true,
        "contiguous_gradients": true,
        "grad_hooks": true,
        "round_robin_gradients": false
    },

    "gradient_clipping": 1.0,

    "wall_clock_breakdown": false,

    "sparse_attention": {
    "mode": "fixed",
    "block": 16,
    "different_layout_per_head": true,
    "num_local_blocks": 4,
    "num_global_blocks": 1,
    "attention": "bidirectional",
    "horizontal_global_attention": false,
    "num_different_global_patterns": 4
  }
}
```

performance result:


|GPT2-large(770M)<br>BATCH_SIZE=1<br>1x4 | HF with DS(ZeRO-1) | HF with DS(ZeRO-2) | HF with DS(ZeRO-2-offload) | HF with DS(ZeRO-3) | HF with DS(ZeRO-3-offload)|
|-- | -- | -- | -- | -- | --|
|CUDA MEM(MB)|14837 | 15983 | 14483 | 16155 | 13145|
|Forward time(ms)|83.59 | 83.72 | 83.47 | 529.49 | 538.30|
|Backward time(ms)|437.23 | 283.19 | 658.34 | 1016.79 | 976.68|
|Step time(ms)|2370.66 | 2372.91 | 4455.99 | 154.37 | 2163.18|
|Total time(ms)|2892 | 2740 | 5199 | 1770 | 3740|
|Throughput(samples/s)|1.38 | 1.46 | 0.77 | 2.26 | 1.06|

|GPT2-large(770M)<br>BATCH_SIZE=1<br>2x4 | HF with DS(ZeRO-1) | HF with DS(ZeRO-2) | HF with DS(ZeRO-2-offload) | HF with DS(ZeRO-3) | HF with DS(ZeRO-3-offload)|
|-- | -- | -- | -- | -- | --|
|CUDA MEM(MB)|14979 | 14903 | 14457 | 14521 | 13147|
|Forward time(ms)|83.89 | 83.90 | 83.61 | 2685.72 | 2701.03|
|Backward time(ms)|5370.95 | 2783.34 | 3180.92 | 4502.87 | 4618.15|
|Step time(ms)|2370.66 | 2445.77 | 2338.42 | 3759.82 | 131.51 | 1335.59|
|Total time(ms)|2892 | 7901 | 5206 | 7063 | 7320 | 8655|
|Throughput(samples/s)|1.01 | 1.53 | 1.13 | 1.09 | 0.92|

|GPT2-large(770M)<br>BATCH_SIZE=1<br>4x4 | HF with DS(ZeRO-1) | HF with DS(ZeRO-2) | HF with DS(ZeRO-2-offload) | HF with DS(ZeRO-3) | HF with DS(ZeRO-3-offload)|
|-- | -- | -- | -- | -- | --|
|CUDA MEM(MB)|14793 | 14401 | 14247 | 13763 | 12981|
|Forward time(ms)|83.36 | 83.43 | 83.71 | 2339.03 | 2414.61|
|Backward time(ms)|4980.09 | 2842.94 | 3194.67 | 4201.04 | 4348.77|
|Step time(ms)|2632.80 | 2777.44 | 3384.50 | 177.98 | 870.95|
|Total time(ms)|2892 | 7701 | 5707 | 6665 | 6722 | 7634|
|Throughput(samples/s)|2.07 | 2.80 | 2.4 | 2.38 | 2.09|


To my knowledge, there are something strange:
- GPU memory has not decreased significantly with the increase in the number of nodes ？
- Offload seem not reduce memory significantly？
- Refer to paper: [ZeRO-Offload](https://arxiv.org/abs/2101.06840), "It can train models with over 13 billion parameters on a single GPU with ZeRO2-Offload",  but the reality is that I can’t even train the GPT2-xl model on 16 GPUs, which is only 1.5B parameters

Are these results in line with expectations？