

TITLE
Slower training speed under DeepSpeed

COMMENTS
3

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.8.2
- Platform: nvcr.io/nvidia/pytorch:21.02-py3 container on CentOS7
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.0 with GPU
- Tensorflow version (GPU?): not used
- Using GPU in script?: YES, Tesla P40
- Using distributed or parallel set-up in script?: YES

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [X] the official example scripts: 
   examples/pytorch/translation/run_translation.py
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: 
   wwm16
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Create container using `sudo docker run -d -it --runtime=nvidia --net=host --ipc=host -v /home/user/:/workspace nvcr.io/nvidia/pytorch:21.02-py3  bash` on a Linux server with CentOS7 and Tesla P40 GPUs.
2. Install python dependencies mentioned above.
3. Run run_translation.py with different parameters listed below:
    1. DDP with fp16 open
     CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node 4 examples/pytorch/translation/run_translation.py  --model_name_or_path t5-small --per_device_train_batch_size 1   --output_dir output_dir --overwrite_output_dir --fp16 --do_train --max_train_samples 500 --num_train_epochs 1 --dataset_name wmt16 --dataset_config "ro-en" --source_lang en --target_lang ro

    2. DDP without fp16
    CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node 4 examples/pytorch/translation/run_translation.py  --model_name_or_path t5-small --per_device_train_batch_size 1   --output_dir output_dir --overwrite_output_dir --do_train --max_train_samples 500 --num_train_epochs 1 --dataset_name wmt16 --dataset_config "ro-en" --source_lang en --target_lang ro ```

     3. DeepSpeed ZeRO2 with fp16 open
     deepspeed examples/pytorch/translation/run_translation.py --deepspeed tests/deepspeed/ds_config_zero2.json --model_name_or_path t5-small --per_device_train_batch_size 1   --output_dir output_dir --overwrite_output_dir --fp16 --do_train --max_train_samples 500 --num_train_epochs 1 --dataset_name wmt16 --dataset_config "ro-en" --source_lang en --target_lang ro

    4. DeepSpeed ZeRO2 without fp16
    deepspeed examples/pytorch/translation/run_translation.py --deepspeed tests/deepspeed/ds_config_zero2.json --model_name_or_path t5-small --per_device_train_batch_size 1   --output_dir output_dir --overwrite_output_dir --do_train --max_train_samples 500 --num_train_epochs 1 --dataset_name wmt16 --dataset_config "ro-en" --source_lang en --target_lang ro

    5. DeepSpeed ZeRO3 with fp16 open
    deepspeed examples/pytorch/translation/run_translation.py --deepspeed tests/deepspeed/ds_config_zero3.json --model_name_or_path t5-small --per_device_train_batch_size 1   --output_dir output_dir --overwrite_output_dir --fp16 --do_train --max_train_samples 500 --num_train_epochs 1 --dataset_name wmt16 --dataset_config "ro-en" --source_lang en --target_lang ro

4. Training metrics are listed according to the above experiments order:
     1. ***** train metrics *****
  epoch                    =        1.0
  train_loss               =     1.5905
  train_runtime            = 0:00:20.29
  train_samples            =        500
  train_samples_per_second =     24.632
  train_steps_per_second   =      6.158
    2. ***** train metrics *****
  epoch                    =        1.0
  train_loss               =      1.482
  train_runtime            = 0:00:17.57
  train_samples            =        500
  train_samples_per_second =     28.448
  train_steps_per_second   =      7.112
    3. ***** train metrics *****
  epoch                    =        1.0
  train_loss               =     1.6752
  train_runtime            = 0:00:32.45
  train_samples            =        500
  train_samples_per_second =     15.406
  train_steps_per_second   =      3.851
    4. ***** train metrics *****
  epoch                    =        1.0
  train_loss               =      1.523
  train_runtime            = 0:00:20.15
  train_samples            =        500
  train_samples_per_second =     24.813
  train_steps_per_second   =      6.203
    5. ***** train metrics *****
  epoch                    =        1.0
  train_loss               =      1.523
  train_runtime            = 0:00:20.15
  train_samples            =        500
  train_samples_per_second =     24.813
  train_steps_per_second   =      6.203
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
According to DeepSpeed's official document, its training speedup could up to 10 times faster. But in my experiments I could not get that much speedup. Since translation script is the tutorial task mentioned in the Transformers's "DeepSpeed Integration"  document, so my expectation is a faster training speed. Is there any environment's limitations in my experiment? Or the speedup is not guaranteed? Thank you guys in advance for help me out. 
<!-- A clear and concise description of what you would expect to happen. -->

@stas00 