

TITLE
Trainer reported loss is wrong when using DeepSpeed and gradient_accumulation_steps > 1

COMMENTS
6

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Environment info
- `transformers` version: 4.7.0.dev0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.0
- PyTorch version (GPU?): 1.8.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: no but using DeepSpeed on a single node

### Who can help
@stas00, @sgugger (trainer.py)

### See Also
https://github.com/microsoft/DeepSpeed/issues/1107

## Information

Model I am using (Roberta)

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)
* [ ] pretraining a Language Model (wikipedia and bookcorpus datasets)

## To reproduce

Steps to reproduce the behavior:

1.  run scripts to pretrain a model with DeepSpeed on a single node with 1 GPU for N steps (gradient_accum_steps=1)
2.  run scripts to pretrain a model with DeepSpeed on a single node with 1 GPU for N steps (gradient_accum_steps=8)
3.  note that vast difference in **loss** reported on console by trainer.py

## Expected behavior

reported loss for any number of gradient_accum_steps, nodes, or GPUs should be the mean of all losses; the same order of magnitude as shown when training with gradient_accum_steps=1, on a single node, with a single GPU.
