text: 
  
  TITLE
  [Trainer] implement gradient_accumulation_steps support in DeepSpeed integration
  
  COMMENTS
  1
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  This PR:
  
  Fixes in a bug:
  - `lr_scheduler.step()` shouldn't be called under DeepSpeed - it's already called in its `optimizer.step()` internally - so it was moving through the scheduler rate change at twice the speed :(
  
  Adds support for `gradient_accumulation_steps`:
  * makes `gradient_accumulation_steps` work with deepspeed - for nuances see: https://github.com/microsoft/DeepSpeed/issues/776 - it required a lot of `if` / `if nots` - not helping the readability of the trainer  - and took a lot of trial and error to figure out - but what to do
  * adds a corresponding doc 
  * adds a first serious quality test for DeepSpeed that measures that `gradient_accumulation_steps` works - modelled after `test_trainer.py`'s own `test_gradient_accumulation` and extends it to compare loss as well, and also tests that the optimizer actually kicked in - with fp16 deepspeed it normally takes a few dozen steps before it kicks in with dynamic scaling enabled.
  * extends `testing_utils` with a `mockenv_context` which is similar to `@mockenv`, but which can be used inside the test as context manager if multiple env vars need to be tested - `@mockenv` is only useful as a decorator. At the end I think I don't really need it as using the same env worked for all tests, but it might come handy if ports don't get released fast enough and then the test will use different ports - I'm concerned about CIs. And it's easier to re-use the class-wide env, rather hardcoding or creating a global variable - so it's just cleaner too.
  
  Suggestion/Question:
  * `get_regression_trainer` is very awesome! But importing it from a test file is not great - probably should move it and its components into a utilities file - `testing_utils.py` or create a new one `testing_training_utils.py` and in the future add other trainer-testing specific utils in there? Though this should be dealt with in a separate PR.
  
  @sgugger 