

TITLE
[DeepSpeed] improve checkpoint loading code plus tests

COMMENTS
1

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
This PR further improves the DeepSpeed integration

* checkpoint resuming code has been cleaned up
* detailed checkpoint saving and resuming from checkpoint tests added
* a small reshuffle made in `test_trainer.py` to enable re-using helper functions in other test modules
* switched `test_trainer.py` to `TestCasePlus` so it's easier to deal with temp dirs during debug
* adjusted `init_deepspeed` to make a a deepcopy of the config dict passed to it, so that the user's copy isn't affected - needed at least for tests

Note that I made a failed attempt to load from resume point fatal under deepspeed. I'm not sure why the normal code just warns if a wrong path is passed. Unless I'm missing something, if a user expects to resume and it is not possible it should be fatal IMHO, so that they can correct their launching code.

@sgugger 