

TITLE
[DeepSpeed] fp32 support

COMMENTS
1

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Things we need to sync with the upcoming `deepspeed==0.3.16` release:

- `zero.Init` now takes a config as an argument
- fp32-support integration, plus doc and tests
- start troubleshooting section

### Future TODO

will probably do in the next PR:
- switch `from_config()` to perform the same `zero.Init` as `from_pretrained` + add test.

###  Blocking events

PRs waiting to be merged before this PR can be merged:

- [x] https://github.com/microsoft/DeepSpeed/pull/1008 `zero.Init(config=ds_config)` new arg 
- [x] https://github.com/microsoft/DeepSpeed/pull/1004 fp32 support
- [x] new release is needed 0.3.16

@sgugger 