text: 
  
  TITLE
  [Deepspeed] Allow HF optimizer and scheduler to be passed to deepspeed 
  
  COMMENTS
  9
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  # Use HF optimizer and/or scheduler unless specified in deepspeed config
  
  If HF is already creating an optimizer and LR scheduler, we should not try to match that config/implementation in a ds_config.json instead we pass it to deepspeed.initialize(..., lr_scheduler=hf_lr_scheduler)
  
  * [x] This PR checks if ds_config has an optimizer or scheduler, if it does not, calls  `create_optimizer` or `create_cheduler` (after splitting it) to create an optimizer or scheduler.  Then HF optimizer and scheduler are passed it to deepspeed.initialize().
  DeepSpeed can handle any optimizer and scheduler if these are passed directly to deepspeed.initialize() as an object.
  
  Due to the chicken-n-egg init problem, the valid combinations are:
  
  
  | Combos       | HF Scheduler | DS Scheduler |
  |--------------|--------------|--------------|
  | HF Optimizer | Yes          | Yes          |
  | DS Optimizer | No           | Yes          |
  
  
  but if `cpu_offload` is used all bets are off - we can only use DS optim/sched.
  
  ----------
  added by @stas00 below:
  
  Added:
  * [x] make init_deepspeed support config dict, besides the config file - this makes the testing much easier
  * [x] add tests for this PR using this new feature of passing the dict
  * [x] various small clean ups
  * [x] update the docs
  * [x] check for `cpu_offload` - add test
  * [x] recode the config overrides to have one true source of values
  * [x] tweak one not working test 
  
  **blocking event: waiting for a new release 0.3.13 from DeepSpeed.**
  
  @sgugger 
  