text: 
  
  TITLE
  [deepspeed integration] HF Trainer takes over GPUs for DP
  
  COMMENTS
  1
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  on multi-gpu machine when `deepspeed` launcher is used w/o the `--num_gpus` arguments to run HF/DS integration, HF Trainer takes over and does DP over all available GPUs, which defeats the purpose of using deepspeed to fit a larger than GPU model.
  
  Normally when no `--num_gpus` argument is passed, deepspeed automatically uses all gpus, so here we have a conflict for those users who are used to that automatic behavior.
  
  So currently we need to always use something like `deepspeed --num_gpus 2` explicitly which tells HF Trainer not use DP over those GPUs.
  
  The proper solution is to code that when `--deepspeed` is passed HF Trainer won't launch DP.
  
  Thanks to Lintang Sutawika for reporting this.