text: 
  
  TITLE
  [t5 model parallel] misc fixes
  
  COMMENTS
  3
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  This PR:
  
  * in 2 places fixes an assumption that devices on the device map are always ` (0, 1, 2, 3)` and:
     1. are ordered by their cuda device id and not `(2, 3, 0, 1)`
     2. have a stride of 1 and not `(0, 2, 3, 5) `
  * adds a missing `to()`, removes a redundant `to()`
  * removes obvious comments
  * removes code that gets run twice
  * this PR continues at #9323 -  I branched off from this PR and implemented an automatic remap of inputs and a lot refactoring. 
  
  I will comment on the reasons for changes in the code.
  
  There is one gotcha wrt py36 w/o cython not having its dict ordered. Please see https://github.com/huggingface/transformers/pull/9316#discussion_r549068073
  
  I think sorting out the logic first device/last device/is_this_the_last_layer_of_this_device and such logic should be abstracted away for readability, and not needing to replicate the same logic in each model. Perhaps `self.device_map` should be a smart class that can provide all the answers via its methods.
  
  @alexorona, I'm studying your t5-mp implementation to do the same for bart. Thank you for doing the hard work of putting the foundation in place and porting 2 models!!!
  
  Please have a look and let me know if my tweaks make sense. Your original code is excellent - I'm just trying to think how to make it easier to replicate it in other models and improve readability, hence a gazillion of questions/suggestions.
  
  Also, if you don't mind I have a few design questions:
  
  1. Could you please comment on why you are splitting the encoder between all devices on the device map and the same for the decoder? Won't it be more efficient performance-wise to put the encoder on the first group of devices and decoder on the second? 
  
  2. I also find it confusing that the device map doesn't map out the whole model, but just the encoder and assumes that the decoder has the same config. I'm not familiar with t5 but other models definitely can have encoder and decoder that don't at all match number of layers-wise. And while this is perhaps not the case for t5, I think the device map should be intuitively similar for all models as we slowly progress with porting other models to MP. That is I think it should include all layers of the model and not half of them.
  
  @patrickvonplaten, @LysandreJik
  
  