text: 
  
  TITLE
  T5 on multiple tasks
  
  COMMENTS
  1
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  Dear Huggingface team,
  I am looking for a code to pretrain T5 on multiple tasks, the best I could find is the code released with wrapper around huggingface T5 model in the original author's repo:
  https://github.com/google-research/text-to-text-transfer-transformer/blob/master/t5/models/hf_model.py
  
  On line 308-310, they create tensorflow datasets I think, and pass it to huggingface model, I was wondering if I could ask for your help, if one wants to add data parallelism to this code to make it efficient in pytorch, could you help me how I can do it in pytorch? thanks a lot, and I appreciate your help.
  