text: 
  
  TITLE
  Multitask pre-training setting
  
  COMMENTS
  2
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  # 🚀 Feature request
  
  Hi everybody!
  
  Is it possible to pre-train T5 model on a mixture of tasks?
  Just to be more clear, let's assume I have created a dataset by following the example provided here:
  https://huggingface.co/transformers/model_doc/t5.html
  
  Specifically I'm referring to the _**Unsupervised denoising training**_ and _**Supervised training**_.
  What I've done is to construct a huge dataset where each tasks is concatenated to the previous one.
  
  Is this enough to experiment with the multitask pertaining setting?
  
  
  
  
  