

TITLE
Using the XLNet or Tranformer-XL as an EncoderDecoder 

COMMENTS
4

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
I want to train a long sequence dataset (MIDI text event representation like the one in [MuseNet](https://openai.com/blog/musenet/#dataset)) from scratch. Since, I can't split the sequence to "sentences" I am using XLNet (or Transformer-XL). I am modelling the task as a sequence2sequence task (with max input seq length of around 40k tokens and output length of 4k tokens) so I want to use an Encoder Decoder Framework. 

Is it possible to use XLNet as the encoder and decoder, or just the encoder and use GPT-2 for example to do the decoding (because of the smaller output sequence length).

Thank you 🤗