

TITLE
Add DistilBERTGeneration comparable to BertGeneration

COMMENTS
6

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# 🚀 Feature request

I noticed the new `BertGeneration` class, which uses BERT-style models as both encoder and decoder, as well as the more general `EncoderDecoder` class. This is all great stuff! It would also be great to be able to use distilled models. I believe this is possible for the encoder, but for the decoder a language head must be added.

Since DistilBert is implemented as its own model, and not as a BertModel, I don't think it's possible (or at least it's not easy) for the end user to do this. At least not loading pretrained models, since any pretrained model needs to be a type approved by `AutoModelForCausalLM`.

## Motivation

Same motivation as using distilled models in general. Same results at higher speed, this time applied to an `EncoderDecoder` model.

## Your contribution

Happy to be an alpha tester for this feature
