

TITLE
EncoderDecoderModel with different model dimensions

COMMENTS
2

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Who can help
@patrickvonplaten, @patil-suraj

## Information

When instantiating an `EncoderDecoderModel` from two pretrained models whose model dimensions are different, a `RunTimeError` occurs at the `CrossAttention` calculation step. 

The reason is, that regardless of a potentially different encoder model dimension, the projection layers for key and value are initialized with the decoder model dimension. 

This leads to a dimensionality mismatch when performing the matrix multiplication of encoder outputs (encoder model dimension) in the key and value projection layers (decoder model dimension).

Looking a little bit deeper in the API I would suspect it should be easy to provide the correct encoder model dimension to the `Attention` module in most Model implementations and their key/value projection layers, if the `add_cross_attention=True` argument is set. Also, I think the encoder model dimension should be easily accessible via `self.encoder.config.d_model` or something along these lines.

Generally, I think there is no reason against using `EncoderDecoderModel` with `encoder='bert-large-cased'` (`d_model=1024`) and `decoder='gpt2'` (`d_model=768`), but currently this setup doesnt't work.

Thanks a lot for looking into it :) 

Best regards
Lars