

TITLE
Add caching mechanism to BERT/RoBERTa/GPT2 for Seq2Seq accelerated generation

COMMENTS
3

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

All Seq2Seq models that make use of `generate()` usually allow `past_key_values` to be cached for both the cross-attention layer and the uni-directional decoder self-attention layer. For this feature request we should implement the feature for Bert2Bert, and, Roberta2Roberta.

We should implement this feature analogs to how it is implemented in Bart. This means that we should 

- 1) add the caching mechanism in the AttentionLayer as shown here for Bart: https://github.com/huggingface/transformers/blob/b01ddc9577b87f057e163d49563ee3f74f4810cf/src/transformers/models/bart/modeling_bart.py#L234
- 2) pass the `past_key_values` as tuple through the layers, making sure that it's optional for the cross-attention layer: https://github.com/huggingface/transformers/blob/b01ddc9577b87f057e163d49563ee3f74f4810cf/src/transformers/models/bart/modeling_bart.py#L433
- 3) Adapt the mask correspondingly. The easiest option is probably to just copy how it's done in Bart and remove the old attention_masking logic (making sure that all tests pass): https://github.com/huggingface/transformers/blob/b01ddc9577b87f057e163d49563ee3f74f4810cf/src/transformers/models/bart/modeling_bart.py#L91 and https://github.com/huggingface/transformers/blob/b01ddc9577b87f057e163d49563ee3f74f4810cf/src/transformers/models/bart/modeling_bart.py#L76
- 4) Add a test for `BertLMHeadModel` and `RobertaForCausalLM` that verifies that the caching mechanism works as expected:
https://github.com/huggingface/transformers/blob/b01ddc9577b87f057e163d49563ee3f74f4810cf/tests/test_modeling_bart.py#L287 
- 5) "Turn on" caching for Encoder-Decoder (this should be the last step and this might cause some other problems - happy to help here!): https://github.com/huggingface/transformers/blob/b01ddc9577b87f057e163d49563ee3f74f4810cf/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py#L427

This might be a good issue for you @patil-suraj  if interested :-) 

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->
