

TITLE
Bert: relative_key position embedding causes error for long sequences

COMMENTS
0

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
### System Info

```shell
- `transformers` version: 4.9.2
- Platform: Linux-5.14.15-arch1-1-x86_64-with-glibc2.33
- Python version: 3.9.7
- PyTorch version (GPU?): 1.9.0+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no
```


### Who can help?

@LysandreJik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Copy paste script from below and run
2. Script:
```python
import torch
from transformers import BertConfig, BertModel

config = {
  'hidden_size': 512,
  'num_attention_heads': 8,
  'position_embedding_type': 'relative_key',
  'max_seq_length': 10,
  'max_position_embeddings': 10
}
encoder_config = BertConfig(**config)
model = BertModel(encoder_config)

batch_size, src_len = 1, 11
x = torch.zeros(batch_size, src_len).int()

model(input_ids=x)
```

### Expected behavior

```shell
Since relative attention is used (Shaw et al.) the script should run without any errors.

However, the script breaks because at least two implementations details (in the PyTorch implementation) prevent this use case:

1. Token type ids are buffered for a specific max. length:
https://github.com/huggingface/transformers/blob/ede5e041911afed37c8284a980342d4a2625b1d5/src/transformers/models/bert/modeling_bert.py#L223

2. The distance in self-attention is not clipped to the maximum distance (as in Shaw et al.):
https://github.com/huggingface/transformers/blob/ede5e041911afed37c8284a980342d4a2625b1d5/src/transformers/models/bert/modeling_bert.py#L328

There is currently no apparent way to prevent this (especially when the model is trained).
```
