

TITLE
Unexpected behavior when using PubMedBERT with AutoModelForMaskedLM

COMMENTS
1

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Information

I'm getting some strange behavior when using `AutoModelForMaskedLM` with PubMedBERT to impute masked tokens. The screenshot below shows a simple example where I would expect PubMedBERT to give reasonable values, but the suggested tokens are really strange. As shown just below this, `bert-base-uncased` seems to behave reasonably.

<img width="1013" alt="Screen Shot 2020-11-12 at 10 28 13 AM" src="https://user-images.githubusercontent.com/3958904/98984623-4e96b400-24d7-11eb-9a5f-faac70b1b399.png">

Same code as above, in text:

```python
import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

model_name = 'microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name).to('cuda')

text = f'Heart disease is {tokenizer.mask_token} leading cause of death in the United States.'
tokenized = tokenizer(text, return_tensors='pt').to('cuda')
print(tokenizer.convert_ids_to_tokens(tokenized.input_ids.squeeze()))
output = model(**tokenized, return_dict=True)
output.logits.size()
print(tokenizer.convert_ids_to_tokens(torch.topk(output.logits[0, 4, :], 10).indices))

model_name = 'bert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name).to('cuda')

tokenized = tokenizer(text, return_tensors='pt').to('cuda')
print(tokenizer.convert_ids_to_tokens(tokenized.input_ids.squeeze()))
output = model(**tokenized, return_dict=True)
print(tokenizer.convert_ids_to_tokens(torch.topk(output.logits[0, 4, :], 10).indices))
```

## Environment info

- `transformers` version: 3.3.1
- Platform: Linux-3.10.0-957.5.1.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No