

TITLE
Mask Token Spacing in RobertaTokenizer

COMMENTS
0

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
### System Info


- `transformers` version: 4.18.0
- Platform: Linux-5.16.13-200.fc35.x86_64-x86_64-with-glibc2.34
- Python version: 3.10.2
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no



### Who can help?

@SaulLu 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import RobertaTokenizer, BertTokenizer

bert_tk = BertTokenizer.from_pretrained("bert-base-cased")
roberta_tk = RobertaTokenizer.from_pretrained("roberta-base")

enc_bert = bert_tk("Testing the spacing of " + bert_tk.pad_token + " and " + bert_tk.mask_token + " tokens.")
enc_roberta = roberta_tk("Testing the spacing of " + roberta_tk.pad_token + " and " + roberta_tk.mask_token + " tokens.")

dec_bert = bert_tk.decode(enc_bert.input_ids, skip_special_tokens=False)
dec_roberta = roberta_tk.decode(enc_roberta.input_ids, skip_special_tokens=False)

print("bert", dec_bert)
print("roberta", dec_roberta)
```

Output is:

```
bert [CLS] Testing the spacing of [PAD] and [MASK] tokens. [SEP]
roberta <s>Testing the spacing of <pad> and<mask> tokens.</s>
```

### Expected behavior


I expected Roberta's `<mask>` token to be surrounded by spaces as it is done with the `<pad>` token. In BERT, this is done the same way for both special tokens - why not in Roberta?

If someone confirms that this is indeed a bug and not expected behavior, I would be happy to try to get to the root cause myself.

