

TITLE
Train robertatokenizer failed due to pad token not found

COMMENTS
3

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: Windows
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7?  3080 RTX
- Tensorflow version (GPU?):
- Using GPU in script?: 
- Using distributed or parallel set-up in script?:

### Who can help
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): Roberta

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
My first step is to download some of the esperberto data from the sites mentioned in this tutorial https://huggingface.co/blog/how-to-train 

Few issues
1. Regarding the tutorial, they make you train a ByteLevelBPETokenizer but this is never used in the training code. The training code isn't even in the tutorial 👎 
2. I came across this https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb
It looks good except whatever ByteLevelBPETokenizer is never used in the training process so I tried to find a way to use it. I tried two approaches both result in the same outcome. I tried using the BPE and not ByteLevelBPETokenizer. I have no clue what is the best practice or why neither of them are working.

This is my code to do the tokenizer. You can uncomment whatever

```
#! pip install tokenizers
#%% Import Statements

from pathlib import Path
from transformers import RobertaTokenizer

from tokenizers import Tokenizer
from tokenizers.trainers import BpeTrainer
from tokenizers.models import BPE 
from tokenizers.pre_tokenizers import Whitespace

# from tokenizers import ByteLevelBPETokenizer
# from tokenizers.implementations import ByteLevelBPETokenizer
from tokenizers.processors import BertProcessing
import os.path as osp

#%% Train Tokenizer
if (not osp.exists('models/BPEtokenizer.json')):
    paths = [str(x) for x in Path("./eo_data/").glob("**/*.txt")]

    # Initialize a tokenizer
    # tokenizer = ByteLevelBPETokenizer()

    # # Customize training
    # tokenizer.train(files=paths, vocab_size=52000, min_frequency=3, special_tokens=[
    #     "<s>",
    #     "<pad>",
    #     "</s>",
    #     "<unk>",
    #     "<mask>"
    # ])

    tokenizer = Tokenizer(BPE())
    trainer = BpeTrainer(vocab_size=52000,min_frequency=3, special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])
    tokenizer.pre_tokenizer = Whitespace()
    tokenizer.train(trainer, paths)
    
    # Save files to disk    
    tokenizer.save('models/BPEtokenizer.json')

#%% Tokenize
tokenizer = Tokenizer.from_file('models/BPEtokenizer.json')
# tokenizer._tokenizer.post_processor = BertProcessing(
#     ("</s>", tokenizer.token_to_id("</s>")),
#     ("<s>", tokenizer.token_to_id("<s>")),
# )
# tokenizer.enable_truncation(max_length=512)
output = tokenizer.encode("Mi estas Julien.😁")
print(output.tokens)

print(output.ids)
# Encoding(num_tokens=7, ...)
# tokens: ['<s>', 'Mi', 'Ġestas', 'ĠJuli', 'en', '.', '</s>']
```


This is my code to do training 

```
import torch
from transformers import RobertaConfig
from transformers import RobertaTokenizerFast
from transformers import RobertaForMaskedLM
from transformers import LineByLineTextDataset
from transformers import DataCollatorForLanguageModeling
from pathlib import Path
from transformers import DataCollatorForLanguageModeling
from tokenizers import ByteLevelBPETokenizer
from transformers import PreTrainedTokenizerFast
from tokenizers import Tokenizer
# Tutorial from https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb#scrollTo=BzMqR-dzF4Ro 

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
# tokenizer = RobertaTokenizerFast.from_pretrained("./EsperBERTo", max_len=512)
# tokenizer = ByteLevelBPETokenizer("models/esperberto-vocab.json","models/esperberto-merges.txt") # ? This actually doesn't work. You will get an error saying tokenizer is not callable. 

tokenizer = PreTrainedTokenizerFast(tokenizer_file='models/BPEtokenizer.json')
# tokenizer = Tokenizer.from_file('models/BPEtokenizer.json')

mlm=False

config = RobertaConfig(
    vocab_size=52_000,
    max_position_embeddings=514,
    num_attention_heads=12,
    num_hidden_layers=6,
    type_vocab_size=1,
)

# Training from scratch 
model = RobertaForMaskedLM(config=config)
model.num_parameters()

paths = [str(x) for x in Path("eo_data/").glob("**/*.txt")]
# Build the dataset 

dataset = LineByLineTextDataset(tokenizer=tokenizer, file_path="eo_data/shuff-orig/eo/eo.txt",block_size=128)

# mlm = mask modeling language 
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=mlm, mlm_probability=0.15
    )



from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="models/EsperBERTo-small",
    overwrite_output_dir=True,
    num_train_epochs=1000,
    per_gpu_train_batch_size=64,
    save_steps=10_000,
    save_total_limit=2,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset)

trainer.train()
```

I keep getting the error `Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.`

Also I couldn't set mlm=True either. Do you have any good tutorials on how to train your own set of data using Roberta? 

If anyone wants to pull my files you can grab them and the dataset here 
https://1drv.ms/u/s!Apa0_j-AivqTpqNz7r0M3NNhCm2W_A?e=BMLvqv

If you guys resolve this then I'll update and post a public google colab 



