

TITLE
TextDataset bug with big files

COMMENTS
2

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
- `transformers` version: 3.0.2
- Platform: Linux-4.4.0-137-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help

@mfuntowicz

## Information

The model I am using (Bert, XLNet ...): XLMRobertaTokenizer

The problem arises when using:
* [ x ] my own modified script: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run the following code 
 ```Python
from transformers import XLMRobertaTokenizer, TextDataset
max_pos = 4096
tokenizer = XLMRobertaTokenizer.from_pretrained("xlm-roberta-base", model_max_length=max_pos)
tokenizer.model_max_length = max_pos
tokenizer.init_kwargs["model_max_length"] = max_pos
train_datapath = "path/to/train.raw"
train_dataset = TextDataset(
    tokenizer=tokenizer,
    file_path=train_datapath,
    block_size=tokenizer.max_len
)
```

## Error Messages

```
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted
```

## Expected behavior

My file `train.raw` contains 349912098 words with 2,3G. When I try with a small dataset (262703214 words and 251M), it works fine. I have modified this line in the TextDataset class https://github.com/huggingface/transformers/blob/a16e568f22a4d07813ba76343309ec20096115a5/src/transformers/data/datasets/language_modeling.py#L68 to understand where is the problem. It happens in the `tokenizer.tokenize(text)` part. I have changed it not to tokenize the entire text directly but to process chunks of the text each time, concatenating the result in a final list. Although memory hungry, this method works fine (and memory is not my problem).

**Note:** I have executed this script in a machine with 504G of RAM, and the script used approx. 36G when it died.
