

TITLE
Slower training time per batch for increasing dataset size

COMMENTS
7

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 4.0.0-rc-1
- `tokenizers` version: 0.9.4
- `datasets` version: 1.1.3
- Platform: Linux-3.10.0-862.el7.x86_64-x86_64-with-centos-7.8.2003-Core
- Python version: 3.7.2
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes, V100
- Using distributed or parallel set-up in script?: No

### Who can help

 @patrickvonplaten @sgugger @LysandreJik 

## Information

Model I am using: BERT, language modelling task.


The problem arises when using:
* [ x] the official example scripts: /examples/language-modeling/run_mlm.py

The tasks I am working on is:
* [ x] my own task or dataset: BERT MLM pre-training with own dataset

I need to pre-train a BERT base model from scratch with own dataset. 
The dataset has millions of lines, each line is a short document.
I am experiencing slower training time given increasing size of the dataset. 

To debug the problem, I have already tried to split original datasets into several smaller files (issue  https://github.com/huggingface/datasets/issues/610), switch on/off the caching mechanism, but no improvements. 

What could it be? I am not able to find the origin of the problem. Thanks a lot!


## To reproduce

Steps to reproduce the behavior:

1. run_mlm.py with increasing dataset sizes.
2. This results in slower training time for each batch.

Below some stats, each batch is 128, and I am running run_mlm.py with --line_by_line option. The increase seems not linear.

| Number lines in dataset  |  seconds per batch |
|---|---|
|  100k |  0.16 |
|  10M |  0.25 |
|  100M |  3 |

BERT parameters:
```
Model config BertConfig {
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 18,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "type_vocab_size": 2,
  "vocab_size": 32000
}
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
I expect that the training time for each batch will remain constant given different datasets' sizes. 
