

TITLE
run_language_modeling.py TPU issue during evaluation

COMMENTS
2

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.3.1
- Platform: ubuntu
- Python version: 3.7
- PyTorch version (GPU?): xla-nightly
- Tensorflow version (GPU?):
- Using GPU in script?: no TPU
- Using distributed or parallel set-up in script?: yes

### Who can help
@sgugger @LysandreJik 
 


## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
[*] the official example scripts: (give details below)
run_language_modeling.py
The tasks I am working on is:
[*] my own task or dataset: (give details below)
txt file for pretraining Roberta model
## To reproduce
I am trying to launch a pretraining using run_language_modeling.py with TPU unfortunately I got some issue during the evaluation and logging_steps with error message  


1. `python run_language_modeling.py --model_name_or_path="roberta-base"  --model_type="roberta" --tokenizer_name="roberta-base" --do_train --evaluate_during_training --mlm --mlm_probability=0.15 --train_data_file="train.txt" --eval_data_file="val.txt" --do_eval --per_device_train_batch=8 --per_device_eval_batch=8 --output_dir="robertaweet" --max_steps=5000000 --logging_dir="log_bertweet" --logging_steps=20 --eval_steps=10 --save_steps=25 --dataloader_num_workers=0 --tpu_num_cores=8  --learning_rate=1e-4`
2.
3.

`020-10-05 15:36:48.946219: W   11372 tensorflow/core/distributed_runtime/rpc/grpc_remote_master.cc:160] RPC failed with status = "Unavailable: Socket closed" and grpc_error_string = "{"created":"@1601912208.946077389","description":"Error received from peer ipv4:10.255.226.90:8470","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket closed","grpc_status":14}", maybe retrying the RPC`
## Expected behavior

training without any error message
