

TITLE
Issues with finetune_trainer.py on multiple gpus

COMMENTS
4

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Environment info     
- `transformers` version:3.5.1
- Platform: google cloud
- Python version: 3.7
- PyTorch version (GPU?): yes 1.6
- Tensorflow version (GPU?): - 
- Using GPU in script?: yes 
- Using distributed or parallel set-up in script?: - 

### Who can help
 Trainer: @sgugger
 Text Generation: @patrickvonplaten @TevenLeScao
 T5: @patrickvonplaten
 examples/seq2seq: @patil-suraj

## Information
Hi
I am trying to train finetune_trainer.py on multiple gpus, one issue I got is 

1) Runtime error: Input,ouptut,and indices must be on the current device, 
Looking into the codes, in training_args.py when you set device for the n_gpu > 0 on line 401, this should be changed from cuda:0 to cuda to me.

2) The accuracy on multiple gpus does not match on single gpu and this is much lower, any idea on this.

Thank you. 

