

TITLE
finetune_trainer being really slow on TPU

COMMENTS
8

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Hi
I need to run finetune_trainer for a large scale project on TPU, here is the command I use 

`python xla_spawn.py --num_cores 8 finetune_trainer.py     --learning_rate=3e-5         --n_val 1000  --model_name_or_path t5-small --data_dir data/wmt_en_ro/ --output_dir /home/rabeeh/temp/ --overwrite_output_dir  --tpu_num_cores=8 --max_source_length=64 --max_target_length=64 --per_device_train_batch_size=32 --per_device_eval_batch_size=32 --label_smoothing=0.1 --task="translation" --logging_steps=200 --eval_steps=500 --num_train_epochs=1 --save_steps=500 --max_source_length=128  --max_target_length=128 --val_max_target_length=128 --test_max_target_length=128
`

It is running slow for me, and I was wondering if I can make it faster, can I make it run distributedly on multiple host machines? I really appreciate providing me with command to run this code efficiently on TPUs on cluster. Is there any setting I am missing in my command? 

thank you. 