text: 
  
  TITLE
  training text_classification with tpu using xla_spawn gives wrong result
  
  COMMENTS
  3
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  I tested text_classification code with tpu and gpu versions shown below.
  tpu(colab with 8cores) and gpu(colab) versions take 2min and 17min, respectively, which is nice.
  Training gpu version gives good behavior, where loss decreases continuously.
  
  However, for tpu version, the dataset is almost divided into 8 segments(so 1/8 times consuming), but each node may not be connected. Results from tpu of each node are just result with gpu with 1/8 of the original dataset size.
  
  1.
  Should I change something?
  Or if I`d like to use TPU, I have to use TF version?
  
  2.
  My final goal is to train Roberta on TPU in Korean.
  There are three options.
  1. Huggingface Trainer with xla  --> I am here.
  2. Huggingface TFTrainer --> TFTrainer supports TPU, but need to make mlm datasets.
  3. Fairseq with xla
  If there are any sources or examples, please let me know.
  
  Thanks,
  
  # tpu version (same one shown in the document 'https://github.com/huggingface/transformers/tree/master/examples/text-classification'):
  python examples/xla_spawn.py \
    --num_cores=8 \
    transformers/examples/text-classification/run_glue.py \
    --do_train \
    --do_eval \
    --task_name=mrpc \
    --num_train_epochs=3 \
    --max_seq_length=128 \
    --learning_rate=5e-5 \
    --output_dir=/tmp/mrpc \
    --overwrite_output_dir \
    --logging_steps=5 \
    --save_steps=5 \
    --tpu_metrics_debug \
    --model_name_or_path=bert-base-cased \
    --per_device_train_batch_size=64 \
    --per_device_eval_batch_size=64
  
  # single gpu version (remove   --num_cores=8,   --tpu_metrics_debug)
  python examples/xla_spawn.py \
    transformers/examples/text-classification/run_glue.py \
    --do_train \
    --do_eval \
    --task_name=mrpc \
    --num_train_epochs=3 \
    --max_seq_length=128 \
    --learning_rate=5e-5 \
    --output_dir=/tmp/mrpc \
    --overwrite_output_dir \
    --logging_steps=5 \
    --save_steps=5 \
    --model_name_or_path=bert-base-cased \
    --per_device_train_batch_size=64 \
    --per_device_eval_batch_size=64