text: 
  
  TITLE
  seq2seq_trainer optimization issue on TPU 
  
  COMMENTS
  5
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  Hi
  I am running seq2seq_trainer.py model on TPU v3-8 instance with pytorch xla 1.7, using adafactor,  here are the logs, could you please assist? thanks
  
  ```
  /root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/optimization.py:506: UserWarning: This overload of add_ is deprecated:
          add_(Number alpha, Tensor other)
  Consider using one of the following signatures instead:
          add_(Tensor other, *, Number alpha) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.)
    exp_avg_sq_row.mul_(beta2t).add_(1.0 - beta2t, update.mean(dim=-1))
  /root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/optimization.py:506: UserWarning: This overload of add_ is deprecated:
          add_(Number alpha, Tensor other)
  Consider using one of the following signatures instead:
          add_(Tensor other, *, Number alpha) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.)
    exp_avg_sq_row.mul_(beta2t).add_(1.0 - beta2t, update.mean(dim=-1))
  /root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/optimization.py:506: UserWarning: This overload of add_ is deprecated:
          add_(Number alpha, Tensor other)
  Consider using one of the following signatures instead:
          add_(Tensor other, *, Number alpha) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.)
    exp_avg_sq_row.mul_(beta2t).add_(1.0 - beta2t, update.mean(dim=-1))
  /root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/optimization.py:506: UserWarning: This overload of add_ is deprecated:
          add_(Number alpha, Tensor other)
  Consider using one of the following signatures instead:
          add_(Tensor other, *, Number alpha) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.)
    exp_avg_sq_row.mul_(beta2t).add_(1.0 - beta2t, update.mean(dim=-1))
  /root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/optimization.py:506: UserWarning: This overload of add_ is deprecated:
          add_(Number alpha, Tensor other)
  Consider using one of the following signatures instead:
          add_(Tensor other, *, Number alpha) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.)
    exp_avg_sq_row.mul_(beta2t).add_(1.0 - beta2t, update.mean(dim=-1))
  /root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/optimization.py:506: UserWarning: This overload of add_ is deprecated:
          add_(Number alpha, Tensor other)
  Consider using one of the following signatures instead:
          add_(Tensor other, *, Number alpha) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.)
    exp_avg_sq_row.mul_(beta2t).add_(1.0 - beta2t, update.mean(dim=-1))
    0%|                                                                                                                                                                         | 1/19380 [00:02<13:07:23,  2.44s/it]/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/optimization.py:506: UserWarning: This overload of add_ is deprecated:
          add_(Number alpha, Tensor other)
  Consider using one of the following signatures instead:
          add_(Tensor other, *, Number alpha) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.)
    exp_avg_sq_row.mul_(beta2t).add_(1.0 - beta2t, update.mean(dim=-1))
  /root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/optimization.py:506: UserWarning: This overload of add_ is deprecated:
          add_(Number alpha, Tensor other)
  Consider using one of the following signatures instead:
          add_(Tensor other, *, Number alpha) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.)
    exp_avg_sq_row.mul_(beta2t).add_(1.0 - beta2t, update.mean(dim=-1))
  {'loss': 10383.4560546875, 'learning_rate': 6e-07, 'epoch': 0.0010319917440660474}    
  
  ```
  
  on GPU also I cannot use adafactor with this error:
  
  ```
  /opt/conda/envs/internship/lib/python3.7/site-packages/transformers/optimization.py:506: UserWarning: This overload of add_ is deprecated:
          add_(Number alpha, Tensor other)
  Consider using one of the following signatures instead:
          add_(Tensor other, *, Number alpha) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.)
    exp_avg_sq_row.mul_(beta2t).add_(1.0 - beta2t, update.mean(dim=-1))
  Traceback (most recent call last):
    File "finetune_t5_trainer.py", line 223, in <module>
      main()
    File "finetune_t5_trainer.py", line 159, in main
      model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
    File "/opt/conda/envs/internship/lib/python3.7/site-packages/transformers/trainer.py", line 797, in train
      self.optimizer.step()
    File "/opt/conda/envs/internship/lib/python3.7/site-packages/torch/optim/lr_scheduler.py", line 67, in wrapper
      return wrapped(*args, **kwargs)
    File "/opt/conda/envs/internship/lib/python3.7/site-packages/transformers/optimization.py", line 510, in step
      update = self._approx_sq_grad(exp_avg_sq_row, exp_avg_sq_col)
    File "/opt/conda/envs/internship/lib/python3.7/site-packages/transformers/optimization.py", line 441, in _approx_sq_grad
      return torch.mm(r_factor.unsqueeze(-1), c_factor.unsqueeze(0))
  RuntimeError: tensors must be 2-D
  ```
  
  
  @patrickvonplaten
  