text: 
  
  TITLE
  Data collator using in Parallel training & Disable to use DistributedDataParallel
  
  COMMENTS
  5
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ### System Info
  
  ```shell
  - `transformers` version: 4.12.0.dev0
  - Platform: Linux-4.15.0-45-generic-x86_64-with-debian-stretch-sid
  - Python version: 3.7.10
  - PyTorch version (GPU?): 1.6.0 (True)
  - Tensorflow version (GPU?): not installed (NA)
  - Flax version (CPU?/GPU?/TPU?): not installed (NA)
  - Jax version: not installed
  - JaxLib version: not installed
  - Using GPU in script?: Yes
  - Using distributed or parallel set-up in script?: Both are tried and failed
  ```
  
  
  ### Who can help?
  
  Library/Trainer: @sgugger
  
  ### Information
  
  - [ ] The official example scripts
  - [X] My own modified scripts
  
  ### Tasks
  
  - [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
  - [X] My own task or dataset (give details below)
  
  ### Reproduction
  
  code sample that reproduces the problem(with data): https://drive.google.com/file/d/1jFLV9Ir0Um3C6MTPjCgZMUKJtmasrQPT/view?usp=sharing
  
  terminal input:
  For DDP:
  CUDA_VISIBLE_DEVICES=5,6 python -m torch.distributed.launch base-Trainer.py --mode "Data" --train_data_path utils/dev-small.json --output_dir outputs/Data_only_load_data/ --do_train --per_device_train_batch_size 4 --save_steps 100000
  For DataParallel:
  CUDA_VISIBLE_DEVICES=5,6 python base-Trainer.py --mode "Data" --train_data_path utils/dev-small.json --output_dir outputs/Data_only_load_data/ --do_train --per_device_train_batch_size 4 --save_steps 100000
  
  some explaination to the task and files:
  1. The task is MLM, with specified mask tokens, not randomly choosed.
  2. tokenizer is basically a roberta-base-tokenizer, adding a special token [pron]
  3. Dataset returns one sample of data each time, with type str or (str, str)
  4. tokenization and label-creating are realized in collater()
  
  error messages:
  when using DataParallel:
  /home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/torch/nn/parallel/_functions.py:61: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  
  when using DDP:
  1. set 2 gpu devices but only one gpu is used.
  5. terminal outputs as follows:
  Traceback (most recent call last):
    File "base-Trainer.py", line 67, in <module>
      main()
    File "base-Trainer.py", line 62, in main
      trainer.train()
    File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/transformers/trainer.py", line 1383, in train
      self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
    File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/transformers/trainer.py", line 1475, in _maybe_log_save_evaluate
      tr_loss_scalar = self._nested_gather(tr_loss).mean().item()
    File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/transformers/trainer.py", line 2385, in _nested_gather
      tensors = distributed_concat(tensors)
    File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 168, in distributed_concat
      dist.all_gather(output_tensors, tensor)
    File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1185, in all_gather
      work = _default_pg.allgather([tensor_list], [tensor])
  RuntimeError: All tensor operands to scatter/gather must have the same size
    0%|▎                                                                                                                    | 500/183354 [02:03<12:32:13,  4.05it/s]
  Traceback (most recent call last):
    File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/runpy.py", line 193, in _run_module_as_main
      "__main__", mod_spec)
    File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/torch/distributed/launch.py", line 261, in <module>
      main()
    File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/torch/distributed/launch.py", line 257, in main
      cmd=cmd)
  subprocess.CalledProcessError: Command '['/home/caoyq/.conda/envs/torch_cp37/bin/python', '-u', 'base-Trainer.py', '--local_rank=0', '--mode', 'Data', '--train_data_path', 'utils/dev-small.json', '--output_dir', 'outputs/Data_only_load_data/', '--do_train', '--per_device_train_batch_size', '4', '--save_steps', '100000']' returned non-zero exit status 1.
  
  ### Expected behavior
  
  ```shell
  1. I want to use DDP, not just DP
  2. From the error messages given above, my data collator may have problems. It would be so nice of you if you could tell me what's wrong and how can I fixed it. If there exists a sample I can follow, please let me know.
  
  best wish
  ```
  