text: 
  
  TITLE
  AttributeError: 'DataParallel' object has no attribute 'save_pretrained'
  
  COMMENTS
  2
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ### System Info
  
  ```shell
  torch==1.10.2+cu113
  transformers==4.18.0
  Python 3.6.9
  Linux "18.04.6 LTS (Bionic Beaver)"
  ```
  
  I am training a T5 transformer (T5ForConditionalGeneration.from_pretrained(model_params["MODEL"])) to generate text. The model works well when I train it on a single GPU. But when I want to parallel the data across several GPUs by doing `model = nn.DataParallel(model)`, I can't save the model.
  
  The error is:
  
  > File "run.py", line 288, in T5Trainer
  >     model.save_pretrained(path)                                                                                                
  >   File "/home/USER_NAME/venv/pt_110/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1178, in __getattr__
  >     type(self).__name__, name))
  > AttributeError: 'DataParallel' object has no attribute 'save_pretrained'
  
  
  
  ### Reproduction
  
  Wrap the model with `model = nn.DataParallel(model)`.
  
  ### Expected behavior
  
  ```shell
  The model should be saved without any issues.
  ```
  