text: 
  
  TITLE
  Resumption of training using the same parameters as before fails with CUDA out of memory error.
  
  COMMENTS
  6
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ### System Info
  
  ```shell
  - `transformers` version: 4.15.0
  - Platform: Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core
  - Python version: 3.6.8
  - PyTorch version (GPU?): 1.7.0 (False)
  - Tensorflow version (GPU?): 2.6.2 (False)
  - Flax version (CPU?/GPU?/TPU?): not installed (NA)
  - Jax version: not installed
  - JaxLib version: not installed
  - Using GPU in script?: Yes
  - Using distributed or parallel set-up in script?: No
  ```
  
  
  ### Who can help?
  
  @ydshieh
  @sgugger
  
  ### Information
  
  - [X] The official example scripts
  - [X] My own modified scripts
  
  ### Tasks
  
  - [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
  - [X] My own task or dataset (give details below)
  
  ### Reproduction
  
  1. I used longformer_training.py to train longformer, without getting a CUDA out of memory error, the training continues until the server time limit is reached.
  [longformer_training.txt](https://github.com/huggingface/transformers/files/8646199/longformer_training.txt)
  
  2. I rented a server again and resumed longformer training using longformer_resuming_training.py. For some reason I got a CUDA out of memory error.
  [longformer_resume_training.txt](https://github.com/huggingface/transformers/files/8646200/longformer_resume_training.txt)
  
  Errors are as follows.
  [error.txt](https://github.com/huggingface/transformers/files/8646204/error.txt)
  
  
  
  ### Expected behavior
  
  ```shell
  I am using Huggingface to research patent documents and I am really in need of help as my term is coming to an end. Your help would be really appreciated.
  The expected behavior is that the same model is trained before and after the training is resumed, so it will not CUDA out of memory after resumption and complete the training successfully.
  Thank you very much in advance.
  ```
  