text: 
  
  TITLE
  Validation loop gives OOM when finetuning T5
  
  COMMENTS
  11
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  While finetuning T5-base on summarization task, using `--sortish_sampler` it gives an OOM error starting from a particular index during the validation loop. After removing those indices and training again, I still get the OOM error, but in the second validation loop, whereas the validation loop worked well the first time in the 0th epoch. I have finetuned t5-base previously as well on the same dataset and the same environment, and it never gave this error. 
  I am using a batch size of 1 during training and evaluation both. 
  
  I am using Colab to finetune the model on a single GPU.
  GPU specs:
  GPU: Tesla K80
  RAM: 12GB
  
  This is the traceback:
  
  Epoch 0:  91% 5644/6231 [25:04<02:36,  3.75it/s, loss=2.089, v_num=16]
  Epoch 0:  91% 5645/6231 [25:06<02:36,  3.75it/s, loss=2.089, v_num=16]
  Epoch 0:  91% 5646/6231 [25:07<02:36,  3.75it/s, loss=2.089, v_num=16]
  Epoch 0:  91% 5647/6231 [25:08<02:35,  3.74it/s, loss=2.089, v_num=16]
  Epoch 1:  84% 5247/6231 [18:08<03:24,  4.82it/s, loss=1.859, v_num=16]
  Validating: 0it [00:00, ?it/s]
  Epoch 1:  84% 5248/6231 [18:10<03:24,  4.81it/s, loss=1.859, v_num=16]
  Epoch 1:  84% 5249/6231 [18:11<03:24,  4.81it/s, loss=1.859, v_num=16]Traceback (most recent call last):
    File "finetune.py", line 441, in <module>
      main(args)
    File "finetune.py", line 416, in main
      logger=logger,
    File "/content/drive/My Drive/Colab Notebooks/transformers_20_Oct_summarization/transformers/examples/lightning_base.py", line 386, in generic_train
      trainer.fit(model)
    File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 440, in fit
      results = self.accelerator_backend.train()
    File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/accelerators/gpu_accelerator.py", line 54, in train
      results = self.train_or_test()
    File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/accelerators/accelerator.py", line 66, in train_or_test
      results = self.trainer.train()
    File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 483, in train
      self.train_loop.run_training_epoch()
    File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/training_loop.py", line 569, in run_training_epoch
      self.trainer.run_evaluation(test_mode=False)
    File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 568, in run_evaluation
      output = self.evaluation_loop.evaluation_step(test_mode, batch, batch_idx, dataloader_idx)
    File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/evaluation_loop.py", line 171, in evaluation_step
      output = self.trainer.accelerator_backend.validation_step(args)
    File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/accelerators/gpu_accelerator.py", line 76, in validation_step
      output = self.__validation_step(args)
    File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/accelerators/gpu_accelerator.py", line 86, in __validation_step
      output = self.trainer.model.validation_step(*args)
    File "finetune.py", line 181, in validation_step
      return self._generative_step(batch)
    File "finetune.py", line 221, in _generative_step
      max_length=self.eval_max_length,
    File "/usr/local/lib/python3.6/dist-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
      return func(*args, **kwargs)
    File "/usr/local/lib/python3.6/dist-packages/transformers/generation_utils.py", line 489, in generate
      model_kwargs=model_kwargs,
    File "/usr/local/lib/python3.6/dist-packages/transformers/generation_utils.py", line 665, in _generate_beam_search
      outputs = self(**model_inputs, return_dict=True)  # (batch_size * num_beams, cur_len, vocab_size)
    File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
      result = self.forward(*input, **kwargs)
    File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_t5.py", line 1212, in forward
      return_dict=return_dict,
    File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
      result = self.forward(*input, **kwargs)
    File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_t5.py", line 767, in forward
      output_attentions=output_attentions,
    File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
      result = self.forward(*input, **kwargs)
    File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_t5.py", line 556, in forward
      output_attentions=output_attentions,
    File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
      result = self.forward(*input, **kwargs)
    File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_t5.py", line 478, in forward
      output_attentions=output_attentions,
    File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
      result = self.forward(*input, **kwargs)
    File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_t5.py", line 374, in forward
      q, k.transpose(3, 2)
  RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 14.73 GiB total capacity; 13.70 GiB already allocated; 13.88 MiB free; 13.72 GiB reserved in total by PyTorch)
  Epoch 1:  84%|████████▍ | 5249/6231 [18:13<03:24,  4.80it/s, loss=1.859, v_num=16]
  
  
  Any help is appreciated. :)