

TITLE
Cuda OOM training gpt2-xl with Trainer in multi-GPUs

COMMENTS
5

REACTIONS
+1: 1
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# ❓ Questions & Help

I am currently trying to finetune the gpt2-xl. I have 2 tesla T4 GPUs. However, I get the CUDA OOM error... when I look at the use of the gpus I see that the first one is full, but the second one still has enough space. Here is my code:

```
from transformers import TextDataset,DataCollatorForLanguageModeling, AutoTokenizer

import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

from transformers import GPT2LMHeadModel, Trainer, TrainingArguments
model = GPT2LMHeadModel.from_pretrained("gpt2-xl").to(device)

from transformers import TextDataset,DataCollatorForLanguageModeling, AutoTokenizer,  TrainingArguments, Trainer
tokenizer = AutoTokenizer.from_pretrained("gpt2-xl")
train_dataset = TextDataset(
          tokenizer=tokenizer,
          file_path='dataset_training.txt',
          block_size=128)

data_collator = DataCollatorForLanguageModeling(
        tokenizer=tokenizer, mlm=False,
    )

training_args = TrainingArguments(
    output_dir='./results',         # output directory
    num_train_epochs=2,              # total # of training epochs
    per_device_train_batch_size=1,  # batch size per device during training
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs', 
    
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
    prediction_loss_only=True,
)

trainer.train()
```
I get "CUDA out of memory. Tried to allocate 40.00 MiB (GPU 0; 14.73 GiB total capacity; 13.61 GiB already allocated; 31.88 MiB free; 13.98 GiB reserved in total by PyTorch)"

When I run nvidia-smi I see:

```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.01    Driver Version: 418.87.01    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+================
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   75C    P0    34W /  70W |  15047MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla T4            Off  | 00000000:00:05.0 Off |                    0 |
| N/A   56C    P0    29W /  70W |   9479MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|===============================================
|    0      1756      C   /opt/conda/bin/python                      15037MiB |
|    1      1756      C   /opt/conda/bin/python                       9469MiB |
+-----------------------------------------------------------------------------+
```
**My question is:** Am I making a mistake? or how can a large model be trained with multiple GPUs?
