

TITLE
Colab pro -fine RoBERTa error tcmalloc: large alloc 6325288960

COMMENTS
4

REACTIONS
+1: 2
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
I want to fine-tune RoBERTa on my newspaper data  (around 8GB) using Colab pro. It works fine on small data. I have given my codes below. Are my codes correct? **Is there any way to handle this memory problem**? The error crushes the colab.

tcmalloc: large alloc 6325288960 bytes == 0x447fa000 @  0x7f3438dcc1e7 0x59221c 0x4ca6f4 0x566daa 0x5a4df1 0x5a5eea 0x4ce082 0x566c02 0x5a4df1 0x5a60ae 0x5bd138 0x50a47f 0x50c1f4 0x507f24 0x509202 0x594b01 0x54a17f 0x5517c1 0x5a9eec 0x50a783 0x50cfd6 0x507f24 0x509c50 0x50a64d 0x50cfd6 0x509918 

 ```

!python "/content/transformers/examples/language-modeling/run_language_modeling.py" \
    --output_dir "/content/drive/My Drive/EPU-NLP/FinetuneModel/output" \
    --model_type roberta \
    --model_name_or_path roberta-base \
    --do_train \
    --per_gpu_train_batch_size 16 \
    --seed 22 \
    --train_data_file "/content/drive/My Drive/EPU-NLP/FinetuneModel/data_all.txt" \
    --block_size 256 \
    --line_by_line \
    --weight_decay 0.01 \
    --adam_epsilon 1e-6 \
    --save_total_limit 500 \
    --learning_rate 6e-4 \
    --num_train_epochs 3 \
    --save_total_limit 500 \
    --save_steps 500 \
    --mlm
```
