

TITLE
[s2s] Trainer vs PTL timings

COMMENTS
7

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
For the following two commands,

+ PTL finishes: 2.01 it/s, ~3H, 21.32 Rouge
+ Trainer: 1.0 it/s, roughly 5.5H, 21.36 Rouge

I wanted to report this so I don't lose track of it. Looked at the code, and don't see any obvious issue, besides that the slowdown is suspiciously close to 2x.



Any idea @patil-suraj ?

### PTL Command

```bash

export BS=32
export GAS=1

python finetune.py \
    --learning_rate=3e-5 \
    --fp16 \
    --gpus 1 \
    --do_train \
    --do_predict \
    --val_check_interval 0.25 \
    --n_val 500 \
    --num_train_epochs 2 \
    --freeze_encoder --freeze_embeds --data_dir cnn_dm \
    --max_target_length 142 --val_max_target_length=142 \
    --train_batch_size=$BS --eval_batch_size=$BS --gradient_accumulation_steps=$GAS \
    --model_name_or_path sshleifer/student_cnn_12_6 \
    --tokenizer_name facebook/bart-large \
    --warmup_steps 500 \
    --output_dir distilbart-cnn-12-6

```


### Trainer command
same as `builtin_trainer/train_distilbart_cnn.sh`:

```bash
export BS=32
export GAS=1
export m=sshleifer/student_cnn_12_6
export tok=facebook/bart-large
export MAX_TGT_LEN=142

python finetune_trainer.py \
    --model_name_or_path $m --tokenizer_name $tok \
    --data_dir cnn_dm \
    --output_dir distilbart-cnn-12-6-trainer --overwrite_output_dir \
    --learning_rate=3e-5 --sortish-sampler \
    --warmup_steps 500 \
    --fp16 \
    --n_val 500 \
    --gradient_accumulation_steps=$GAS \
    --per_device_train_batch_size=$BS --per_device_eval_batch_size=$BS \
    --freeze_encoder --freeze_embeds \
    --num_train_epochs=2 \
    --save_steps 3000 --eval_steps 3000 \
    --logging_first_step \
    --max_target_length 142 --val_max_target_length $MAX_TGT_LEN --test_max_target_length $MAX_TGT_LEN \
    --do_train --do_eval --do_predict --evaluate_during_training \
    --predict_with_generate --sortish_sampler
```