

TITLE
PEGASUS generation/decoding VERY Slow

COMMENTS
6

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Environment info
- `transformers` version: 3.4.0
- Platform: Linux-4.19.112+-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.7.0+cu110 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@sshleifer 
 
## Information

Model I am using: *PEGASUS-large*, *PEGASUS-cnn_dailymail*, *PEGASUS-xsum*

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)


The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

I compared the generation/decoding performance of BART and PEGASUS both loaded via AutoModelSeq2SeqForLM fine-tuned on a custom dataset for the same amount of time. Generation was done using basic greedy decoding. PEGASUS models are anywhere between 5-15x slower than BART. Fine-tuning speed was on-par for both models.

## To reproduce

Steps to reproduce the behavior:
1. Load PEGASUS and BART from any of the mentioned checkpoints (using AutoModelSeq2SeqForLM)
1. Fine-tune models
2. Decode using greedy decoding
3. Compare fine-tuning performance with other Seq2Seq Model (BART-large)


## Expected behavior
Decoding performance on-par with BART-large?
