

TITLE
pegasus/cnn_dm  12-2 distillation performing poorly

COMMENTS
8

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.1.0
- Platform: Linux-5.4.0-1028-aws-x86_64-with-debian-buster-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


### Who can help
 
@sshleifer 
## Information

I am trying to distil the pegasus model to reduce the runtime and memory requirements. I am following **No Teacher Distillation** approach. However, the model generates bad quality text.

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name): CNN
* [ ] my own task or dataset: (give details below)

## To reproduce
I have trained the model using below command:

**Download data:**
wget https://cdn-datasets.huggingface.co/summarization/cnn_dm_v2.tgz
tar -xzvf cnn_dm_v2.tgz  # empty lines removed
mv cnn_cln cnn_dm

**Command to train:**
python finetune.py --learning_rate=3e-5 --do_train --do_predict --val_check_interval 0.25 --n_val 500 --num_train_epochs 6 --freeze_encoder --freeze_embeds --data_dir ./cnn_dm/ --max_target_length 142 --val_max_target_length=142 --train_batch_size=1 --eval_batch_size=1 --gradient_accumulation_steps=256 --model_name_or_path sshleifer/student_pegasus_cnn_12_2 --tokenizer_name google/pegasus-cnn_dailymail --warmup_steps 500 --output_dir distilpegasus-cnn-12-2 --gpus 1 --adafactor --num_workers=0 --fp16_opt_level=O1 --fp16

**Inference code:**

```
from transformers import PegasusForConditionalGeneration, PegasusTokenizer, PegasusConfig
import torch

PEGASUS_MODEL = '/home/ubuntu/finetune/transformers/examples/seq2seq/distilpegasus-cnn-12-2/best_tfmr'
PEGASUS_TOKENIZER = 'google/pegasus-cnn_dailymail'

class PegasusSummarizer:
    def __init__(self):
        self.torch_device = 'cpu'
        self.tokenizer = PegasusTokenizer.from_pretrained(PEGASUS_TOKENIZER)
        self.model = PegasusForConditionalGeneration.from_pretrained(PEGASUS_MODEL).to(self.torch_device)

    def summarize(self, text):
        src_text = text
        batch = self.tokenizer.prepare_seq2seq_batch([src_text],truncation=True,padding='longest').to(self.torch_device)
        translated = self.model.generate(**batch)
        tgt_text = self.tokenizer.batch_decode(translated, skip_special_tokens=True)
        return tgt_text

summarizer = PegasusSummarizer()
print(summarizer.summarize('''(CNN)For the first time in eight years, a TV legend returned to doing what he does best. Contestants told to "come on down!" on the April 1 edition of "The Price Is Right" encountered not host Drew Carey but another familiar face in charge of the proceedings. Instead, there was Bob Barker, who hosted the TV game show for 35 years before stepping down in 2007. Looking spry at 91, Barker handled the first price-guessing game of the show, the classic "Lucky Seven," before turning hosting duties over to Carey, who finished up. Despite being away from the show for most of the past eight years, Barker didn't seem to miss a beat.'''))

```
**Output:** ['"It\'s time for the first time in a five-year anniversary of the show.']
**Output of google/pegasus-cnn_dailymail model**:['Barker hosted "The Price Is Right" for 35 years.<n>He stepped down in 2007.']


test_results.txt output:
src_pad_frac = tensor(0., device='cuda:0')
src_pad_tok = tensor(0, device='cuda:0')
step_count = 26
test_avg_gen_len = 48.63716275021758
test_avg_gen_time = 1.3503953615824382
test_avg_loss = 3.6937525272369385
test_avg_rouge1 = 19.983542428198433
test_avg_rouge2 = 4.130034786771105
test_avg_rougeL = 14.352700217580503
test_avg_rougeLsum = 18.460456248912102
test_loss = tensor(3.6938, device='cuda:0')
test_rouge2 = tensor(4.1300, device='cuda:0')
tpb = tensor(511, device='cuda:0')
val_avg_gen_len = 50.144
val_avg_gen_time = 1.513235685825348
val_avg_loss = 3.77506422996521
val_avg_rouge1 = 16.9548154
val_avg_rouge2 = 3.1666046
val_avg_rougeL = 12.980990400000001
val_avg_rougeLsum = 15.404284
val_loss = tensor(3.7751, device='cuda:0')
val_rouge2 = tensor(3.1666, device='cuda:0')


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
I expect the output to be much cleaner and higher Rouge score. Any help in this regard would be of great help.

I am trying to retrain the model by removing  **--freeze_encoder**.
<!-- A clear and concise description of what you would expect to happen. -->
