

TITLE
Finetune mbart rouge score difference between training and evaluation part

COMMENTS
5

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
### Environment info
- transformers from source
- google colab

### Information
When I use the `sshleifer/student_cnn_12_6` model for `finetune_trainer.py` and then I run finetuned model in `run_eval.py`,  I can get close and high rouge scores.However when I give `facebook/mbart-large-cc25` as model and tokenizer to `finetune_trainer.py` and also, I used the same `LID` for both eos and bos , the generated text and rouge score produced by the `finetune_trainer.py` evaluation and prediction section are not good, and when I run the fine-tuned model with the `run_eval.py` rouge scores are very close to 0. What could be reason for low rouge scores and particularly the rouge score difference between training and evaluation when using mbart model.

`Finetune_trainer.py` arguments
>!python /content/transformers/examples/seq2seq/finetune_trainer.py 
    --model_name_or_path facebook/mbart-large-cc25 \ 
    --tokenizer_name facebook/mbart-large-cc25 \ 
    --data_dir /content/transformers/cnn_dm_tr \ 
    --output_dir finetuned_model --overwrite_output_dir \
    --learning_rate=3e-5 \
    --warmup_steps 500 --sortish_sampler \
    --fp16 \
    --n_val 500 \
    --freeze_encoder --freeze_embeds \
    --src_lang tr_TR --tgt_lang tr_TR \
    --gradient_accumulation_steps=1 \
    --per_device_train_batch_size=4 --per_device_eval_batch_size=4 \
    --num_train_epochs=2 \
    --save_steps 3000 --eval_steps 3000 \
    --logging_first_step \
    --max_target_length 56 --val_max_target_length 142 --test_max_target_length 142 \
    --do_train --do_eval --do_predict \
    --evaluation_strategy steps \
    --predict_with_generate --sortish_sampler \
    "$@"

`Finetune_trainer.py` results
>test_loss = 7.9716
test_rouge1 = 5.6445
test_rouge2 = 1.6458
test_rougeL = 4.8763
test_rougeLsum = 5.3712

>val_loss = 7.9894
val_rouge1 = 5.0368
val_rouge2 = 1.6249
val_rougeL = 4.1304
val_rougeLsum = 4.6041

`Run_eval.py` arguments
>!python /content/transformers/examples/seq2seq/run_eval.py 
    /content/finetuned_model \
   /content/transformers/cnn_dm_tr/test.source \
   dbart_cnn_12_6_test_gens.txt \
   --reference_path /content/transformers/cnn_dm_tr/test.target \ 
   --score_path dbart_cnn_12_6_test_rouge.json \
   --n_obs 100 \
   --task summarization --bs 2 --fp16

`Run_eval.py` results
>{'rouge1': 0.1091, 'rouge2': 0.0, 'rougeL': 0.1091, 'rougeLsum': 0.1091, 'n_obs': 50, 'runtime': 601, 'seconds_per_sample': 12.02}

### Who can help
@patil-suraj @sshleifer

### Expected behavior
Be able to get high and close rouge scores for `run_eval.py` and `finetune_trainer.py` when I use `mbart-large-cc25` as model and tokenizer.