

TITLE
mBART finetuned on XSUM

COMMENTS
3

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Environment info

- `transformers` version: 4.1.0.dev0
- Platform: Linux-5.4.0-58-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.0.dev20201216+cu110 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: distributed


### Who can help

 mBART: @patrickvonplaten
 examples/seq2seq: @patil-suraj


## Information

Model I am using: mBART

The problem arises when using:
* [ X ] the official example scripts:
 I used the official seq2seq training example here:
 https://github.com/huggingface/transformers/tree/master/examples/seq2seq
* [ X ] my own modified scripts: (give details below)
my training script is as follows (no changes to finetune_trainer.py):
```shell
python -m torch.distributed.launch --nproc_per_node=2 finetune_trainer.py \
    --data_dir "./xsum" \
    --output_dir "./my_models" \
    --overwrite_output_dir \
    --model_name_or_path "facebook/mbart-large-cc25" \
    --fp16 \
    --freeze_encoder \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --learning_rate=3e-5 \
    --do_train --do_eval --do_predict \
    --evaluation_strategy steps \
    --predict_with_generate \
    --n_val 1000 \
    --max_target_length=60 \
    --val_max_target_length=60 \
    --test_max_target_length=100 \
    "$@"
```
The tasks I am working on is:
* [ X ] XSUM


## To reproduce

Steps to reproduce the behavior:

1. follow https://github.com/huggingface/transformers/blob/master/examples/seq2seq/README.md
2. launch training script with my modifications (batch_size, freeze_encoder, max_target_length ..)
3. inference on two texts (french and english) using the following code:
```python
def sum_mbart_xsum(text):
    print("---------------------------------------------------------------------------------")
    print("                                MBART large xsum                                  ")
    print("---------------------------------------------------------------------------------")

    tokenizer = MBartTokenizer.from_pretrained("/home/mohamed/Desktop/Summarization/mbart-xsum")
    model = MBartForConditionalGeneration.from_pretrained("/home/mohamed/Desktop/Summarization/mbart-xsum")

    article_input_ids = tokenizer.batch_encode_plus([text], return_tensors='pt', max_length=1024, truncation=True)[
        'input_ids']
    summary_ids = model.generate(article_input_ids,
                                 num_beams=6,
                                 length_penalty=1.0,
                                 max_length=142,
                                 no_repeat_ngram_size=3)

    summary_txt = tokenizer.decode(summary_ids.squeeze(), skip_special_tokens=True)
    return summary_txt

```
## Results
1- eval/test results:
```jsonc
{
    "epoch": 3.0,
    "test_gen_len": 28.1,
    "test_loss": 1.7692,
    "test_n_ojbs": -1,
    "test_rouge1": 32.7618,
    "test_rouge2": 12.022,
    "test_rougeL": 25.6512,
    "test_rougeLsum": 25.6499,
    "test_runtime": 2778.8939,
    "test_samples_per_second": -0.0,
    "train_n_ojbs": -1,
    "train_runtime": 94633.1507,
    "train_samples_per_second": -0.0,
    "val_gen_len": 28.0,
    "val_loss": 1.7993,
    "val_n_ojbs": 1000,
    "val_rouge1": 32.9862,
    "val_rouge2": 11.528,
    "val_rougeL": 25.6517,
    "val_rougeLsum": 25.7055,
    "val_runtime": 267.0092,
    "val_samples_per_second": 3.745
}
```
2- Inference:
* Out of context summarizations (gives sth related to training data) --> (sth wrong with my finetuning configuration or inference function?)
* For French texts, results are in English and poor summary --> (why the language changes?)

## Expected behavior

My main objective of finetuning mBART on Xsum is to evaluate the multilingual level of mBART. Basically answering the following question: Should I finetune on a dataset with multiple languages to be able to summarize in multiple languages? Or the multilingual characteristic is preserved with mBART and just finetune on english (xsum) only
Current problems:
1- (inference results and questions above)
2- Why `facebook/bart-large-xsum` understands french (even if bart is trained on english)?
