

TITLE
Connot correctly fine-tune Bert for generation

COMMENTS
7

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# 📚 Migration

## Information

<!-- Important information -->

Model I am using Bert for generation (a Seq2Seq style):

The problem arises when using:
* The official example scripts from : [transformers/examples/legacy/seq2seq/finetune_trainer.py](https://github.com/huggingface/transformers/blob/master/examples/legacy/seq2seq/finetune_trainer.py)
```
config = AutoConfig.from_pretrained(model_args.config_name)
tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_args.model_name_or_path)
```

The above is the official code to fine-tune a text generation model such as BART (i.e., just specify model_name_or_path as facebook/BART-base). **I am trying to use the BERTGEN instead of using BART of T5.** So, I modified the above code into the following one.

* My own modified scripts: 
```
config = BertConfig.from_pretrained("bert-large-uncased")
tokenizer = BertTokenizer.from_pretrained("bert-large-uncased")
encoder = BertGenerationEncoder.from_pretrained("bert-large-uncased", bos_token_id=101, eos_token_id=102)
decoder = BertGenerationDecoder.from_pretrained("bert-large-uncased", add_cross_attention=True, is_decoder=True, bos_token_id=101, eos_token_id=102)
model = EncoderDecoderModel(encoder=encoder, decoder=decoder)
```

The tasks I am working on is:
* An official task: Gigaword, CNN/DailyMail


## Details

Since the examples in the github repo do not contain `Bert for generation`, so I adopt the code from the documents [here](https://huggingface.co/transformers/model_doc/bertgeneration.html). The only modification has been shown as aboves. However, the model performance cannot reach the reported performance in their [paper](https://arxiv.org/pdf/1907.12461.pdf), even has been left by a large margin. I felt the model has not been correctly trained. 

**I am wondering whether there is an example code for using Bert for generation. Thank you very much for your help in advance.**

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
 
- `transformers` version: 4.7.0
- Python version: 3.6
- PyTorch version (GPU?): 1.8
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

<!-- IMPORTANT: which version of the former library do you use? -->
* `pytorch-transformers` or `pytorch-pretrained-bert` version (or branch): 4.7.0

