

TITLE
Which language is available for EncoderDecoderModel pre-trained model?

COMMENTS
5

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Hi, I have a question.

From the example code of [EncoderDecoderModel](https://huggingface.co/transformers/model_doc/encoderdecoder.html#transformers.EncoderDecoderModel.forward), pre-trained model 'bert-base-uncased' looks trained by EN-FR sentence pairs. Is pre-trained model for decoder is trained in French? How can I load pre-trained model in EN-EN pair? Am I confusing?

```
from transformers import EncoderDecoderModel, BertTokenizer
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased') # initialize Bert2Bert from pre-trained checkpoints

# training
model.config.decoder_start_token_id = tokenizer.cls_token_id
model.config.pad_token_id = tokenizer.pad_token_id
model.config.vocab_size = model.config.decoder.vocab_size

input_ids = tokenizer("Hello, my dog is cute", return_tensors="pt").input_ids
labels = tokenizer("Salut, mon chien est mignon", return_tensors="pt").input_ids
outputs = model(input_ids=input_ids, labels=input_ids)
loss, logits = outputs.loss, outputs.logits

# save and load from pretrained
model.save_pretrained("bert2bert")
model = EncoderDecoderModel.from_pretrained("bert2bert")

# generation
generated = model.generate(input_ids, decoder_start_token_id=model.config.decoder.pad_token_id)
```