

TITLE
wrong decoder_input_ids[:,0] for MarianMT models ?

COMMENTS
4

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.3.1
- Platform: Linux
- Python version: 3.7.0
- PyTorch version (GPU?): 1.6.0
- Using GPU in script?: Yes

### Who can help: @sshleifer 

## Information

Model I am using (Bert, XLNet ...): MarianMTModel

The problem arises when using:
* [x] the official example scripts: (give details below)

```from transformers import MarianTokenizer
tok = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-en-de')
src_texts = [ "I am a small frog.", "Tom asked his teacher for advice."]
tgt_texts = ["Ich bin ein kleiner Frosch.", "Tom bat seinen Lehrer um Rat."]  # optional
batch_enc: BatchEncoding = tok.prepare_seq2seq_batch(src_texts, tgt_texts=tgt_texts)
# model(**batch) should work
```

model(**batch)  doesn't work as intended because [shift_tokens_right](https://github.com/huggingface/transformers/blob/03ec02a667d5ed3075ea65b9f89ef7135e97f6b4/src/transformers/modeling_bart.py#L226) adds eos token to generate the target sequence.

```
shift_tokens_right(batch["labels"], model.config.pad_token_id)
```
returns
```
[[0,   105,   495,    53,  5324, 17279,   649,     3],
 [0,  2136,  8818,   715,  5832,    91,   688,     3]]
```
instead of
```
[[58100,   105,   495,    53,  5324, 17279,   649,     3],
 [58100,  2136,  8818,   715,  5832,    91,   688,     3]]
```
Here, "58100" is the decoder_start_token_id.