

TITLE
Forward outputs on multiple sequences is wrong

COMMENTS
0

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
### System Info

```shell
latest version of transformers
pytorch
python 3.10
```


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("t5-small")
model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")
model.resize_token_embeddings(len(tokenizer))
model.to("cuda")
model.eval()



# sequences
seq1 = "summarize: Calling the model (which means the forward method) uses the labels for teacher forcing. This means inputs to the decoder are the labels shifted by one"
output1 = "calling the model uses the labels for teacher forcing. inputs to the decoder"

seq2 = "summarize: When you call the generate method, the model is used in the autoregressive fashion"
output2 = "the model is used in the auto-aggressive fashion."

seq3 = "summarize: However, selecting the token is a hard decision, and the gradient cannot be propagated through this decision"
output3 = "the token is a hard decision, and the gradient cannot be propagated through this decision"

input_sequences = [seq1, seq2, seq3]
output_seq = [output1, output2, output3]

# encoding input and attention mask
encoding = tokenizer(
    input_sequences,
    padding="longest",
    max_length=128,
    truncation=True,
    return_tensors="pt",
)

input_ids, attention_mask = encoding.input_ids.to("cuda"), encoding.attention_mask.to("cuda")

# labels
target_encoding = tokenizer(
    output_seq, padding="longest", max_length=128, truncation=True
)
labels = target_encoding.input_ids
labels = torch.tensor(labels).to("cuda")
labels[labels == tokenizer.pad_token_id] = -100

# Call the models
logits = model(input_ids=input_ids, attention_mask=attention_mask, labels=labels).logits

# Apply softamx() and batch_decode()

X = logits
X = F.softmax(X, dim=-1)
ids = X.argmax(dim=-1)
y = tokenizer.batch_decode(sequences=ids, skip_special_tokens=True)

# results: batch_size=3

['call the model uses the labels for teacher forcing  inputs to the decoder are',
 'the model is used in the auto-aggressive fashion  the the the',
 'the token is a hard decision, and the gradient cannot be propagated through this decision ']

# results: batch_size =1 i.e. consider 1 seq each time

['call the model uses the labels for teacher forcing  inputs to the decoder are']

['the model is used in the auto-aggressive fashion ']

['the token is a hard decision, and the gradient cannot be propagated through this decision ']
```

### Expected behavior

```shell
running model on a batch should give the same result as running on a single sequence
```
