

TITLE
T5: forward and generate produce different results even for greedy decoding of a single token

COMMENTS
4

REACTIONS
+1: 2
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Tagging @sshleifer following earlier discussions. 

## Environment info
     
- `transformers` version: 3.2.0 (also 3.0.2)
- Platform: Linux-4.4.0-17763-Microsoft-x86_64-with-glibc2.10  (WSL, but also on normal Ubuntu)
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: Fails independently of this
- Using distributed or parallel set-up in script?: Fails independently of this

## Information

I am using the T5 model to train on a seq2seq task. The model.forward() and model.generate() can differ, even for greedy decoding of a single token, since model.generate() seems to add a padding token before generating. This matters for "classification-style" tasks where we usually decode a single token (positive/negative for sentiment for instance).


The problem arises when using:
* [X] my own modified scripts: (give details below)

Very similar to example: this code is enough:
```python

from transformers import T5Tokenizer, T5ForConditionalGeneration
import torch

tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = T5ForConditionalGeneration.from_pretrained("t5-small")


def run_comparison(input_sent):
    input_ids = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(input_sent))
    input_ids = torch.tensor([input_ids])

    target_ids = tokenizer.convert_tokens_to_ids(tokenizer.tokenize("chocolate </s>"))
    target_ids = torch.tensor([target_ids])

    res_fwd = torch.argmax(model(input_ids=input_ids, labels=target_ids)[1], -1)
    res_gen = model.generate(input_ids=input_ids, max_length=2)
    
    print("Running comparison for %s" % input_sent)
    print("Using model.forward(): ", tokenizer.decode(res_fwd[0]), res_fwd[0])
    print("Using model.generate(): ", tokenizer.decode(res_gen[0]), res_gen[0])

run_comparison("I love </s>")
```

Outputs:

Running comparison for I love </s>
Using model.forward():  and tensor([ 3, 11])
Using model.generate():   tensor([0, 3])


## To reproduce

Steps to reproduce the behavior:

1. Run code above.

## Expected behavior

model.forward() and model.generate() give the same output for single token greedy decoding. 
