

TITLE
Unable to pass encoder_outputs to generate calls

COMMENTS
5

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: Github Main branch

### Who can help
 TextGeneration: @TevenLeScao 
 

## Information

Model I am using (Bert, XLNet ...): T5

I'm unable to pass preconputed encoder_outputs to the `.generate()` method.

I've tried defining:

```python
model_kwargs = {"encoder_outputs": encoder_outputs}
output = model.generate(model_kwargs)
```
But I've noticed some validation errors for `input_ids`. Even if I replace input_ids with dummy values, I've noticed the model_kwargs is always replaced here:

https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L448

I think it can be fixed (without optimization) by just replacing:

```python
model_kwargs["encoder_outputs"] = encoder_outputs
```

with:

```python
model_kwargs.setdefault("encoder_outputs", encoder_outputs)
```

If you agree I can try to open a PR to fix this.

Best,