

TITLE
Generate does not take into account config.decoder.eos_token_id

COMMENTS
5

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
As reported by some people (see https://github.com/NielsRogge/Transformers-Tutorials/issues/53 and on the [forum](https://discuss.huggingface.co/t/trocr-repeated-generation/12361)), the `generate()` method currently does not take into account `config.decoder.eos_token_id`, only` config.eos_token_id` to properly stop generation.

Hence, models that are made using `EncoderDecoderModel`/`VisionEncoderDecoderModel`/`SpeechEncoderDecoderModel` will not properly stop generation if `config.eos_token_id` is not set.

cc @patrickvonplaten @patil-suraj 