

TITLE
Use of global attention of Longformer when generating

COMMENTS
3

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
I'm training Longformer2Roberta, the encoder part of this Seq2Seq model is Longformer. The one feature Longformer brings is global attention, I found the use of it during training, but it is never used at the point of generation. I guess it should be used somewhere here https://github.com/huggingface/transformers/blob/03e46c1de3864b8464a1b40d2a414b35f6b7f0df/src/transformers/generation_utils.py#L402. 

I guess @patrickvonplaten is working on these models.