

TITLE
[Reformer, Longformer, Roberta, GPT2, CTRL] attention_mask should be at second argument

COMMENTS
1

REACTIONS
+1: 1
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Reformer, Longformer and Roberta have some models where the `attention_mask` is not at the second position. IMO, this was not done on purpose, but sloppy implementation. In order to use `torchscript` with `attention_mask`, the `forward()` args should be refactored. This is a breaking change however.

Additionally, GPT2 and CTRL also don't have `attention_mask` at their 2nd position in the forward pass. One can argue that it's more intuitive to have `past_key_values` at the second position, but leaving it there would also mean that torchscript + `attention_mask` can never really be used with GPT2. I think we should re-order the position_ids here as well, even though this is a big breaking change.

