

TITLE
[XLNet] attention_mask / input_mask - Why two `attention_mask` inputs?

COMMENTS
3

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
For whatever reason XLNet accepts both an `attention_mask` and a `input_mask`. As far as I understand `attention_mask` = 1 - `input_mask`. I don't think having the same input twice (one is the inverse of the other) has any advantage. We should remove `input_mask` IMO (first depreciate, then remove). Also, `attention_mask` should be put in the 2nd positino to make model compatible with torchscript.