

TITLE
LXMERT cross_modality_matching logits order in code vs. documentation

COMMENTS
1

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Issue #7266 evolved into an entirely different issue described below.

# Possible documentation error
Independently of considering #8333 or not, it is clear that the results of the cross-modality matching in LXMERT (for example in [this demo](https://colab.research.google.com/drive/18TyuMfZYlgQ_nXo-tr8LCnzUaoX0KS-h?usp=sharing)) are better (below random chance vs. above random chance) when the first logit of the `output_lxmert['cross_relationship_score']` *– (torch.FloatTensor of shape (batch_size, 2))* represents *a mismatch* rather than *a match*.

This **contradicts the [documentation](https://huggingface.co/transformers/model_doc/lxmert.html)** which in my understanding assigns the first logit to `is_match` (True):
> cross_relationship_score – (torch.FloatTensor of shape (batch_size, 2)): Prediction scores of the textual matching objective (classification) head (scores of True/False continuation before SoftMax).

@LysandreJik, @eltoto1219: Is the documentation right or wrong? How was the model trained and which logit was intended to predict the match and which one the mismatch?

I would be very happy to have a clear answer about the order of the logits in training and code and its correspondence to the documentation. I know how easy it is to flip the logits around and choose the one delivering better results on the data at hand. But in my understanding of scientific conduct, I can not just choose this by wishful thinking and ignoring the documentation: there might be other things going on exhibiting this behavior.

Thank you in advance!