

TITLE
GPT2DoubleHeadsModel documentation example question (error in documentation)?

COMMENTS
1

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Hello,
I was reading the documentation for the GPT2DoubleHeadsModel, and I have a question.
In the documentation, the example shown is:
```python
import torch
from transformers import GPT2Tokenizer, GPT2DoubleHeadsModel
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2DoubleHeadsModel.from_pretrained('gpt2, return_dict=True)

# Add a [CLS] to the vocabulary (we should train it also!)
num_added_tokens = tokenizer.add_special_tokens({'cls_token': '[CLS]'})
embedding_layer = model.resize_token_embeddings(len(tokenizer))  # Update the model embeddings with the new vocabulary size
choices = ["Hello, my dog is cute [CLS]", "Hello, my cat is cute [CLS]"]
encoded_choices = [tokenizer.encode(s) for s in choices]
cls_token_location = [tokens.index(tokenizer.cls_token_id) for tokens in encoded_choices]
input_ids = torch.tensor(encoded_choices).unsqueeze(0)  # Batch size: 1, number of choices: 2
mc_token_ids = torch.tensor([cls_token_location])  # Batch size: 1
outputs = model(input_ids, mc_token_ids=mc_token_ids)
lm_logits = outputs.lm_logits
mc_logits = outputs.mc_logits
```
Now, I don't see any main question statement in this example, although I see two multiple-choice options. The way I used to use `GPT2DoubleHeadsModel` is I first did `tokenizer(question_statement, option_statement)` and do `encoded_dict['input_ids']` to extract `input_ids` and similarly `encoded_dict['token_type_ids']` to extract the `token_type_ids`. Has this changed? I am getting an impression that the example is wrong (maybe the example could apply for BERT, but not GPT2DoubleHeadsModel). Is this an error in the documentation? I thought that, since GPT-2 does causal language modeling, question statement and option statement has to be encoded together, and put the `[CLS]` token at the end (usually), so that GPT-2 can apply the causal language modeling process to solve the multiple-choice problem.

Reading this again, I think the examples for BERTForMultipleChoice and GPT2DoubleHeadsModel are flipped.

 Thanks,