

TITLE
TFBertModel not working at all with any type of model

COMMENTS
6

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: macOS
- Python version: 3.7
- PyTorch version (GPU?): NO
- Tensorflow version (GPU?): 2.3.1
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): Bert "dbmdz/bert-base-italian-xxl-cased"

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

<---------------------------------------------->

Hi @LysandreJik , @stefan-it , @sgugger , I'm trying to use `dbmdz/bert-base-italian-xxl-cased`for creating a keras model for a classification task.
I've followed the documentation but I continue to receive the following error:

```
tensorflow.python.framework.errors_impl.InvalidArgumentError:  indices[5,0] = 102 is not in [0, 2)
	 [[node functional_1/bert/embeddings/token_type_embeddings/embedding_lookup (defined at /anaconda3/envs/profanity-detector/lib/python3.7/site-packages/transformers/modeling_tf_bert.py:186) ]] [Op:__inference_train_function_29179]
```

This is the model:

```
from transformers import TFBertModel, BertTokenizer

bert_model = TFBertModel.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
tokenizer = BertTokenizer.from_pretrained("dbmdz/bert-base-italian-xxl-cased")

input_ids = tf.keras.layers.Input(shape=(constants.MAX_SEQ_LENGTH,), dtype=tf.int32)
token_type_ids = tf.keras.layers.Input(shape=(constants.MAX_SEQ_LENGTH,), dtype=tf.int32)
attention_mask = tf.keras.layers.Input(shape=(constants.MAX_SEQ_LENGTH,), dtype=tf.int32)

seq_output, _ = bert_model({
        "input_ids": input_ids,
        "token_type_ids": token_type_ids,
        "attention_mask": attention_mask
})

pooling = tf.keras.layers.GlobalAveragePooling1D()(seq_output)
dropout = tf.keras.layers.Dropout(0.2)(pooling)
output = tf.keras.layers.Dense(constants.CLASSES, activation="softmax")(dropout)

model = tf.keras.Model(
        inputs=[input_ids, token_type_ids, attention_mask],
        outputs=[output]
)

model.compile(optimizer=tf.optimizers.Adam(lr=0.00001), loss='sparse_categorical_crossentropy', metrics=['accuracy'])
```

My dataset is tokenized by this method:
```
    def map_to_dict(self, input_ids, attention_masks, token_type_ids, labels):
        return {
            "input_ids": input_ids,
            "token_type_ids": token_type_ids,
            "attention_mask": attention_masks,
        }, labels

    def tokenize_sequences(self, tokenizer, max_sequence_length, data, labels):
        try:
            token_ids = []
            token_type_ids = []
            attention_mask = []

            for sentence in data:
                bert_input = tokenizer.encode_plus(
                    sentence,
                    add_special_tokens=True,  # add [CLS], [SEP]
                    max_length=max_sequence_length,  # max length of the text that can go to BERT
                    truncation=True,
                    pad_to_max_length=True,  # add [PAD] tokens
                    return_attention_mask=True  # add attention mask to not focus on pad tokens
                )

                token_ids.append(bert_input["input_ids"])
                token_type_ids.append(bert_input["token_type_ids"])
                attention_mask.append(bert_input["attention_mask"])

            return tf.data.Dataset.from_tensor_slices((token_ids, token_type_ids, attention_mask, labels)).map(self.map_to_dict)
        except Exception as e:
            stacktrace = traceback.format_exc()

            logger.error("{}".format(stacktrace))

            raise e

ds_train_encoded = tokenize_sequences(tokenizer, 512, X_train, y_train).shuffle(10000).batch(6)
```
X_train examples:
```
["Questo video è davvero bellissimo", "La qualità del video non è proprio il massimo"......]
```

y_train examples:
```
[[1], [0]...]
```

I continue to receive the error described before.

```
tensorflow.python.framework.errors_impl.InvalidArgumentError:  indices[5,0] = 102 is not in [0, 2)
	 [[node functional_1/bert/embeddings/token_type_embeddings/embedding_lookup (defined at /anaconda3/envs/profanity-detector/lib/python3.7/site-packages/transformers/modeling_tf_bert.py:186) ]] [Op:__inference_train_function_29179]
```

If I try to use TFBertForSequenceClassification everything works fine (for this reason I'm excluding tokenization problems).

Can you please provide a solution or a well documented guide for using TFBertModel class with Keras model (I cannot find it)?

Thank you
