

TITLE
No loss in model output for TFElectraForPreTraining

COMMENTS
1

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
No loss in model output for TFElectraForPreTraining, even though labels are provided.
This is from docs for ELECTRA:
`loss (optional, returned when labels is provided, tf.Tensor of shape (1,)) – Total loss of the ELECTRA objective.`

The problem is that I want to get loss and logits, but model outputs only logits.

My code snippet:
```
>>> import numpy as np
>>> from transformers import ElectraTokenizer, TFElectraForPreTraining
>>>
>>> model = TFElectraForPreTraining.from_pretrained('google/electra-base-discriminator')
All model checkpoint layers were used when initializing TFElectraForPreTraining.

All the layers of TFElectraForPreTraining were initialized from the model checkpoint at google/electra-base-discriminator.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFElectraForPreTraining for predictions without further training.
>>> tokenizer = ElectraTokenizer.from_pretrained('google/electra-base-discriminator')
>>> inputs = tokenizer('simple phrase', return_tensors='tf')
>>> labels = np.array([0] * 4)
>>> out = model(inputs, labels=labels)
>>> len(out)
1
```