

TITLE
Difference between CLS hidden state and pooled_output

COMMENTS
9

REACTIONS
+1: 6
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Hi,

The first ouput of the TFBertModel is last_hidden_state. And I assume the CLS embdding is the first element of this object, so last_hidden_state [0]?

But then you have also the pooled_output. In the docs it is written that this comes from a linear layer on top.

1. This comes originally from the pre training. Can I imagine that this model for pre training is something like the BertForSequenceClassification?
2. Can this pooled_output be fine_tune when you fine tuning the weights of the BERT Model? I assume that this will be fixed and just the hidden states are fine tunable?