text: 
  
  TITLE
  Masking in Pooling Layer from BERT Output?
  
  COMMENTS
  1
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  In Keras when the embedding layers using masking it is propagate to layer afterwards like Pooling or RNN-Layers. I wonder if this holds for using transformers Bert Models? I.e. in the following are the attention masks used in the following Pooling as masking too? So, the averages do not include padding tokens?
  
      
  ```
      id_ = Input((MAX_SEQUENCE_LENGTH,), dtype=tf.int32)
      mask_ = Input((MAX_SEQUENCE_LENGTH,), dtype=tf.int32)
      atn_ = Input((MAX_SEQUENCE_LENGTH,), dtype=tf.int32)
      
  
      bert_model = TFBertModel.from_pretrained('bert-base-uncased')
      
      embedding = bert_model(id_, attention_mask=mask_, token_type_ids=atn_)[0]
      
      x = GlobalAveragePooling1D()(embedding)    #are here attention_mask used as mask?
      x = Dropout(0.2)(x)
      out = Dense(3, activation='softmax')(x)
  
      model = Model(inputs=[id_, mask_, atn_], outputs=out)
      model.compile(loss='sparse_categorical_crossentropy', optimizer=opt)
  ```
  
  
  https://discuss.huggingface.co/t/bert-output-for-padding-tokens/1550