text: 
  
  TITLE
  Add SENet Blocks in Encoding Layers
  
  COMMENTS
  1
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  # 🚀 Feature Request
  
  I read the article "[SesameBERT: Attention for Anywhere](https://arxiv.org/pdf/1910.03176.pdf)" and would like to add SENet blocks in the Huggingface implementation. The article's authors made an implementation with [Tensorflow](https://github.com/ICLR2020Sesame/SesameBert/blob/master/modeling.py), but I would like to use the lib in pytorch.
  
  ## Motivation
  
  The use of ([Squeeze-and-Excitation Networks](https://arxiv.org/abs/1709.01507)) SENet Blocks has obtained state-of-the-art results. And they seem to be promising in NLP.
  
  ## Your contribution
  
  I know that it is possible to modify the [[BertLayer()](https://github.com/huggingface/transformers/blob/61c506349134db0a0a2fd6fb2eff8e29a2f84e79/src/transformers/models/bert/modeling_bert.py#L430)] and [[BertEnconder()](https://github.com/huggingface/transformers/blob/61c506349134db0a0a2fd6fb2eff8e29a2f84e79/src/transformers/models/bert/modeling_bert.py#L513)] classes
  
  Any suggestions on how to modify the code so that you can apply the idea used in the article?
  
  ![image](https://user-images.githubusercontent.com/25322394/120578580-0f3ca000-c3f4-11eb-9a14-93bf610c3a60.png)
  