text: 
  
  TITLE
  Implementing efficient self attention in T5
  
  COMMENTS
  1
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 1
  rocket: 0
  eyes: 0
  
  BODY
  # 🌟 New model addition
  
  My teammates and I (including @ice-americano) would like to use efficient self attention methods such as Linformer, Performer and Nystromformer
  
  ## Model description
  
  These new methods serve as approximations of regular attention, but reduce complexity from quadratic in the inputs to linear.  We would like to add a parameter to T5 where users can specify an efficient attention method to use instead of regular attention.  Ideally, this would be implemented across all models, but the models tend to have varying implementations of attention, rendering this generalization fairly tedious.
  
  ## Open source status
  
  * [x] the model implementation is available: repos are https://github.com/mlpen and https://github.com/lucidrains/performer-pytorch
  * [ ] the model weights are available: N/A
  * [x] who are the authors: @mlpen and @lucidrains
  