text: 
  
  TITLE
  Feature Request: El-Attention
  
  COMMENTS
  0
  
  REACTIONS
  +1: 1
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  # 🚀 Feature request
  I've looked into the paper titled "[EL-Attention: Memory Efficient Lossless Attention for Generation](https://arxiv.org/abs/2105.04779)".
  It proposes a method for calculating attention that forgoes creating multi-head attention from the hidden state. This saves computational time and frees memory. 
  
  ## Motivation
  El-attention seems to have no downsides, and promises significant memory and performance gains during training and inference. 
  
  ## Your contribution
  The main difficulty may be in that it requires being added directly in to each model's attention mechanism code, or requires a ton of new subclasses for each part of each model. Maybe an easier solution to this would be a pipeline to use custom  attention mechanism code. 