

TITLE
Add FAVOR+ / Performer attention

COMMENTS
46

REACTIONS
+1: 55
-1: 0
laugh: 0
hooray: 19
heart: 18
rocket: 16
eyes: 0

BODY
# 🌟 FAVOR+ / Performer attention addition

Are there any plans to add this new attention approximation block to Transformers library?

## Model description
The new attention mechanism with linear time and space complexity was introduced in
 _Rethinking Attention with Performers_ [[https://arxiv.org/abs/2009.14794](https://arxiv.org/abs/2009.14794)].
Authors of the paper claim that the new attention mechanism is backward-compatbile with already existing models 
> Backwards compatibility with pretrained models is available as a benefit from softmax approximation, via small finetuning (required due to error propagation)


<!-- Important information -->

## Open source status

* [x] the model implementation is available: it's an original Trax implementation from Google: https://github.com/google-research/google-research/tree/master/performer/fast_self_attention
* [ ] the model weights are available: probably not required as it's a building block for models rather than fully new architecture
* [x] who are the authors: Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, Adrian Weller
