

TITLE
Long-Short Transformer

COMMENTS
3

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# 🌟 New model addition

## Model description

https://arxiv.org/abs/2107.02192

In this paper, they propose Long-Short Transformer, an efficient self-attention mechanism for modeling long sequences with linear complexity for both language and vision tasks. It aggregates a novel long-range attention with dynamic projection to model distant correlations and a short-term attention to capture fine-grained local correlations. Transformer-LS can be applied to both autoregressive and bidirectional models without additional complexity.

## Open source status

* [x] the model implementation is available: https://github.com/NVIDIA/transformer-ls
* [x] the model weights are available: https://github.com/NVIDIA/transformer-ls
* [x] who are the authors: Chen Zhu (@zhuchen03) and Wei Ping and Chaowei Xiao and Mohammad Shoeybi and Tom Goldstein and Anima Anandkumar and Bryan Catanzaro (NVIDIA, University of Maryland)