Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

By Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, Franccois Fleuret. 2020

2,953 citations recorded by the source corpus.

Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Quoc V. Le
  4. Luke S. Zettlemoyer
  5. Yoshua Bengio
  6. Percy Liang
  7. Dacheng Tao
  8. Noah A. Smith

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Ming-Hsuan Yang
  4. Xiaogang Wang
  5. Trevor Darrell
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Yi Yang

In Model Architectures

  1. Quoc V. Le
  2. Jianfeng Gao
  3. Tri Dao
  4. Yoshua Bengio
  5. Christopher R'e
  6. Dacheng Tao
  7. Yi Tay
  8. Jianwei Yang

In Efficiency & Compression

  1. Song Han
  2. Yunhe Wang
  3. Quoc V. Le
  4. Luke S. Zettlemoyer
  5. Herv'e J'egou
  6. Tri Dao
  7. Chuang Gan
  8. Han Cai

Citations in the corpus