Reformer: The Efficient Transformer

By Nikita Kitaev, Lukasz Kaiser, Anselm Levskaya. 2020

3,006 citations recorded by the source corpus.

Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of Transformers. For one, we replace dot-product attention by one that uses locality-sensitive hashing, changing its complexity from O($L^2$) to O($L\log L$), where $L$ is the length of the sequence. Furthermore, we use reversible residual layers instead of the standard residuals, which allows storing activations only once in the training process instead of $N$ times, where $N$ is the number of layers. The resulting model, the Reformer, performs on par with Transformer models while being much more memory-efficient and much faster on long sequences.

Subfields: Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Ming-Hsuan Yang
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Alan L. Yuille

In Model Architectures

  1. Quoc V. Le
  2. Yoshua Bengio
  3. Jianfeng Gao
  4. Tri Dao
  5. Mohit Bansal
  6. Alan L. Yuille
  7. Yi Tay
  8. Christopher R'e

In Efficiency & Compression

  1. Tri Dao
  2. Song Han
  3. Kurt Keutzer
  4. Dan Alistarh
  5. Quoc V. Le
  6. Luke S. Zettlemoyer
  7. Mohammad Rastegari
  8. Zi-Mei Yao

Citations in the corpus