Reformer: The Efficient Transformer
By Nikita Kitaev, Lukasz Kaiser, Anselm Levskaya. 2020
3,006 citations recorded by the source corpus.
Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of Transformers. For one, we replace dot-product attention by one that uses locality-sensitive hashing, changing its complexity from O($L^2$) to O($L\log L$), where $L$ is the length of the sequence. Furthermore, we use reversible residual layers instead of the standard residuals, which allows storing activations only once in the training process instead of $N$ times, where $N$ is the number of layers. The resulting model, the Reformer, performs on par with Transformer models while being much more memory-efficient and much faster on long sequences.
Subfields: Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Alan L. Yuille
In Model Architectures
In Efficiency & Compression
- Tri Dao
- Song Han
- Kurt Keutzer
- Dan Alistarh
- Quoc V. Le
- Luke S. Zettlemoyer
- Mohammad Rastegari
- Zi-Mei Yao
Citations in the corpus
- PaLM: Scaling Language Modeling with Pathways (2022, background): “Sparsity in the sequence length is an area that allows for training efficiently with extremely long sequences (Zaheer et al., 2020; Tay et al., 2020; Choromanski et al., 2020; Kitaev et al., 2020; Roy et al., 2020; Child et al., 2019; Gale et al., 2019).”
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “Instead, memory storage/movement is the limiting factor in image classi-fication (Gomez et al., 2017), semantic segmentation (Chen et al., 2017), language modeling (Kitaev et al., 2020), and other tasks (Jain et al., 2020).”
- Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, background): “, 2021), efficient attention (Tay et al., 2020c; Kitaev et al., 2020; Choromanski et al., 2020; Tay et al., 2020b; Ainslie et al., 2020; Jaegle et al., 2021), architecture search (So et al.”
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (2021, methodology): “Sparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020).”
- Persistent Anti-Muslim Bias in Large Language Models (2021, background): “These languagemodels have evolved, from learning individual word vectors with single-layer models [8], to more complex language generation architectures such as recurrent neural networks [5] and most recently transformers [4, 6, 14].”
- Rethinking Attention with Performers (2020, methodology): “We emphasize that in principle, FAVOR+ can also be combined with other techniques, such as reversible layers (Kitaev et al., 2020) or cluster-based attention (Roy et al.”
- Big Bird: Transformers for Longer Sequences (2020, methodology): “[50] further reduced the complexity to O(N log(N)) by using LSH to compute nearest neighbors.”
- Efficient Content-Based Sparse Attention with Routing Transformers (2020, methodology): “While similar in spirit to our approach, the approach of Kitaev et al. (2020) keeps the randomly initialized hyper-planes fixed throughout, while we use mini-batch spherical k-means to learn the space-partitioning centroids.”
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, methodology): “Conceptually, both FlashAttention and Rabe and Staats [66] operate on blocks of the attention matrix using the well-established technique of tiling (or softmax scaling) [51, 60].”
- Self-attention Does Not Need $O(n^2)$ Memory (2021, methodology): “So the space complexity of transformers is a particular concern, c.f. Kitaev et al. (2020); Roy et al. (2021); Zaheer et al. (2020).”
- Efficiently Scaling Transformer Inference (2022, background): “Several approaches (Gupta and Agrawal, 2020) to improve the inference efficiency of Transformer models focus on model architecture improvements, for example efficient attention layers (Roy et al., 2020; Choromanski et al., 2020; Kitaev et al., 2020; Sukhbaatar et al., 2019; Child et al., 2019), dist”
- Longformer: The Long-Document Transformer (2020, methodology): “06 Reformer (Kitaev et al., 2020) - - 1.”
- Landmark Attention: Random-Access Infinite Context Length for Transformers (2023, methodology): “Finally, we point out the aforementioned data structures can also be applied on top of our method to search for relevant blocks.”
- HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “Reformer (Kitaev et al., 2020) Longformer (Beltagy et al.”
- Adaptive Semiparametric Language Models (2021, background): “, 2020), and reformer (Kitaev et al., 2020)—can be seen as models of working memory.”
- Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “From the computational perspective, the all-to-all nature of BERT’s attention patterns at each transformer encoder layer means that it exhibits quadratic complexity in both time and space with respect to input length [Kitaev et al., 2020], and so simply throwing more hardware at the problem (e.”
- It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, background): “While there are some variants of the Transformer that can deal with much longer contexts (e.g., Kitaev et al., 2020; Beltagy et al., 2020), it has yet to be investigated to what extent such models are able to actually make good use of priming examples over such a long context span.”
- Recipes for Building an Open-Domain Chatbot (2020, background): “While several recent works have extended neural architectures to possess longer contexts (Dai et al., 2019; Rae et al., 2020; Kitaev et al., 2020; Beltagy et al., 2020), we have neither implemented those, nor do we believe the current evaluation setup is the right one for measuring their success.”