Efficient Transformers: A Survey

By Yi Tay, Mostafa Dehghani, Dara Bahri, Donald Metzler. 2020

1,579 citations recorded by the source corpus.

Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision, and reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable staple in the modern deep learning stack. Recently, a dizzying number of “X-former” models have been proposed—Reformer, Linformer, Performer, Longformer, to name a few—which improve upon the original Transformer architecture, many of which make improvements around computational and memory efficiency. With the aim of helping the avid researcher navigate this flurry, this article characterizes a large and thoughtful selection of recent efficiency-flavored “X-former” models, providing an organized and comprehensive overview of existing work and models across multiple domains.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, background): “…(Lan et al., 2019; Tay et al., 2019; Zhang et al., 2021), efficient attention (Tay et al., 2020c; Kitaev et al., 2020; Choromanski et al., 2020; Tay et al., 2020b; Ainslie et al., 2020; Jaegle et al., 2021), architecture search (So et al., 2019), alternative non Transformer-based architectures…”
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, background): “While recurrent neural networks and modified attention mechanisms (Tay et al., 2020) can claim a better computational complexity in the sequence length, the cost nevertheless always scales up as sequences get longer.”
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, methodology): “[78] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler.”
SCROLLS: Standardized CompaRison Over Long Language Sequences (2022, methodology): “To address this problem, a wide variety of efficient alternatives and approximations have been proposed over the past couple of years (Tay et al., 2020b; Fournier et al., 2021).”
A Review of Sparse Expert Models in Deep Learning (2022, background): “Finally, Tay et al. (2020) surveys a broader set of methods aimed at increasing the computational efficiency of Transformers, of which, sparse expert models are one promising approach.”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “The Perceiver IO architecture relies on the same primitives as Transformers: so why aren’t Transformers all you need? The answer is that Transformers scale very poorly in both compute and memory [82].”
Perceiver: General Perception with Iterative Attention (2021, neutral): “The primary contribution of this body of work is a set of modules with similar ﬂexibility to the Transformer’s densely-connected self-attention block, but at sub-quadratic computational cost (see Tay et al. 2020; 2021b for more detailed reviews).”
Sparse is Enough in Scaling Transformers (2021, methodology): “Luckily, a number of methods have been proposed to solve this problem for Transformers, see [41] for a survey.”
The Efficiency Misnomer (2021, background): “That said, it is still common practice to parameter-match models to make ‘fair’ comparisons (Mehta et al., 2020; Lee-Thorp et al., 2021; Tay et al., 2020a; Xue et al., 2021; Wightman et al., 2021), even if one model is in reality slower or faster than another.”
Deformable DETR: Deformable Transformers for End-to-End Object Detection (2020, background): “Recently, many efforts have been made to address this problem (Tay et al., 2020b), which can be roughly divided into three categories in practice.”
Long Range Arena: A Benchmark for Efficient Transformers (2020, background): “For the sake of brevity, we refer interested readers to (Tay et al., 2020c) for a detailed survey of this line of research.”
Pre-Trained Models: Past, Present and Future (2021, background): “Despite the bunch of modifications on approximate computing of the quadratic growing pointwise attention (Tay et al., 2020), a question is that we humans do not present such a long-range attention mechanism.”
Transformers in Vision: A Survey (2021, background): “Given the topological map and natural language inputs, a VLN task using the Transformer model bears resemblance to sequence prediction in NLP. Specifically, at each time instance, the cross-modal Transformer predicts a single node of the topological map in the navigation plan.”
ViViT: A Video Vision Transformer (2021, background): “Many variants of the transformer have also been proposed to reduce the computational cost of selfattention when processing longer sequences [10, 11, 36, 61, 62, 72] and to improve parameter efficiency [39, 14].”
LambdaNetworks: Modeling Long-Range Interactions Without Attention (2021, background): “…Parmar et al., 2018; Ramachandran et al., 2019) • axial attention patterns (Ho et al., 2019; Wang et al., 2020a; Shen et al., 2020) • static sparse attention patterns (Child et al.; Beltagy et al., 2020) • dynamic sparse attention patterns (Kitaev et al., 2020) See Tay et al. (2020) for a review.”
Bottleneck Transformers for Visual Recognition (2021, background): “(2) The memory and computation for self-attention scale quadratically with spatial dimensions [58], causing overheads for training and inference.”
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding (2021, background): “Readers may refer to [39] and [38] for a comprehensive survey and benchmarks, respectively.”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, neutral): “A good survey paper is Tay et al. (2020).”
Random Feature Attention (2021, background): “A large body of research has been devoted to improving their time and memory efficiency (Tay et al., 2020c).”
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (2021, background): “Many ef-ﬁcient self-attention models have been proposed (Choromanski et al., 2020; Wang et al., 2020; Zaheer et al., 2020) to tackle this problem; see (Tay et al., 2020b;a) for a comprehensive overview.”