Transformer Quality in Linear Time

By Weizhe Hua, Zihang Dai, Hanxiao Liu, Quoc V. Le. 2022

357 citations recorded by the source corpus.

We revisit the design choices in Transformers, and propose methods to address their weaknesses in handling long sequences. First, we propose a simple layer named gated attention unit, which allows the use of a weaker single-head attention with minimal quality loss. We then propose a linear approximation method complementary to this new layer, which is accelerator-friendly and highly competitive in quality. The resulting model, named FLASH, matches the perplexity of improved Transformers over both short (512) and long (8K) context lengths, achieving training speedups of up to 4.9$\times$ on Wiki-40B and 12.1$\times$ on PG-19 for auto-regressive language modeling, and 4.8$\times$ on C4 for masked language modeling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “LambdaNetworks [2], AFT [97] and FLASH [44] are other attempts at replacing attention in the context of image classiﬁcation and language modeling.”
Long Range Language Modeling via Gated State Spaces (2022, methodology): “A comparison with FLASH (Hua et al., 2022) is included in Appendix C (Table 4).”
RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose (2023, methodology): “We adopt the transformer variant, Gated Attention Unit (GAU) [23], which has faster speed, lower memory cost, and better performance compared to the vanilla transformer [55].”
Parallelizing Linear Transformers with the Delta Rule over Sequence Length (2024, neutral): “The memory-efﬁcient representation makes it possible to straightforwardly extend the chunkwise parallel strategy for training linear attention models [33, 105, 116] to the DeltaNet case.”
Simple Baselines for Image Restoration (2022, background): “GLU or its variants has verified their effectiveness in NLP[29,9,8], and there is a prosperous trend of them in computer vision[30,37,16,19].”
Gated Delta Networks: Improving Mamba2 with Delta Rule (2024, neutral): “It is also interesting to investigate hybrid linear/softmax attention within a single layer (Hua et al., 2022a; Zancato et al., 2024; Munkhdalai et al., 2024; Nunez et al., 2024; Dong et al., 2025; Zhang et al., 2025).”
EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention (2023, background): “Memory access overhead is a critical factor affecting model speed [15,28,31,65].”
Gated Linear Attention Transformers with Hardware-Efficient Training (2023, methodology): “Hua et al. (2022) first propose the chunkwise linear attention form, arguing that the training algorithm of Katharopoulos et al. (2020) is slow in practice.”