FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
By Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher R'e. Neural Information Processing Systems, 2022
4,798 citations recorded by the source corpus.
Transformers are slow and memory-hungry on long sequences, since the time and memory complexity of self-attention are quadratic in sequence length. Approximate attention methods have attempted to address this problem by trading off model quality to reduce the compute complexity, but often do not achieve wall-clock speedup. We argue that a missing principle is making attention algorithms IO-aware -- accounting for reads and writes between levels of GPU memory. We propose FlashAttention, an IO-aware exact attention algorithm that uses tiling to reduce the number of memory reads/writes between GPU high bandwidth memory (HBM) and GPU on-chip SRAM. We analyze the IO complexity of FlashAttention, showing that it requires fewer HBM accesses than standard attention, and is optimal for a range of SRAM sizes. We also extend FlashAttention to block-sparse attention, yielding an approximate attention algorithm that is faster than any existing approximate attention method. FlashAttention trains Transformers faster than existing baselines: 15% end-to-end wall-clock speedup on BERT-large (seq. length 512) compared to the MLPerf 1.1 training speed record, 3$\times$ speedup on GPT-2 (seq. length 1K), and 2.4$\times$ speedup on long-range arena (seq. length 1K-4K). FlashAttention and block-sparse FlashAttention enable longer context in Transformers, yielding higher quality models (0.7 better perplexity on GPT-2 and 6.4 points of lift on long-document classification) and entirely new capabilities: the first Transformers to achieve better-than-chance performance on the Path-X challenge (seq. length 16K, 61.4% accuracy) and Path-256 (seq. length 64K, 63.1% accuracy).
Subfields: Information Systems and Management, Model Architectures, Efficiency & Compression, Human-Computer Interaction
View Paper: https://arxiv.org/abs/2205.14135
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Systems and Management
In Model Architectures
- Mohit Bansal
- Yoshua Bengio
- Gabriel Synnaeve
- Furu Wei
- Quoc V. Le
- Christoph Feichtenhofer
- Tri Dao
- Armand Joulin
In Efficiency & Compression
- Song Han
- Kurt Keutzer
- Tri Dao
- Armand Joulin
- Herv'e J'egou
- Gabriel Synnaeve
- Martin Jaggi
- Edouard Grave
In Human-Computer Interaction
- Tri Dao
- Limin Wang
- Yu Qiao
- Yann LeCun
- Ming-Hsuan Yang
- Alexander G. Hauptmann
- Jitendra Malik
- Hamed Pirsiavash
Citations in the corpus
- LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “First, we use an efficient implementation of the causal multi-head attention operator, inspired by Rabe and Staats (2021) and Dao et al. (2022).”
- Mistral 7B (2023, methodology): “In practice, for a sequence length of 16K and W = 4096 , changes made to FlashAttention [11] and xFormers [18] yield a 2x speed improvement over a vanilla attention baseline.”
- Self-attention Does Not Need $O(n^2)$ Memory (2021, background): “Dao et al. (2022) provide a CUDA implementation of memory-efficient attention and demonstrate that the reduced memory requirements can translate to significant speedups on GPUs.”
- The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “We use FlashAttention (Dao et al., 2022) in a custom codebase.”
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023, background): “Recently, decoder-only models are extremely popular, and since these models do not have separate self-attention and cross-attention, we expect GQA to have a stronger advantage over MQA.”
- Efficiently Scaling Transformer Inference (2022, neutral): “Finally, inference cost from the attention mechanism scales quadratically with input sequence length (Sukhbaatar et al., 2019; Choromanski et al., 2020; Dao et al., 2022).”
- Landmark Attention: Random-Access Infinite Context Length for Transformers (2023, methodology): “Finally, we point out that our method can be naturally combined with flash attention [9], reducing the overhead further.”
- Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, methodology): “We use Flash Attention (Dao et al., 2022) for improved hardware throughput.”
- StarCoder: may the source be with you! (2023, methodology): “We used FlashAttention (Dao et al., 2022) to speed up the attention computation and reduce its memory footprint, allowing us to scale to a 8K context length.”
- LightGlue: Local Feature Matching at Light Speed (2023, methodology): “In practice, we also use efficient self-attention [14] and mixed-precision to significantly reduce run time and memory requirements.”
- H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (2023, neutral): “, Reformer [7], Flash Attention [8], and Performer [9], are designed to overcome the quadratic memory required by attention mechanisms when modeling long sequences but still require a large cache size.”
- Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass (2025, positive): “For example, packages like Deepspeed-Inference [38], FlashAttention [7, 8] provide fused kernels, model parallelism, and data parallelism.”
- OpenFold: retraining AlphaFold2 yields new insights into its learning mechanisms and capacity for generalization (2023, positive): “As such, OpenFold can be readily used by the widest community of developers and interfaces with a rich ecosystem of existing machine learning software 26–29 .”
- VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, methodology): “Regarding temporal convolution, we stack four convolutions with 1 × 1 × 3 kernel, ensuring the temporal receptive field is ample for capturing temporal dependencies; regarding temporal transformer, we stack one Transformer layer and accelerate its inference using flash attention [12].”