FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
By Tri Dao. 2023
2,998 citations recorded by the source corpus.
Scaling Transformers to longer sequence lengths has been a major problem in the last several years, promising to improve performance in language modeling and high-resolution image understanding, as well as to unlock new applications in code, audio, and video generation. The attention layer is the main bottleneck in scaling to longer sequences, as its runtime and memory increase quadratically in the sequence length. FlashAttention exploits the asymmetric GPU memory hierarchy to bring significant memory saving (linear instead of quadratic) and runtime speedup (2-4$\times$ compared to optimized baselines), with no approximation. However, FlashAttention is still not nearly as fast as optimized matrix-multiply (GEMM) operations, reaching only 25-40\% of the theoretical maximum FLOPs/s. We observe that the inefficiency is due to suboptimal work partitioning between different thread blocks and warps on the GPU, causing either low-occupancy or unnecessary shared memory reads/writes. We propose FlashAttention-2, with better work partitioning to address these issues. In particular, we (1) tweak the algorithm to reduce the number of non-matmul FLOPs (2) parallelize the attention computation, even for a single head, across different thread blocks to increase occupancy, and (3) within each thread block, distribute the work between warps to reduce communication through shared memory. These yield around 2$\times$ speedup compared to FlashAttention, reaching 50-73\% of the theoretical maximum FLOPs/s on A100 and getting close to the efficiency of GEMM operations. We empirically validate that when used end-to-end to train GPT-style models, FlashAttention-2 reaches training speed of up to 225 TFLOPs/s per A100 GPU (72\% model FLOPs utilization).
Subfields: Computer Networks and Communications, Computer Vision and Pattern Recognition, Hardware and Architecture, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Networks and Communications
- Percy Liang
- Yinmin Zhong
- Ion Stoica
- Adrien Bazoge
- Stella Biderman
- Ricardo Martin-Brualla
- Bo Li
- Song Han
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Ming-Hsuan Yang
- Jitendra Malik
- Yi Yang
- Alan L. Yuille
In Hardware and Architecture
- Ion Stoica
- Adrien Bazoge
- Stella Biderman
- Ricardo Martin-Brualla
- Bo Li
- Song Han
- Peiyi Wang
- Patrick Lewis
In Model Architectures
Citations in the corpus
- Llemma: An Open Language Model For Mathematics (2023, methodology): “We use Flash Attention 2 (Dao, 2023) to improve throughput and further reduce memory requirements.”
- DeepSeek LLM: Scaling Open-Source Language Models with Longtermism (2024, methodology): “We also leverage the flash attention (Dao, 2023; Dao et al., 2022) technique to improve hardware utilization.”
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, positive): “…FP8 training based on DeepGEMM [68] and liger-kernel’s fused cross-entropy operator [47] to accelerate the training process, FlashAttention-3 [26, 27] to support packed inputs and speed up attention computation, and the TMA-Adaptive FP8 Grouped GEMM kernel [118] to optimize the training of…”
- LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (2024, background): “While some of these limitations can be offset by recent advances in improving the training and inference efficiency of large models (Hu et al., 2022; Dao, 2024), these techniques can technically be applied to smaller bidirectional models as well.”
- Atom: Low-bit Quantization for Efficient and Accurate LLM Serving (2023, methodology): “Since the highly efficient INT4 calculation is supported by NVIDIA GPUs, we evaluate Atom with W4A4 quantization on 24GB RTX 4090 with CUDA 12.1.”
- LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models (2023, neutral): “Its format is as follows (three instances, for example), 1 { 2 "type": "text_only", 3 "instances": [ 4 { "text": "SAMPLE_TEXT_1" },”
- Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts (2024, neutral): “This work lays the groundwork for future advancements in scaling and enhancing the efficiency of time series foundation models, paving the way toward time series general intelligence.”
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024, neutral): “Dao [14] restructured the algorithm as F LASH A TTENTION -2 to also parallelize over the sequence length dimension and perform the inner loop of the forward pass over blocks of the key and value matrices, thus improving the occupancy and distribution of work on the GPU.”
- Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024, positive): “Notably, DistillSpec (Zhou et al., 2023) conducted a comprehensive comparison of different KD strategies on Speculative De-coding across various downstream tasks, pointing out that the choice of the optimal KD algorithm largely depends on specific tasks and the verification strategy.”
- Fine-Tuning LLaMA for Multi-Stage Text Retrieval (2023, background): “, 2022), flash attention (Dao, 2023), and gradient checkpointing to reduce GPU memory usage.”
- GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting (2024, methodology): “To enable efficient training and inference, we adopt Flash-Attention-v2 [18] in the xFormers [31] library, gradient checkpointing [15], and mixed-precision training [38] with BF16 data type.”
- Efficient Streaming Language Models with Attention Sinks (2023, methodology): “Solutions have ranged from system-focused optimizations like FlashAttention (Dao et al., 2022; Dao, 2023), which accelerates attention computation and reduces memory footprint, to approximate attention methods (Zaheer et al., 2020; Beltagy et al., 2020; Wang et al., 2020; Kitaev et al., 2020) that t”
- YaRN: Efficient Context Window Extension of Large Language Models (2023, positive): “The simplicity of YaRN allows it to be implemented readily in all inference and training libraries, including compatibility with Flash Attention 2 [10].”
- Long Context Transfer from Language to Vision (2024, methodology): “These includes FlashAttention-2 [19], Ring Attention [45, 40], activation checkpointing, and parameter offload [59].”
- OpenVLA: An Open-Source Vision-Language-Action Model (2024, methodology): “techniques for large transformer model training such as automatic mixed precision (AMP, PyTorch [75]), FlashAttention [76], and fully sharded data parallelism (FSDP, Zhao et al. [77]).”
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (2024, methodology): “We employ Flash Attention-2 (Dao, 2023) to accelerate both training and inference.”
- YOLOv12: Attention-Centric Real-Time Object Detectors (2025, positive): “Additionally, we employ FlashAtten-tion to overcome inherent memory accessing problems of the attention mechanism [13, 14].”
- MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (2024, positive): “This approach leverages FlashAttention [Dao24] to reduce GPU memory overhead and incorporates the information from the V matrix, enabling end-to-end selection of the best pattern, which further enhances performance.”
- SnapKV: LLM Knows What You are Looking for Before Generation (2024, methodology): “Specifically, in LLMs, each decoding step involves generating a single token, with the transfer of weights between High Bandwidth Memory (HBM) and cache contributing to significant overhead [33, 34].”