FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision

By Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, P. Ramani, Tri Dao. Neural Information Processing Systems, 2024

595 citations recorded by the source corpus.

Attention, as a core layer of the ubiquitous Transformer architecture, is the bottleneck for large language models and long-context applications. FlashAttention elaborated an approach to speed up attention on GPUs through minimizing memory reads/writes. However, it has yet to take advantage of new capabilities present in recent hardware, with FlashAttention-2 achieving only 35% utilization on the H100 GPU. We develop three main techniques to speed up attention on Hopper GPUs: exploiting asynchrony of the Tensor Cores and TMA to (1) overlap overall computation and data movement via warp-specialization and (2) interleave block-wise matmul and softmax operations, and (3) block quantization and incoherent processing that leverages hardware support for FP8 low-precision. We demonstrate that our method, FlashAttention-3, achieves speedup on H100 GPUs by 1.5-2.0$\times$ with FP16 reaching up to 740 TFLOPs/s (75% utilization), and with FP8 reaching close to 1.2 PFLOPs/s. We validate that FP8 FlashAttention-3 achieves 2.6$\times$ lower numerical error than a baseline FP8 attention.

View Paper: https://www.semanticscholar.org/paper/5d1ca53a0b41f4f5c960cd9997556d7180d6b88d

Citations in the corpus

• Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, unknown): “On the other hand, Self Forcing always uses full attention during training and can leverage highly optimized attention kernels such as FlashAttention-3 [72].”
• Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2024, unknown): “At the start of this work, Flash Attention 3 (Shah et al., 2024), the most recent iteration for Nvidia H100 GPUs, did not include support for sliding window attention.”
• FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space (2025, unknown): “To improve throughput, we use Flash Attention 3 [50] and regional compilation of individual Transformer blocks.”
• ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performance (2025, unknown): “These include FlashAttention (Dao et al., 2022; Dao, 2024; Shah et al., 2024), alternating global and local attention layers(Team et al., 2024), sequence packing (Portes et al., 2023), and rotary positional embeddings (RoPE)(Su et al., 2021).”
• AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages (2026, unknown): “We maximized training throughput and memory efficiency by employing sequence packing, DeepSpeed ZeRO-1/ZeRO-2 (Rasley et al., 2020), Flash Attention 3 (Shah et al., 2024), and Liger Kernel (Hsu et al., 2025).”