GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

By J. Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr'on, Sumit K. Sanghai. 2023

1,604 citations recorded by the source corpus.

Multi-query attention (MQA), which only uses a single key-value head, drastically speeds up decoder inference. However, MQA can lead to quality degradation, and moreover it may not be desirable to train a separate model just for faster inference. We (1) propose a recipe for uptraining existing multi-head language model checkpoints into models with MQA using 5% of original pre-training compute, and (2) introduce grouped-query attention (GQA), a generalization of multi-query attention which uses an intermediate (more than one, less than number of query heads) number of key-value heads. We show that uptrained GQA achieves quality close to multi-head attention with comparable speed to MQA.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Luke S. Zettlemoyer
3. Jianfeng Gao
4. Furu Wei
5. Graham Neubig
6. Quoc V. Le
7. Percy Liang
8. Yoshua Bengio

In Model Architectures

1. Furu Wei
2. Quoc V. Le
3. Yoshua Bengio
4. Li Dong
5. Angela Fan
6. Armand Joulin
7. Oriol Vinyals
8. Ankur Bapna

Citations in the corpus

• Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “We also increased the size of the pretraining corpus by 40%, doubled the context length of the model, and adopted grouped-query attention (Ainslie et al., 2023).”
• Mistral 7B (2023, background): “Mistral 7B leverages grouped-query attention (GQA) [1], and sliding window attention (SWA) [6, 3].”
• Retentive Network: A Successor to Transformer for Large Language Models (2023, methodology): “We compare with grouped-query attention (Ainslie et al., 2023) and evaluate the method in the context of RetNet.”
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (2024, methodology): “Whereas we apply Group Query Attention (Ainslie et al., 2023) to MiniCPM-1.”
• Qwen2 Technical Report (2024, methodology): “Key differences from Qwen are described below: Grouped Query Attention We adopt Grouped Query Attention (GQA, Ainslie et al., 2023) instead of conventional multi-head attention (MHA).”
• Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, background): “Beyond the basic version of the attention mechanism, various variants are developed to capture the different relationships among tokens, such as multi-head attention [41], multi-query attention [42], and grouped-query attention [43].”
• Gemma 3 Technical Report (2025, positive): “We use a Grouped-Query Attention (GQA) (Ainslie et al., 2023) with post-norm and pre-norm with RMSNorm (Zhang and Sen-nrich, 2019).”
• Large Language Diffusion Models (2025, neutral): “We use vanilla multi-head attention instead of grouped query attention [24] for simplicity, as LLaDA is incompatible with KV caching, resulting in a different number of key and value heads.”
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (2024, neutral): “Note that the KV cache is larger in L LAMA 2-7B model as the L LAMA 2-70B uses grouped-query attention [Ainslie et al., 2023].”
• Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, methodology): “…including utilizing pre-trained checkpoints (e.g., compression network) [47], shortening the context window, using light-weight modeling mechanisms such as (grouped) multi-query attention [48, 49] or efficient architectures (e.g. Mamba [50]), downsampling data and dropping tokens if necessary.”
• Qwen3 Technical Report (2025, positive): “5 (Yang et al., 2024b), including using Grouped Query Attention (GQA, Ainslie et al., 2023), SwiGLU (Dauphin et al., 2017), Rotary Positional Embeddings (RoPE, Su et al., 2024), and RMSNorm (Jiang et al., 2023) with pre-normalization.”
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (2024, background): “B ← filter_finished_requests( B ) (MQA) [61], Group Query Attention (GQA) [30] in leading edge LLM models like LLaMA2 [66], Falcon [31] and Yi [24] has also significantly helped in alleviating memory bottleneck in LLM inference.”
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models (2024, methodology): “…investigate commonly applied methods to reduce the memory and computation consumption on the language model side, such as grouped-query attention [Ainslie et al., 2023], paged attention [Kwon et al., 2023], flash attention [Dao et al., 2022, Dao, 2023], and model quantization [Nagel et al.,…”
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (2024, methodology): “Newer GPT model families are adopting memory-efficient attention mechanisms like GQA [10] and MQA [44].”
• TinyLlama: An Open-Source Small Language Model (2024, methodology): “To reduce memory bandwidth overhead and speed up inference, we use grouped-query attention (Ainslie et al., 2023) in our model.”
• ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens (2024, neutral): “This lag in the advancement of long-context evaluation methodologies impedes both the comparative analysis of diverse long-context LLMs and the pinpointing of potential enhancements in long-context processing .”
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023, background): “Multi-query attention (MQA) [15] and grouped-query attention (GQA) [1] are variants of attention where multiple heads of query attend to the same head of key and value, in order to reduce the size of KV cache during inference.”