H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

By Zhenyu (Allen) Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R'e, Clark W. Barrett and 2 more. Neural Information Processing Systems, 2023

817 citations recorded by the source corpus.

Large Language Models (LLMs), despite their recent impressive accomplishments, are notably cost-prohibitive to deploy, particularly for applications involving long-content generation, such as dialogue systems and story writing. Often, a large amount of transient state information, referred to as the KV cache, is stored in GPU memory in addition to model parameters, scaling linearly with the sequence length and batch size. In this paper, we introduce a novel approach for implementing the KV cache which significantly reduces its memory footprint. Our approach is based on the noteworthy observation that a small portion of tokens contributes most of the value when computing attention scores. We call these tokens Heavy Hitters (H$_2$). Through a comprehensive investigation, we find that (i) the emergence of H$_2$ is natural and strongly correlates with the frequent co-occurrence of tokens in the text, and (ii) removing them results in significant performance degradation. Based on these insights, we propose Heavy Hitter Oracle (H$_2$O), a KV cache eviction policy that dynamically retains a balance of recent and H$_2$ tokens. We formulate the KV cache eviction as a dynamic submodular problem and prove (under mild assumptions) a theoretical guarantee for our novel eviction algorithm which could help guide future work. We validate the accuracy of our algorithm with OPT, LLaMA, and GPT-NeoX across a wide range of tasks. Our implementation of H$_2$O with 20% heavy hitters improves the throughput over three leading inference systems DeepSpeed Zero-Inference, Hugging Face Accelerate, and FlexGen by up to 29$\times$, 29$\times$, and 3$\times$ on OPT-6.7B and OPT-30B. With the same batch size, H2O can reduce the latency by up to 1.9$\times$. The code is available at https://github.com/FMInference/H2O.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e586a4591ba0303b769f2c07cbddaf1899cb72e4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models (2025, neutral): “Unlike model compression techniques such as quantization [29,50,97] or KV cache compression [55, 76, 107, 110], which focus on reducing model size for lightweight inference, efficient reasoning in LLMs emphasizes smart and concise reasoning by optimizing the length of generated reasoning sequences…”
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (2023, methodology): “It is worth mentioning that C local+frequent is a very strong baseline as it is identical to the H 2 O method (Zhang et al., 2023) and the Scissorhands method (Liu et al., 2023a).”
PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling (2024, methodology): “Previous work on KV cache compression [22, 34, 32] often allocates a fixed KV cache size across LLM layers.”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “For example, a 30B parameter LLM operating with an input length of 1024 and a batch size of 128 might necessitate up to 180GB for the KV-cache [223], thereby restricting the feasible size of the context window.”
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (2024, positive): “Language Modeling Following the approach of StreamingLLM [XTC + 24] and H2O [ZSZ + 24], we evaluate our methods against baselines on the language modeling task based on the PG-19 dataset [RPJ + 20].”
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention (2025, neutral): “For GQA, we set the number of groups to 4, with a total of 64 attention heads.”
SnapKV: LLM Knows What You are Looking for Before Generation (2024, background): “This variability challenges the effectiveness of static compression methods that depend on constant weighted importance or fixed policies [7, 6, 8].”