Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
By Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, Jianfeng Gao. International Conference on Learning Representations, 2023
495 citations recorded by the source corpus.
In this study, we introduce adaptive KV cache compression, a plug-and-play method that reduces the memory footprint of generative inference for Large Language Models (LLMs). Different from the conventional KV cache that retains key and value vectors for all context tokens, we conduct targeted profiling to discern the intrinsic structure of attention modules. Based on the recognized structure, we then construct the KV cache in an adaptive manner: evicting long-range contexts on attention heads emphasizing local contexts, discarding non-special tokens on attention heads centered on special tokens, and only employing the standard KV cache for attention heads that broadly attend to all tokens. Moreover, with the lightweight attention profiling used to guide the construction of the adaptive KV cache, FastGen can be deployed without resource-intensive fine-tuning or re-training. In our experiments across various asks, FastGen demonstrates substantial reduction on GPU memory consumption with negligible generation quality loss. We will release our code and the compatible CUDA kernel for reproducibility.
View Paper: https://www.semanticscholar.org/paper/6c323c535365e1c7cbfd9703cbec3b5650a3346b
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Sehoon Kim
- A. Gholami
- Coleman Hooper
- Kurt Keutzer
- Zefan Cai
- Yuhong Li
- Yichi Zhang
- Hiva Mohammadzadeh
Citations in the corpus
- PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling (2024, background): “To tackle these memory constraints, recent studies have explored the optimization of KV caching, including approaches such as low-rank decomposition of the KV cache [8] or pruning non-essential KV cache [34, 22, 12].”
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (2024, background): “Some of these methods aim to only store important tokens in the KV cache and to evict less important tokens, thereby maintaining low memory usage [11, 22, 35].”
- SnapKV: LLM Knows What You are Looking for Before Generation (2024, methodology): “There are many perspectives to mitigate these problems, including KV cache eviction during token generation [5–8].”
- SGLang: Efficient Execution of Structured Language Model Programs (2023, unknown): “Additionally, SGLang is compatible with many other common inference optimizations [60, 39, 3, 23, 59, 10, 26, 15, 19, 32, 31, 11].”
- Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention (2025, unknown): “Recent advances demonstrate this potential through diverse strategies: KV-cache eviction methods (Zhang et al., 2023b; Li et al., 2024; Zhou et al., 2024), blockwise KV-cache selection methods (Tang et al., 2024; Xiao et al., 2024a; Gao et al., 2024), and sampling, clustering or hashing-based…”