KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
By Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, Xia Hu. International Conference on Machine Learning, 2024
545 citations recorded by the source corpus.
Efficiently serving large language models (LLMs) requires batching of many requests to reduce the cost per request. Yet, with larger batch sizes and longer context lengths, the key-value (KV) cache, which stores attention keys and values to avoid re-computations, significantly increases memory demands and becomes the new bottleneck in speed and memory usage. Additionally, the loading of the KV cache causes the computational core to be idle, which limits the inference speed. A straightforward and effective solution to reduce KV cache size is quantization, which decreases the total bytes taken by KV cache. However, there is a lack of in-depth studies that explore the element distribution of KV cache to understand the hardness and limitation of KV cache quantization. To fill the gap, we conducted a comprehensive study on the element distribution in KV cache of popular LLMs. Our findings indicate that the key cache should be quantized per-channel, i.e., group elements along the channel dimension and quantize them together. In contrast, the value cache should be quantized per-token. From this analysis, we developed a tuning-free 2bit KV cache quantization algorithm named KIVI. With hardware-friendly implementation, KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using $\mathbf{2.6\times}$ less peak memory (including model weight). This reduction in memory usage enables up to $\mathbf{4\times}$ larger batch size, bringing $\mathbf{2.35\times \sim 3.47\times}$ throughput on real LLM inference workload. The source code is available at https://github.com/jy-yuan/KIVI.
Subfields: Computational Theory and Mathematics, Computer Networks and Communications
View Paper: https://www.semanticscholar.org/paper/a3e000e0d7f64c1d094c2a8bf6f43992cbabe91b
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Theory and Mathematics
In Computer Networks and Communications
Citations in the corpus
- Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models (2025, unknown): “Unlike model compression techniques such as quantization [29,50,97] or KV cache compression [55, 76, 107, 110], which focus on reducing model size for lightweight inference, efficient reasoning in LLMs emphasizes smart and concise reasoning by optimizing the length of generated reasoning sequences…”
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (2024, unknown): “However, key vectors in the attention module are also known to suffer from outliers [Hooper et al., 2024, Liu et al., 2024].”
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024, background): “Recent work suggests that for inference the KV cache is highly compressible down to 4, 3-, or even 2-bits [26, 33].”
- SGLang: Efficient Execution of Structured Language Model Programs (2023, unknown): “Additionally, SGLang is compatible with many other common inference optimizations [60, 39, 3, 23, 59, 10, 26, 15, 19, 32, 31, 11].”