Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time

By Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, Anshumali Shrivastava. Neural Information Processing Systems, 2023

475 citations recorded by the source corpus.

Large language models(LLMs) have sparked a new wave of exciting AI applications. Hosting these models at scale requires significant memory resources. One crucial memory bottleneck for the deployment stems from the context window. It is commonly recognized that model weights are memory hungry; however, the size of key-value embedding stored during the generation process (KV cache) can easily surpass the model size. The enormous size of the KV cache puts constraints on the inference batch size, which is crucial for high throughput inference workload. Inspired by an interesting observation of the attention scores, we hypothesize the persistence of importance: only pivotal tokens, which had a substantial influence at one step, will significantly influence future generations. Based on our empirical verification and theoretical analysis around this hypothesis, we propose Scissorhands, a system that maintains the memory usage of the KV cache at a fixed budget without finetuning the model. In essence, Scissorhands manages the KV cache by storing the pivotal tokens with a higher probability. We validate that Scissorhands reduces the inference memory usage of the KV cache by up to 5X without compromising model quality. We further demonstrate that Scissorhands can be combined with 4-bit quantization, traditionally used to compress model weights, to achieve up to 20X compression.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d6eeb2898bd9bd34744194ef543062dda6c4531a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression (2024, background): “An alternative approach is to compress or trim the context hidden or KV caches (Chevalier et al., 2023; Ge et al., 2024; Zhang et al., 2023; Liu et al., 2023; Xiao et al., 2024).”
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (2024, background): “Some of these methods aim to only store important tokens in the KV cache and to evict less important tokens, thereby maintaining low memory usage [11, 22, 35].”
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (2024, background): “Similarly, Scissorhands (Liu et al., 2024) exploit the persistence of the importance hypothesis in KV cache sparsification.”
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (2023, methodology): “It is worth mentioning that C local+frequent is a very strong baseline as it is identical to the H 2 O method (Zhang et al., 2023) and the Scissorhands method (Liu et al., 2023a).”
SGLang: Efficient Execution of Structured Language Model Programs (2023, unknown): “Additionally, SGLang is compatible with many other common inference optimizations [60, 39, 3, 23, 59, 10, 26, 15, 19, 32, 31, 11].”
SnapKV: LLM Knows What You are Looking for Before Generation (2024, background): “This variability challenges the effectiveness of static compression methods that depend on constant weighted importance or fixed policies [7, 6, 8].”