Efficient Streaming Language Models with Attention Sinks

By Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis. International Conference on Learning Representations, 2023

2,131 citations recorded by the source corpus.

Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach -- but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attention sink, that keeping the KV of initial tokens will largely recover the performance of window attention. In this paper, we first demonstrate that the emergence of attention sink is due to the strong attention scores towards initial tokens as a"sink"even if they are not semantically important. Based on the above analysis, we introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence lengths without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more. In addition, we discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment. In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup. Code and datasets are provided at https://github.com/mit-han-lab/streaming-llm.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/fdc53c2c10742464087c0525f77e32604827a21d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (2024, methodology): “5 can be found in Table 10 and Table 9 Model Qasper QMSum MultiNews TREC TriviaQA SAMSum LCC RepoBench-P Average One essential recipe of KIVI is the per-channel quantization scheme designed based on the observation made in Section 3.2 and Figure 2.”
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (2024, neutral): “Additionally, KV cache loading is memory bandwidth bound, regardless of batch size or sequence length, meaning that the dequantization overhead introduced by non-uniform quantization methods is not problematic (since the added computation does not introduce any additional latency).”
∞Bench: Extending Long Context Evaluation Beyond 100K Tokens (2024, methodology): “Approaches like Sliding Window Attention (Beltagy et al., 2020), LM-Infinite (Han et al., 2023), and StreamingLLM (Xiao et al., 2023) introduce attention variants for handling infinitely long sequences without overwhelming computation or memory overhead.”
LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (2024, methodology): “Beyond positional embedding interpolation, some research achieves input context extension using the original LLM context window length by manipulating attention mechanisms (Han et al., 2023; Xiao et al., 2023; Ratner et al., 2022).”
Long-context LLMs Struggle with Long In-context Learning (2024, positive): “These methods (Hao et al., 2022; Chen et al., 2023a; Peng et al., 2023b; Ratner et al., 2023; Xiao et al., 2024) can unlock lots of complex real-world applications from long-document question-answering, multi-document summarization, long-horizon agent tasks, repo-level code understanding.”
Retrieval-Augmented Generation for Large Language Models: A Survey (2023, background): “With ongoing efforts to expand LLM context windows to virtually unlimited sizes, the adaptation of RAG to these changes presents a significant research question [Xu et al. , 2023c, Packer et al. , 2023, Xiao et al. , 2023].”
SnapKV: LLM Knows What You are Looking for Before Generation (2024, background): “There are many perspectives to mitigate these problems, including KV cache eviction during token generation [5–8].”
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (2024, positive): “Then we demonstrate our patterns are significantly more efficient compared to other sparse methods [RCHG + 24, XTC + 24, PPJF24].”
RULER: What's the Real Context Size of Your Long-Context Language Models? (2024, methodology): “Various sparse attention mechanisms (Child et al., 2019; Jaszczur et al., 2021) such as shifted sparse attention (Chen et al., 2024), dilated attention (Ding et al., 2023), and attention sinks (Han et al., 2023; Xiao et al., 2024b) were employed to enable efficient context scaling.”
Scaling and evaluating sparse autoencoders (2024, positive): “This phenomenon was explained in [Sun et al., 2024, Xiao et al., 2023], which demonstrate that these activations serve as crucial attention resting states.”
Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference (2024, neutral): “For TOVA (Oren et al., 2024) and StreamingLLM (Xiao et al., 2023), we evaluated them on the 10k and 100k sequence lengths.”
Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, positive): “Inspired by research in large language models [92], we propose a rolling KV cache mechanism for autoregressive diffusion models that allows infinitely long video generation without any need of recomputing the KV cache.”
gpt-oss-120b&gpt-oss-20b Model Card (2025, neutral): “Each attention head has a learned bias in the denominator of the softmax, similar to off-by-one attention and attention sinks [16][17], which enables the attention mechanism to pay no attention to any tokens.”
Point Transformer V3: Simpler, Faster, Stronger (2023, background): “This issue also observed in recent advancements in transformer technology [93], is attributed to “attention sinks” stemming from the dot-product and softmax operations.”
PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling (2024, methodology): “Previous work on KV cache compression [22, 34, 32] often allocates a fixed KV cache size across LLM layers.”
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression (2024, background): “An alternative approach is to compress or trim the context hidden or KV caches (Chevalier et al., 2023; Ge et al., 2024; Zhang et al., 2023; Liu et al., 2023; Xiao et al., 2024).”
An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models (2024, neutral): “We use greedy search for all experiments and provide details for each task in section A in the supplement material.”
Evaluating Very Long-Term Conversational Memory of LLM Agents (2024, background): “Despite recent advancements in dialogue models based on LLMs for extended contexts (Bertsch et al., 2023; Xiao et al., 2023), as well as the integration of retrieval augmented generation (RAG) Figure 1: An example in L O C O M O .”
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention (2025, positive): “These limitations in inference efficiency and training viability motivate our fundamental redesign of sparse attention mechanisms.”