Efficient Memory Management for Large Language Model Serving with PagedAttention
By Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Cody Hao Yu, Joseph E. Gonzalez, Haotong Zhang, Ion Stoica. 2023
7,142 citations recorded by the source corpus.
High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show that vLLM improves the throughput of popular LLMs by 2--4× with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms. vLLM's source code is publicly available at https://github.com/vllm-project/vllm.
Subfields: Artificial Intelligence, Computer Networks and Communications, Large Language Models, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Yejin Choi
- Luke S. Zettlemoyer
- Furu Wei
- Quoc V. Le
- Hannaneh Hajishirzi
In Computer Networks and Communications
- Ion Stoica
- Joseph E. Gonzalez
- Yinmin Zhong
- Sébastien Bubeck
- Vishrav Chaudhary
- Marie-Anne Lachaux
- Armando Solar-Lezama
- George Pappas
In Large Language Models
- Furu Wei
- Shuming Ma
- Shaohan Huang
- Wenhui Wang
- Marie-Anne Lachaux
- Vishrav Chaudhary
- Wayne Xin Zhao
- Yejin Choi
In Efficiency & Compression
Citations in the corpus
- Mistral 7B (2023, methodology): “This release is accompanied by a reference implementation1 facilitating easy deployment either locally or on cloud platforms such as AWS, GCP, or Azure using the vLLM [17] inference server and SkyPilot 2.”
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “Specifically, we generate 10 candidate answers for each problem either using API or using vLLM (Kwon et al., 2023).”
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “Also, the efficient inference techniques (Kwon et al., 2023; Leviathan et al., 2023; Xia et al., 2023, 2024), which determines the exploration efficiency of policy models, also play an exceedingly important role.”
- Refusal in Language Models Is Mediated by a Single Direction (2024, neutral): “For the other benchmarks, we use the default settings, with the exception that models are run using vLLM (Kwon et al., 2023).”
- Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation (2023, methodology): “In addition to the inference speed, the memory efficiency can also be improved through various technologies such as model quantization [49] and PagedAttention [50].”
- SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning (2024, neutral): “Specifically, our contributions can be summarized as follows: [31],andLMDeploy[7],togethertheseintegrations providesupportforinferenceagainstmosttextandmulti-modal large models.”
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (2024, methodology): “Thus, optimizing LLM inference has been a key focus for many recent systems [29,53,58,59,63,75,77].”
- LLMSTEP: LLM proofstep suggestions in Lean (2023, positive): “LLMSTEP supports a variety of compute constraints, with server implementations that run on CPU, a CUDA GPU, or a Google Colab notebook with GPU, as well as a server with fast inference via vLLM [8].”
- NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness (2024, methodology): “We utilize the vLLM library (Kwon et al., 2023) for generating LM outputs for tasks in NoFunEval .”
- Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, background): “Previous works proposed a series of solutions like KV-cache control management [98] or KV-cache compression [159] to improve throughput or reduce latency.”
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (2024, background): “SmoothQuant can compress KV cache to 8bit with minor performance loss.”
- Seed-TTS: A Family of High-Quality Versatile Speech Generation Models (2024, methodology): “…applied methods to reduce the memory and computation consumption on the language model side, such as grouped-query attention [Ainslie et al., 2023], paged attention [Kwon et al., 2023], flash attention [Dao et al., 2022, Dao, 2023], and model quantization [Nagel et al., 2021, Guo et al., 2024].”
- Yi: Open Foundation Models by 01.AI (2024, methodology): “We use PagedAttention[41] to improve memory utilization and improve decoding.”
- DeepSeek LLM: Scaling Open-Source Language Models with Longtermism (2024, methodology): “These datasets include HellaSwag, PIQA, WinoGrande, RACE-Middle, RACE-High, MMLU, ARC-Easy, ARC-Challenge, OpenBookQA, CHID, C-Eval, CMMLU, C3 and CCPM.”
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (2024, methodology): “We found that it is necessary to use a custom prompt for Llama-3-70B as the prompt from Chao et al. (2023) often leads to refusals.”
- Emu3: Next-Token Prediction is All You Need (2024, methodology): “Additionally, we utilize the open-source vLLM library [40] to accelerate the labeling process.”
- Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference (2024, positive): “We choose two widely used long-context models for our evaluation: LongChat-v1.”
- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (2024, methodology): “Going one step further, we envision and call for actions to design new hardware and system specifically optimized for 1-bit LLMs, given the new computation paradigm enabled in BitNet [WMD + 23].”