Efficient Memory Management for Large Language Model Serving with PagedAttention

By Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Cody Hao Yu, Joseph E. Gonzalez, Haotong Zhang, Ion Stoica. 2023

7,142 citations recorded by the source corpus.

High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show that vLLM improves the throughput of popular LLMs by 2--4× with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms. vLLM's source code is publicly available at https://github.com/vllm-project/vllm.

Subfields: Artificial Intelligence, Computer Networks and Communications, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Percy Liang
  4. Yejin Choi
  5. Luke S. Zettlemoyer
  6. Furu Wei
  7. Quoc V. Le
  8. Hannaneh Hajishirzi

In Computer Networks and Communications

  1. Ion Stoica
  2. Joseph E. Gonzalez
  3. Yinmin Zhong
  4. Sébastien Bubeck
  5. Vishrav Chaudhary
  6. Marie-Anne Lachaux
  7. Armando Solar-Lezama
  8. George Pappas

In Large Language Models

  1. Furu Wei
  2. Shuming Ma
  3. Shaohan Huang
  4. Wenhui Wang
  5. Marie-Anne Lachaux
  6. Vishrav Chaudhary
  7. Wayne Xin Zhao
  8. Yejin Choi

In Efficiency & Compression

  1. Joseph E. Gonzalez
  2. Ion Stoica
  3. Furu Wei
  4. Song Han
  5. Tao Ge
  6. Jiaming Tang
  7. Esha Choukse
  8. Guangxuan Xiao

Citations in the corpus