High-throughput Generative Inference of Large Language Models with a Single GPU

By Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark W. Barrett, Joseph Gonzalez and 4 more. 2023

797 citations recorded by the source corpus.

The high computational and memory requirements of large language model (LLM) inference make it feasible only with multiple high-end accelerators. Motivated by the emerging demand for latency-insensitive tasks with batched processing, this paper initiates the study of high-throughput LLM inference using limited resources, such as a single commodity GPU. We present FlexGen, a high-throughput generation engine for running LLMs with limited GPU memory. FlexGen can be flexibly configured under various hardware resource constraints by aggregating memory and computation from the GPU, CPU, and disk. By solving a linear programming problem, it searches for efficient patterns to store and access tensors. FlexGen further compresses the weights and the attention cache to 4 bits with negligible accuracy loss. These techniques enable FlexGen to have a larger space of batch size choices and thus significantly increase maximum throughput. As a result, when running OPT-175B on a single 16GB GPU, FlexGen achieves significantly higher throughput compared to state-of-the-art offloading systems, reaching a generation throughput of 1 token/s for the first time with an effective batch size of 144. On the HELM benchmark, FlexGen can benchmark a 30B model with a 16GB GPU on 7 representative sub-scenarios in 21 hours. The code is available at https://github.com/FMInference/FlexGen

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Materials Chemistry, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Materials Chemistry

In Large Language Models

In Efficiency & Compression

Citations in the corpus

Efficient Memory Management for Large Language Model Serving with PagedAttention (2023, background): “Notably, FlexGen [46] studies how to swap weights and token states for LLM inference with limited GPU memory, but it does not target the online serving settings.”
Fast Inference of Mixture-of-Experts Language Models with Offloading (2023, methodology): “To use these LLMs on more affordable hardware setups, one must either compress model parameters (Dettmers et al., 2022; Frantar et al., 2022) or offload parameters to a cheaper storage, be it RAM or SSD (Pudipeddi et al., 2020; Sheng et al., 2023).”
SGLang: Efficient Execution of Structured Language Model Programs (2023, positive): “…to support additional output modalities, adapting RadixAttention to operate across multiple levels of the memory hierarchy (e.g., DRAM, Disk) [43], enabling fuzzy semantic matching within RadixAttention, providing higher-level primitives atop SGLang, fixing starvation in cache-aware…”
A Simple and Effective Pruning Approach for Large Language Models (2023, background): “The fast pace of LLM quantization research [1, 13–15, 22, 38, 70–72] has led to substantial resource savings for these models [43, 59].”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, background): “In response to this, some strategies have resorted to quantizing the KV cache [159], [36], but quantization will certainly compromises performance.”
Atom: Low-bit Quantization for Efficient and Accurate LLM Serving (2023, methodology): “To apply mixed-precision quantization while maintaining regular memory access, Atom re-purposes the reordering technique introduced in RPTQ (Yuan et al., 2023), where the objec-tive was to improve inference accuracy.”
PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU (2023, neutral): “Model ofﬂoading, which partitions the model between GPU and CPU at the Transformer layer level [3,14,37].”
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (2023, positive): “Performance experiments show our framework achieves 29 , 29 , 3 higher throughputs compared to three leading inference systems, DeepSpeed Zero-Inference (Aminabadi et al., 2022), Hugging Face Ac-celerate (HuggingFace), and FlexGen (Sheng et al., 2023) respectively.”
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time (2023, background): “[7] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y.”
Splitwise: Efficient Generative LLM Inference Using Phase Splitting (2023, methodology): “LLM inference serving is a rapidly developing field, with several recent works optimizing batching [23], [25], [51], [53], [81], scheduling [22], [42], [51], [66], [73], [79], and memory usage [32], [35], [51], [74].”
Large Language Models for Software Engineering: A Systematic Literature Review (2023, background): “We also face issues like the need for extensive labeled data [347], significant computational resources [118, 248], and avoiding potential ethical problems [5, 40, 325].”
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (2023, background): “, W4A16), where only weights are quantized into low-bit integers [17, 13, 44, 39].”
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (2023, neutral): “Furthermore, we delved into the potential of combining the group-wise quantization (Shen et al., 2020a; Sheng et al., 2023a) with LLM-QAT.”
DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (2024, background): “In offline applications that are not latency-sensitive, users typically have lower requirements for response time [45].”
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (2024, methodology): “The baseline configurations used by Atom and FlexGen are included for reference [28, 36].”
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (2024, methodology): “Following previous work (Sheng et al., 2023), the group size G in Algorithm 1 for quantization is set as 32 across all experiments, the residual length R for key and value cache is set to 128.”