PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU

By Yixin Song, Zeyu Mi, Haotong Xie, Haibo Chen. Symposium on Operating Systems Principles, 2023

311 citations recorded by the source corpus.

This paper introduces PowerInfer, a high-speed Large Language Model (LLM) inference engine on a personal computer (PC) equipped with a single consumer-grade GPU. The key principle underlying the design of PowerInfer is exploiting the high locality inherent in LLM inference, characterized by a power-law distribution in neuron activation. This distribution indicates that a small subset of neurons, termed hot neurons, are consistently activated across inputs, while the majority, cold neurons, vary based on specific inputs. PowerInfer exploits such an insight to design a GPU-CPU hybrid inference engine: hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons are computed on the CPU, thus significantly reducing GPU memory demands and CPU-GPU data transfers. PowerInfer further integrates adaptive predictors and neuron-aware sparse operators, optimizing the efficiency of neuron activation and computational sparsity. The evaluation shows that PowerInfer significantly outperforms llama.cpp by up to 11.69× while retaining model accuracy across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU. For the OPT-30B model, PowerInfer achieves performance comparable to that of a high-end server-grade A100 GPU, reaching 82% of its token generation rate on a single consumer-grade RTX 4090 GPU.

Subfields: Artificial Intelligence, Computer Networks and Communications, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ddacee7382548fd9976e846c92500cfa3b6741db

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xin Jin
2. Yinmin Zhong
3. Xuanzhe Liu
4. Junda Chen
5. Shengyu Liu
6. Jianbo Hu
7. Yibo Zhu
8. Hao Zhang

In Computer Networks and Communications

1. Xin Jin
2. Yinmin Zhong
3. Yibo Zhu

In Computer Vision and Pattern Recognition

1. Da Ma
2. Su Zhu
3. Xin Jin
4. Hong-Shen Xu
5. Shuai Fan
6. Yinmin Zhong
7. Han-Qi Li
8. Liangtai Sun

Citations in the corpus

• Small Language Models are the Future of Agentic AI (2025, unknown): “That this behavior appears to be more subdued in SLMs [65, 71] suggests that SLMs may be fundamentally more efficient by the virtue of having a smaller proportion of their parameters contribute to the inference cost without a tangible effect on the output.”
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (2024, background): “Small-scale enterprises and individual researchers often lack the resources to deploy LLMs on large-scale clusters [45,48].”
• Sparsity-Accelerated Training for Large Language Models (2024, background): “Concretely, wanda (Sun et al., 2023) considers the magnitude of the input and parameter product, while maxip (Liu et al., 2023; Song et al., 2023) focuses on the input mean and parameter product.”