SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification

By Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi and 5 more. International Conference on Architectural Support for Programming Languages and Operating Systems, 2023

395 citations recorded by the source corpus.

This paper introduces SpecInfer, a system that accelerates generative large language model (LLM) serving with tree-based speculative inference and verification. The key idea behind SpecInfer is leveraging small speculative models to predict the LLM's outputs; the predictions are organized as a token tree, whose nodes each represent a candidate token sequence. The correctness of all candidate token sequences represented by a token tree is verified against the LLM in parallel using a novel tree-based parallel decoding mechanism. SpecInfer uses an LLM as a token tree verifier instead of an incremental decoder, which significantly reduces the end-to-end latency and computational requirement for serving generative LLMs while provably preserving model quality. Our evaluation shows that SpecInfer outperforms existing LLM serving systems by 1.5-2.8× for distributed LLM inference and by 2.6-3.5× for offloading-based LLM inference, while preserving the same generative performance. SpecInfer is publicly available at https://github.com/flexflow/FlexFlow/

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/f0c31511134abdd23f990310e8a2f2eb3a629b62

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Heming Xia
2. Peiyi Wang
3. Zhifang Sui
4. Tianyu Liu
5. Qingxiu Dong
6. Ion Stoica
7. Peter Bailis
8. Tao Ge

Citations in the corpus

• Break the Sequential Dependency of LLM Inference Using Lookahead Decoding (2024, methodology): “It shows notable speedup without a draft model and can linearly decrease the decoding steps with exponential investment in per-step FLOPs.”
• Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024, methodology): “To tackle this problem, multiple studies have proposed various approximate verification criteria (Stern et al., 2018; Xia et al., 2022; Kim et al., 2023).”