Break the Sequential Dependency of LLM Inference Using Lookahead Decoding

By Yichao Fu, Peter Bailis, Ion Stoica, Hao Zhang. International Conference on Machine Learning, 2024

336 citations recorded by the source corpus.

Autoregressive decoding of large language models (LLMs) is memory bandwidth bounded, resulting in high latency and significant wastes of the parallel processing power of modern accelerators. Existing methods for accelerating LLM decoding often require a draft model (e.g., speculative decoding), which is nontrivial to obtain and unable to generalize. In this paper, we introduce Lookahead decoding, an exact, parallel decoding algorithm that accelerates LLM decoding without needing auxiliary models or data stores. It allows trading per-step log(FLOPs) to reduce the number of total decoding steps, is more parallelizable on single or multiple modern accelerators, and is compatible with concurrent memory-efficient attention (e.g., FlashAttention). Our implementation of Lookahead decoding can speed up autoregressive decoding by up to 1.8x on MT-bench and 4x with strong scaling on multiple GPUs in code completion tasks. Our code is avialable at https://github.com/hao-ai-lab/LookaheadDecoding

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/f1a9e0830bc36c048fa4659beaa62609869895b5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yuhui Li
2. Fangyun Wei
3. Chao Zhang
4. Hongyang Zhang

Citations in the corpus

• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (2024, methodology): “Online Speculative Decoding (Liu et al., 2023) trains the draft model on a distribution of queries.”
• EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees (2024, background): “REST (Fu et al., 2024) and LLMA (Yang et al., 2023a) generate drafts through retrieval.”