Accelerating Large Language Model Decoding with Speculative Sampling

By Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, John M. Jumper. 2023

973 citations recorded by the source corpus.

We present speculative sampling, an algorithm for accelerating transformer decoding by enabling the generation of multiple tokens from each transformer call. Our algorithm relies on the observation that the latency of parallel scoring of short continuations, generated by a faster but less powerful draft model, is comparable to that of sampling a single token from the larger target model. This is combined with a novel modified rejection sampling scheme which preserves the distribution of the target model within hardware numerics. We benchmark speculative sampling with Chinchilla, a 70 billion parameter language model, achieving a 2-2.5x decoding speedup in a distributed setup, without compromising the sample quality or making modifications to the model itself.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023, background): “Speculative sampling (Chen et al., 2023; Leviathan et al., 2022; Kim et al., 2023) ameliorates the memory bandwidth bottleneck by proposing multiple tokens with a smaller model which are then scored in parallel by a larger model.”
Fast Inference from Transformers via Speculative Decoding (2022, methodology): “After we initially published our work, an independent implementation of speculative decoding (Chen et al., 2023) showed similar 2X-2.”
EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (2024, methodology): “Speculative sampling (Leviathan et al., 2023; Chen et al., 2023a) based methods address this by dividing the process into a low-cost draft stage and a parallelized verification stage over the drafted tokens, allowing for multiple tokens to be validated in a single LLM pass.”
Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation (2022, methodology): “Chen et al. (2023) is similar to Leviathan et al. (2023) but it was the first to validate this methodology to accelerate a large language model (i.e., 70B Chinchilla) with a 4B drafter model, thus receiving the most attention.”
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024, methodology): “Following Stern et al. (2019) and Xia et al. (2022), subsequent work extended Speculative Decoding to support nucleus sampling (Leviathan et al., 2023; Chen et al., 2023a), accelerating the target LLM’s inference without changing its output distribution.”
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (2023, neutral): “Speculative decoding has recently emerged for LLM inference acceleration without retraining or model architecture modiﬁcation Leviathan et al. (2023); Chen et al. (2023); Sun et al. (2023).”
Break the Sequential Dependency of LLM Inference Using Lookahead Decoding (2024, methodology): “Using sampling gives smaller speedups as the acceptance ratio is lower according to the sampling verification algorithm 4, which aligns with the results in the previous research (Chen et al., 2023; Leviathan et al., 2023).”
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (2024, methodology): “Due to the proprietary nature of speculative decoding methods (Chen et al., 2023; Leviathan al., 2022), open-source alternatives 3 were deployed for evaluation.”
Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (2024, methodology): “With the rapid development of large language models, advanced inference techniques [Kwon et al. 2023; Chen et al. 2023a; Dettmers 2022] are proposed in the LLM community to optimize the inference speed.”
A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models (2024, neutral): “A cross-attention layer is trained and GPT-2 [96] is used as the decoder to produce captions.”
SqueezeLLM: Dense-and-Sparse Quantization (2023, background): “While some approaches concentrate on enhancing the efficiency of the decoding process [6, 26, 37, 54, 70], another line of research focuses on making the Transformer architecture itself more efficient.”
Large Language Models as Tool Makers (2023, methodology): “The ability of LLMs to self-debug has been demonstrated effectively in recent research (Madaan et al., 2023; Chen et al., 2023c; Lu et al., 2023; Kim et al., 2023).”
LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding (2024, positive): “Moreover, a popular research trend in LLM acceleration is speculative decoding Leviathan et al. (2023); Chen et al. (2023) that has no drop in accuracy, where a large model, referred to as the main model, is accompanied with a faster model, referred to as the draft model.”
Big Little Transformer Decoder (2023, methodology): “Concurrently, [4] also proposes an interesting algorithm to accelerate generative inference using a more powerful model to score and speculatively sample predictions from a less powerful model.”
Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text (2024, neutral): “Speculative decoding is similar (Chen et al., 2023; Leviathan et al., 2023), it uses a weaker model to plan completions.”
EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees (2024, methodology): “We compare EAGLE-2 with recent lossless speculative sampling methods, including standard speculative sampling (Leviathan et al., 2023; Chen et al., 2023a; Joao Gante, 2023), PLD (Saxena, 2023), Medusa (Cai et al., 2024), Lookahead (Fu et al., 2023), Hy-dra (Ankner et al., 2024), and EAGLE (Li et…”