Fast Inference from Transformers via Speculative Decoding

By Yaniv Leviathan, Matan Kalman, Yossi Matias. International Conference on Machine Learning, 2022

1,765 citations recorded by the source corpus.

Inference from large autoregressive models like Transformers is slow - decoding K tokens takes K serial runs of the model. In this work we introduce speculative decoding - an algorithm to sample from autoregressive models faster without any changes to the outputs, by computing several tokens in parallel. At the heart of our approach lie the observations that (1) hard language-modeling tasks often include easier subtasks that can be approximated well by more efficient models, and (2) using speculative execution and a novel sampling method, we can make exact decoding from the large models faster, by running them in parallel on the outputs of the approximation models, potentially generating several tokens concurrently, and without changing the distribution. Our method can accelerate existing off-the-shelf models without retraining or architecture changes. We demonstrate it on T5-XXL and show a 2X-3X acceleration compared to the standard T5X implementation, with identical outputs.

Subfields: Artificial Intelligence, Hardware and Architecture, Information Systems and Management, Large Language Models

View Paper: https://arxiv.org/abs/2211.17192

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Hardware and Architecture

In Information Systems and Management

In Large Language Models

Citations in the corpus

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023, background): “Recently, decoder-only models are extremely popular, and since these models do not have separate self-attention and cross-attention, we expect GQA to have a stronger advantage over MQA.”
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “Also, the efficient inference techniques (Kwon et al., 2023; Leviathan et al., 2023; Xia et al., 2023, 2024), which determines the exploration efficiency of policy models, also play an exceedingly important role.”
Accelerating Large Language Model Decoding with Speculative Sampling (2023, background): “Coincidentally, the work in this manuscript was undertaken concurrently and independently of the work on speculative decoding from Leviathan et al. (2022).”
Break the Sequential Dependency of LLM Inference Using Lookahead Decoding (2024, methodology): “Several approaches have been proposed – one such approach is speculative decoding (Chen et al., 2023; Leviathan et al., 2023) and its variants (He et al., 2023; Stern et al., 2018; Cai et al., 2024; Li et al., 2023; Liu et al., 2023; Miao et al., 2023).”
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (2023, neutral): “Speculative decoding has recently emerged for LLM inference acceleration without retraining or model architecture modiﬁcation Leviathan et al. (2023); Chen et al. (2023); Sun et al. (2023).”
Multi-Agent Collaboration Mechanisms: A Survey of LLMs (2025, neutral): “Adapters are adaptive intelligent modules that allow the agent to incorporate additional knowledge from others through mechanisms such as speculative decoding and parameter-efficient adapter, which can further enrich the model’s response capabilities [40, 72, 98].”
Large Language Models as Tool Makers (2023, methodology): “In addition, recent research has proposed methods to adaptively control decoding in LLMs to improve text generation efficiency (Leviathan et al., 2022; Chen et al., 2023a; Xia et al., 2023).”
Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation (2022, methodology): “G.1 Speculative Decoding with an Autoregressive Drafter Following this work, some subsequent research (Leviathan et al., 2023; Chen et al., 2023) has also explored using AR models (e.g., smaller language models) as independent drafters to accelerate inference.”
Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (2023, methodology): “…explored different methodologies, such as pre-training (Azer-bayev et al., 2023), fine-tuning (Luo et al., 2023; Yu et al., 2023b; Wang et al., 2023b), prompting (Wei et al., 2022; Fu et al., 2022), and verification (Wang et al., 2023d; Li et al., 2023b; Zhu et al., 2023; Leviathan et al., 2023).”
QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models (2023, methodology): “We consider an individual user application, like (Frantar et al., 2022; Leviathan et al., 2023; Park et al., 2022), where a single prompt (sampled from C4) should be processed to generate a 128-token response.”
A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models (2024, neutral): “A cross-attention layer is trained and GPT-2 [96] is used as the decoder to produce captions.”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “Considering the difficulties of generation, speculative decoding [122] utilizes small but efficient models to assist the generation of PLMs.”
RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment (2023, background): “The full list of hyper-parameters can be found in Appendix C.”
Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training (2023, neutral): “• Tree-Search with speculative decoding Speculative decoding (Leviathan et al., 2023) is a pivotal technique to accelerate LLM inference by employing a smaller draft model to predict the target model’s outputs.”
EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (2024, methodology): “Speculative sampling (Leviathan et al., 2023) conducted experiments with a batch size of 1, a setting subsequently adopted by other works such as DistillSpec (Zhou et al., 2023) and BiLD (Kim et al., 2023).”
Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, neutral): “…of training builds upon the retriever trained in the previous cycle, facilitating iterative optimization in subsequent rounds. f) Retrieval Transformation : Retrieval Transformation involves rephrasing retrieved content to better activate the generator’s potential, resulting in improved output.”
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models (2023, neutral): “Its format is as follows (three instances, for example), 1 { 2 "type": "text_only", 3 "instances": [ 4 { "text": "SAMPLE_TEXT_1" },”
Robotic Control via Embodied Chain-of-Thought Reasoning (2024, neutral): “We note that further optimizations (both with TensorRT-LLM or with other techniques, like speculative decoding [71]) are possible and that the policy can be made smoother with techniques like action chunking [2], but we leave these approaches to future works.”
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024, methodology): “Following SpecDec, Leviathan et al. (2023) and Chen et al. (2023a) made concurrent contributions by proposing Speculative Sampling, expanding this paradigm to encompass the lossless acceleration of nucleus sampling.”