Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation

By Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, Zhifang Sui. 2022

189 citations recorded by the source corpus.

We propose Speculative Decoding (SpecDec), for the first time ever, to formally study exploiting the idea of speculative execution to accelerate autoregressive (AR) decoding. Speculative Decoding has two innovations: Spec-Drafter -- an independent model specially optimized for efficient and accurate drafting -- and Spec-Verification -- a reliable method for verifying the drafted tokens efficiently in the decoding paradigm. Experimental results on various seq2seq tasks including machine translation and abstractive summarization show our approach can achieve around $5\times$ speedup for the popular Transformer architectures with comparable generation quality to beam search decoding, refreshing the impression that the draft-then-verify paradigm introduces only $1.4\times$$\sim$$2\times$ speedup. In addition to the remarkable speedup, we also demonstrate 3 additional advantages of SpecDec, revealing its practical value for accelerating generative models in real-world applications. Our models and codes are available at https://github.com/hemingkx/SpecDec.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing, Efficiency & Compression, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Heming Xia
2. Peiyi Wang
3. Zhifang Sui
4. Yuhui Li
5. Fangyun Wei
6. Chao Zhang
7. Hongyang Zhang
8. Zhihong Shao

In Molecular Biology

1. Peter Bailis
2. Tao Ge

In Natural Language Processing

1. Heming Xia
2. Peiyi Wang
3. Zhifang Sui
4. Zhihong Shao
5. Tianyu Liu
6. Qingxiu Dong
7. Yuhui Li
8. Fangyun Wei

In Efficiency & Compression

1. Zhihong Shao
2. Yuhui Li
3. Fangyun Wei
4. Chao Zhang
5. Daya Guo
6. Hongyang Zhang
7. Ion Stoica
8. Peter Bailis

In Signal Processing

1. Daya Guo
2. Ion Stoica
3. Peter Bailis
4. Yu Wu
5. Tao Ge

Citations in the corpus

• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “Also, the efficient inference techniques (Kwon et al., 2023; Leviathan et al., 2023; Xia et al., 2023, 2024), which determines the exploration efficiency of policy models, also play an exceedingly important role.”
• Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024, methodology): “To this end, Stern et al. (2018) To further unleash the potential of this promising paradigm, Xia et al. (2022) introduced Speculative Decoding (SpecDec), which utilizes an independent drafter, notably a specialized Non-Autoregressive Transformer, to perform the drafting task both accurately and…”
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (2024, background): “The key to enhancing acceleration in speculative sampling lies in reducing the time overhead and improving the acceptance rate of the draft by the original LLM (Chen et al., 2023b; Xia et al., 2023; Santilli et al., 2023).”
• Break the Sequential Dependency of LLM Inference Using Lookahead Decoding (2024, background): “It shows notable speedup without a draft model and can linearly decrease the decoding steps with exponential investment in per-step FLOPs.”