EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees

By Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang. Conference on Empirical Methods in Natural Language Processing, 2024

331 citations recorded by the source corpus.

Inference with modern Large Language Models (LLMs) is expensive and time-consuming, and speculative sampling has proven to be an effective solution. Most speculative sampling methods such as EAGLE use a static draft tree, implicitly assuming that the acceptance rate of draft tokens depends only on their position. Interestingly, we found that the acceptance rate of draft tokens is also context-dependent. In this paper, building upon EAGLE, we propose EAGLE-2, which introduces a new technique of context-aware dynamic draft tree into drafting modeling. This improvement leverages the fact that the draft model of EAGLE is well-calibrated: the confidence scores from the draft model approximate acceptance rates with small errors. We conducted extensive evaluations on three series of LLMs and six tasks, with EAGLE-2 achieving speedup ratios of up to **5x**, which is 1.3x that of EAGLE. EAGLE-2 also ensures that the distribution of the generated text remains unchanged, making it a **lossless** acceleration algorithm.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/cab58a0263d454604896dce6b8fbf4df1dd99ff0