Confident Adaptive Language Modeling

By Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, Donald Metzler. 2022

329 citations recorded by the source corpus.

Recent advances in Transformer-based large language models (LLMs) have led to significant performance improvements across many tasks. These gains come with a drastic increase in the models' size, potentially leading to slow and costly use at inference time. In practice, however, the series of generations made by LLMs is composed of varying levels of difficulty. While certain predictions truly benefit from the models' full capacity, other continuations are more trivial and can be solved with reduced compute. In this work, we introduce Confident Adaptive Language Modeling (CALM), a framework for dynamically allocating different amounts of compute per input and generation timestep. Early exit decoding involves several challenges that we address here, such as: (1) what confidence measure to use; (2) connecting sequence-level constraints to local per-token exit decisions; and (3) attending back to missing hidden representations due to early exits in previous tokens. Through theoretical analysis and empirical experiments on three diverse text generation tasks, we demonstrate the efficacy of our framework in reducing compute -- potential speedup of up to $\times 3$ -- while provably maintaining high performance.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

A Review of Sparse Expert Models in Deep Learning (2022, methodology): “Still future models may benefit by combining other adaptive computation techniques – as an example, in addition to choosing which expert, a network might choose the number of layers to use, as well (Schuster et al., 2022).”
DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models (2023, background): “This finding is also consistent with the assumptions in early exiting LMs (Schuster et al., 2022).”
Big Little Transformer Decoder (2023, methodology): “To reproduce CALM [48] in our experimental setup, we have fine-tuned the pre-trained mT5-small model on IWSLT 2017 De-En and WMT 2014 De-En datasets.”
LightGlue: Local Feature Matching at Light Speed (2023, methodology): “This is inspired by multiple works that successfully apply this strategy to language and vision tasks [62, 20, 71, 80, 40].”
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners (2023, background): “In contrast to typical calibration methods that associate uncertainty with point-valued outputs, CP-based methods for language modeling provide coverage guarantees for set-valued predictors [42, 43, 44, 45].”
SqueezeLLM: Dense-and-Sparse Quantization (2023, background): “While some approaches concentrate on enhancing the efficiency of the decoding process [6, 26, 37, 54, 70], another line of research focuses on making the Transformer architecture itself more efficient.”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “This focus on cherry-picked results from toy models is a form of streetlight interpretability (Casper, 2023), examining AI systems under only ideal conditions of maximal interpretability.”
LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding (2024, unknown): “Figure 6 shows how range of decoding parameters varies for different tasks.”
Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models (2023, background): “…LLMs being overconfident (as discussed in (Mielke et al., 2022; Si et al., 2022)) is orthogonal to our work, as we evaluate measures basing how they rank different samples - such measures may then be calibrated by distribution-free uncertainty quantification meth-ods like in Schuster et al. (2022).”