Consistent Accelerated Inference via Confident Adaptive Transformers

By Tal Schuster, Adam Fisch, Tommi S. Jaakkola, Regina Barzilay. 2021

86 citations recorded by the source corpus.

We develop a novel approach for confidently accelerating inference in the large and expensive multilayer Transformers that are now ubiquitous in natural language processing (NLP). Amortized or approximate computational methods increase efficiency, but can come with unpredictable performance costs. In this work, we present CATs – Confident Adaptive Transformers – in which we simultaneously increase computational efficiency, while guaranteeing a specifiable degree of consistency with the original model with high confidence. Our method trains additional prediction heads on top of intermediate layers, and dynamically decides when to stop allocating computational effort to each input using a meta consistency classifier. To calibrate our early prediction stopping rule, we formulate a unique extension of conformal prediction. We demonstrate the effectiveness of this approach on four classification and regression tasks.

Subfields: Artificial Intelligence, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Fast Inference from Transformers via Speculative Decoding (2022, methodology): “These adaptive computation methods (e.g. Han et al., 2021; Sukhbaatar et al., 2019; Schuster et al., 2021; Scardapane et al., 2020; Bapna et al., 2020; Elbayad et al., 2019; Schwartz et al., 2020) aim to use less compute resources for easier inference steps.”
Efficient Transformers: A Survey (2020, background): “Another line of work involved fast exit during inference which allows us to exit compute if the model is confident of its predictions [65].”
Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space (2022, methodology): “Such methods often use small neural models to determine when to stop the execution process (Schwartz et al., 2020b; Elbayad et al., 2020; Hou et al., 2020; Xin et al., 2020, 2021; Li et al., 2021; Schuster et al., 2021).”
A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification (2021, background): “One compelling use-case of conformal prediction is speeding up and decreasing the computational cost of the test-time evaluation of complex models [98, 99].”
Conformal Risk Control (2022, background): “Most relevant to us is recent work on risk control in high probability (Vovk, 2012; Bates et al., 2021; Angelopoulos et al., 2021a) and its applications (Park et al., 2020; Fisch et al., 2022; Schuster et al., 2021; 2022; Sankaranarayanan et al., 2022; An-gelopoulos et al., 2022a;b, inter alia ).”
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners (2023, background): “In contrast to typical calibration methods that associate uncertainty with point-valued outputs, CP-based methods for language modeling provide coverage guarantees for set-valued predictors [42, 43, 44, 45].”
A Survey on Dynamic Neural Networks for Natural Language Processing (2022, background): “DeeBERT (Xin et al., 2020b) two-stage; sum of CE loss entropy < θ RightTool (Schwartz et al., 2020) joint; sum of CE loss calibrated max class probability > θ FastBERT (Liu et al., 2020) two-stage; self-distillation entropy < θ RomeBERT (Geng et al., 2021) joint; self-distillation + GR entropy < θ S”
Confident Adaptive Language Modeling (2022, methodology): “These measures can be calibrated to reliably guarantee consistency of the early prediction with the full model [57].”