The Right Tool for the Job: Matching Model and Instance Complexities

By Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, Noah A. Smith. 2020

197 citations recorded by the source corpus.

As NLP models become larger, executing a trained model requires significant computational resources incurring monetary and environmental costs. To better respect a given inference budget, we propose a modification to contextual representation fine-tuning which, during inference, allows for an early (and fast) “exit” from neural network calculations for simple instances, and late (and accurate) exit for hard instances. To achieve this, we add classifiers to different layers of BERT and use their calibrated confidence scores to make early exit decisions. We test our proposed modification on five different datasets in two tasks: three text classification datasets and two natural language inference benchmarks. Our method presents a favorable speed/accuracy tradeoff in almost all cases, producing models which are up to five times faster than the state of the art, while preserving their accuracy. Our method also requires almost no additional training resources (in either time or parameters) compared to the baseline BERT model. Finally, our method alleviates the need for costly retraining of multiple models at different levels of efficiency; we allow users to control the inference speed/accuracy tradeoff using a single trained model, by setting a single variable at inference time. We publicly release our code.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, background): “, 2021) as well as early exit strategies for inference (Liu et al., 2020; Schwartz et al., 2020b; Xin et al., 2020).”
Fast Inference from Transformers via Speculative Decoding (2022, methodology): “These adaptive computation methods (e.g. Han et al., 2021; Sukhbaatar et al., 2019; Schuster et al., 2021; Scardapane et al., 2020; Bapna et al., 2020; Elbayad et al., 2019; Schwartz et al., 2020) aim to use less compute resources for easier inference steps.”
Consistent Accelerated Inference via Confident Adaptive Transformers (2021, methodology): “As discussed in §1, our method is inspired by the approach of Schwartz et al. (2020) and others (Liu et al., 2020; Geng et al., 2021; Zhou et al., 2020), where they preempt computation if the softmax value of any early clas-siﬁer is above a predeﬁned threshold.”
Dynamic Neural Networks: A Survey (2021, background): “BERT [7]) for improving their efficiency on NLP tasks [57], [58], [59], [60].”
Pre-trained models for natural language processing: A survey (2020, methodology): “Early Exit DeeBERT [110], RightTool [111], FastBERT [112], PABEE [113], Liao et al.”
Measuring and Narrowing the Compositionality Gap in Language Models (2022, background): “…to use more computation to solve harder problems has been previously explored in both language (Graves, 2016; Seo et al., 2017; Dong et al., 2019; Schwartz et al., 2020; Schwarzschild et al., 2021; Nye et al., 2021; Wei et al., 2022b; Zhou et al., 2022) and vision (Bolukbasi et al., 2017; Huang…”
Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space (2022, methodology): “Such methods often use small neural models to determine when to stop the execution process (Schwartz et al., 2020b; Table 5: Evaluation results on the challenging subset of R EAL T OXIC P ROMPTS , showing the percentage of toxic completions for 6 toxicity attributes, as well as language model…”
BERxiT: Early Exiting for BERT with Better Fine-Tuning and Extension to Regression (2021, methodology): “Early exiting (Schwartz et al., 2020; Xin et al., 2020a; Liu et al., 2020) has been proposed to accelerate the inference of BERT and models with similar architecture, i.e., those comprising multiple transformer layers (Vaswani et al., 2017) with a classiﬁer at the top.”
FastFormers: Highly Efficient Transformer Models for Natural Language Understanding (2020, methodology): “For future directions, some other methods such as early exiting (Xin et al., 2020; Liu et al., 2020; Schwartz et al., 2020; Zhou et al., 2020) and linear time complexity self-attention models (Shen et al., 2018; Wang et al., 2020) could be added to the proposed recipes and possibly improve the…”
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression (2021, methodology): “Many dynamic inference acceleration methods (Xin et al., 2020b; Schwartz et al., 2020b; Liu et al., 2020; Xin et al., 2020a; Li et al., 2020) use entropy or the maximum value of the predicted probability distribution as the signal for early exiting.”
Adaptable Adapters (2022, background): “Schwartz et al. (2020) propose to add an output layer to each transformer layer.”
A-ViT: Adaptive Tokens for Efficient Vision Transformer (2021, background): “There are a number of ways to improve the efficiency of transformers including weight sharing across transformer blocks [26], dynamically controlling the attention span of each token [5, 40], allowing the model to output the result in an earlier transformer block [38, 56], and applying pruning [53].”
Confident Adaptive Language Modeling (2022, background): “Multiple early-exit techniques for encoder-only Transformers (e.g., BERT [20]) have been recently proposed [8; 34; 43; 44; 45; 60; 68; 83; 90; 92].”
BERT Loses Patience: Fast and Robust Inference with Early Exit (2020, methodology): “Very recently, FastBERT [29] and DeeBERT [30] adapted confidence-based BranchyNet [26] for PLMs while RightTool [31] leveraged the same early-exit criterion as in the Shallow-Deep Network [5].”
Deep Learning Through the Lens of Example Difficulty (2021, background): “While early-exit strategies in computer vision (Teerapittayanon et al., 2016; Huang et al., 2018) and NLP (Dehghani et al., 2018; Liu et al., 2020b; Schwartz et al., 2020; Xin et al., 2020) suggest predictions for easier examples require less computation, the above example difficulty notions do not ”