BERT Loses Patience: Fast and Robust Inference with Early Exit

By Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, Furu Wei. 2020

447 citations recorded by the source corpus.

In this paper, we propose Patience-based Early Exit, a straightforward yet effective inference method that can be used as a plug-and-play technique to simultaneously improve the efficiency and robustness of a pretrained language model (PLM). To achieve this, our approach couples an internal-classifier with each layer of a PLM and dynamically stops inference when the intermediate predictions of the internal classifiers remain unchanged for a pre-defined number of steps. Our approach improves inference efficiency as it allows the model to make a prediction with fewer layers. Meanwhile, experimental results with an ALBERT model show that our method can improve the accuracy and robustness of the model by preventing it from overthinking and exploiting multiple classifiers for prediction, yielding a better accuracy-speed trade-off compared to existing early exit methods.

Subfields: Artificial Intelligence, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Efficiency & Compression

Citations in the corpus

Towards Interpretable Natural Language Understanding with Explanations as Latent Variables (2020, neutral): “, DistilBERT [35], BERT-of-Theseus [36], DeeBERT [37], PABEE [38], etc.”
Consistent Accelerated Inference via Confident Adaptive Transformers (2021, methodology): “(2020) and others (Liu et al., 2020; Geng et al., 2021; Zhou et al., 2020), where they preempt computation if the softmax value of any early classifier is above a predefined threshold.”
Dynamic Neural Networks: A Survey (2021, background): “BERT [7]) for improving their efficiency on NLP tasks [57], [58], [59], [60].”
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (2020, methodology): “We will also investigate the combination of our compression-based approach with recently proposed dynamic acceleration method (Zhou et al., 2020b) to further improve the efficiency of pretrained language models.”
Pre-trained models for natural language processing: A survey (2020, methodology): “Early Exit DeeBERT [110], RightTool [111], FastBERT [112], PABEE [113], Liao et al.”
DynaBERT: Dynamic BERT with Adaptive Width and Depth (2020, background): “In [4, 7, 6, 13, 29, 34], Transformerbased models with adaptive depth are proposed to dynamically select some of the Transformer layers during inference.”
TILDE: Term Independent Likelihood moDEl for Passage Re-ranking (2021, unknown): “Recent works have attempted to dynamically identify appropriate points for early exciting for accelerating the BERT inference step, at the expense of small degradation of model quality [11, 26, 32]; yet, the up to 40% savings in runtime [26] still render these methods impractical for deployed search”
Lossless Acceleration for Seq2seq Generation with Aggressive Decoding (2022, background): “…of light-weighted models to handle various instances based on their difficulties, and dynamic early exiting (Xin et al., 2020; Liu et al., 2020; Zhou et al., 2020; Sun et al., 2021) that introduces internal classifiers within deep neural networks to allow early exiting if the prediction is…”
Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges (2021, methodology): “[172] GLUE [146] CoLA [151], SST-2 [131], MRPC [22], STS-B [9], QQP [54], MNLI [152], QNLI [114], WNLI [71], RTE [7, 18, 33, 37] BERT [21] ALBERT [66] A, C, L, T Link”
Early Exiting BERT for Efficient Document Ranking (2020, methodology): “The idea of early exiting for neural networks originates from BranchyNet (Teerapittayanon et al., 2017), and is also applied to NLP tasks in several papers (Xin et al., 2020; Schwartz et al., 2020; Liu et al., 2020; Zhou et al., 2020).”
A-ViT: Adaptive Tokens for Efficient Vision Transformer (2021, background): “There are a number of ways to improve the efficiency of transformers including weight sharing across transformer blocks [26], dynamically controlling the attention span of each token [5, 40], allowing the model to output the result in an earlier transformer block [38, 56], and applying pruning [53].”
Confident Adaptive Language Modeling (2022, unknown): “Multiple early-exit techniques for encoder-only Transformers (e.g., BERT [20]) have been recently proposed [8; 34; 43; 44; 45; 60; 68; 83; 90; 92].”
A Survey on Dynamic Neural Networks for Natural Language Processing (2022, methodology): “For example, some works use the ratio of layers involved in computation to estimate speed-up ratio (Zhou et al., 2020a; Sun et al., 2021; Liao et al., 2021).”
UnNatural Language Inference (2020, background): “High conﬁdence of Transformer models can be attributed to the overthinking phenomenon commonly observed in deep neural networks (Kaya et al., 2019) and BERT-based models (Zhou et al., 2020).”
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2020, unknown): “Training these classiﬁers can either be done from scratch (Xin et al., 2020; Zhou et al., 2020b; Tambe et al., 2020) or through distilling the output from the ﬁnal classiﬁer (Liu et al., 2020).”
RomeBERT: Robust Training of Multi-Exit BERT (2021, background): “PABEE (Zhou et al., 2020) takes into consideration the agreement among multi-exits and explores inference with multiple classifiers.”
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression (2021, background): “Moreover, when discussing ethical concerns, previous studies (Wang et al., 2020; Zhou et al., 2020) ignore the risk that model compression could introduce additional biases.”
BERxiT: Early Exiting for BERT with Better Fine-Tuning and Extension to Regression (2021, background): “PABEE does not provide exact speedup numbers; therefore we estimate the values from their ﬁgures.”