DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference

By Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, Jimmy J. Lin. 2020

534 citations recorded by the source corpus.

Large-scale pre-trained language models such as BERT have brought significant improvements to NLP applications. However, they are also notorious for being slow in inference, which makes them difficult to deploy in real-time applications. We propose a simple but effective method, DeeBERT, to accelerate BERT inference. Our approach allows samples to exit earlier without passing through the entire model. Experiments show that DeeBERT is able to save up to ~40% inference time with minimal degradation in model quality. Further analyses show different behaviors in the BERT transformer layers and also reveal their redundancy. Our work provides new ideas to efficiently apply deep transformer-based models to downstream tasks. Code is available at https://github.com/castorini/DeeBERT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Towards Interpretable Natural Language Understanding with Explanations as Latent Variables (2020, neutral): “, DistilBERT [35], BERT-of-Theseus [36], DeeBERT [37], PABEE [38], etc.”
AdapterDrop: On the Efficiency of Adapters in Transformers (2020, background): “(2) Robustly trained transformers in which the model depth can be reduced at run-time, thereby decreasing inference time dynamically (Fan et al., 2020; Elbayad et al., 2020; Xin et al., 2020; Hou et al., 2020).”
Consistent Accelerated Inference via Confident Adaptive Transformers (2021, methodology): “We ﬁx F rather than train it jointly with the new components of G to avoid any reduction in F ’s performance (Xin et al., 2020b).”
Dynamic Neural Networks: A Survey (2021, background): “, BERT [7]) for improving their efficiency on NLP tasks [57], [58], [59], [60].”
Pre-trained models for natural language processing: A survey (2020, methodology): “DeeBERT [110], FastBERT [112], Liao et al.”
BinaryBERT: Pushing the Limit of BERT Quantization (2020, background): “…weight-sharing (Dehghani et al., 2019; Lan et al., 2020; Huang et al., 2021), dynamic networks with adaptive depth and/or width (Hou et al., 2020; Xin et al., 2020; Zhou et al., 2020), and quantization (Zafrir Figure 1: Performance of quantized BERT with varying weight bit-widths and 8-bit…”
DynaBERT: Dynamic BERT with Adaptive Width and Depth (2020, background): “In [4, 7, 6, 13, 29, 34], Transformerbased models with adaptive depth are proposed to dynamically select some of the Transformer layers during inference.”
Confident Adaptive Language Modeling (2022, unknown): “Multiple early-exit techniques for encoder-only Transformers (e.g., BERT [20]) have been recently proposed [8; 34; 43; 44; 45; 60; 68; 83; 90; 92].”
Lossless Acceleration for Seq2seq Generation with Aggressive Decoding (2022, background): “, 2020) that sequentially applies a series of light-weighted models to handle various instances based on their difficulties, and dynamic early exiting (Xin et al., 2020; Liu et al., 2020; Zhou et al., 2020; Sun et al., 2021) that introduces internal classifiers within deep neural networks to allow e”
BERT Loses Patience: Fast and Robust Inference with Early Exit (2020, methodology): “We also add DeeBERT [29], a BranchyNet variant on BERT alongside our BranchyNet implementation.”
SqueezeBERT: What can computer vision teach NLP about efficient neural networks? (2020, unknown): “DeeBERT (Xin et al., 2020), FastBERT (Liu et al.”
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2020, unknown): “Training these separate classifiers can be done either from scratch (Xin et al., 2020; Zhou et al., 2020b; Tambe et al., 2020) or by distilling the output of the final classifier (Liu et al.”
Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges (2021, methodology): “For different tasks, Soldaini and Moschitti [132], Xin et al. [156] and Liu et al. [84] propose EE frameworks based on BERT [21] and RoBERTa [85] that share almost the same network architecture.”
TILDE: Term Independent Likelihood moDEl for Passage Re-ranking (2021, unknown): “…have attempted to dynamically identify appropriate points for early exciting for accelerating the BERT inference step, at the expense of small degradation of model quality [11, 26, 32]; yet, the up to 40% savings in runtime [26] still render these methods impractical for deployed search engines.”
SPINN: synergistic progressive inference of neural networks over device and cloud (2020, background): “A growing body of work from both the research [23, 35, 72, 81, 84] and industry communities [55, 74] has proposed transforming a given model into a progressive inference network by introducing intermediate exits throughout its depth.”
Deep Learning Through the Lens of Example Difficulty (2021, background): “While early-exit strategies in computer vision (Teerapittayanon et al., 2016; Huang et al., 2018) and NLP (Dehghani et al., 2018; Liu et al., 2020b; Schwartz et al., 2020; Xin et al., 2020) suggest predictions for easier examples require less computation, the above example difficulty notions do not ”
Transformers in Time Series: A Survey (2022, background): “…also have architecture-level model designs to ﬁt different purposes, such as lightweight [Wu et al. , 2020b; Mehta et al. , 2021], cross-block connectivity [ Bapna et al. , 2018 ] , adaptive computation time [ Dehghani et al. , 2019; Xin et al. , 2020 ] , and recurrence [ Dai et al. , 2019 ] .”
BERxiT: Early Exiting for BERT with Better Fine-Tuning and Extension to Regression (2021, methodology): “Early exiting (Schwartz et al., 2020; Xin et al., 2020a; Liu et al., 2020) has been proposed to accelerate the inference of BERT and models with similar architecture, i.e., those comprising multiple transformer layers (Vaswani et al., 2017) with a classiﬁer at the top.”
Surprisal-Triggered Conditional Computation with Neural Networks (2020, background): “In early exiting, a classiﬁer is attached to an intermediate layer of a network and trained to determine whether stopping at that layer will result in a misclassiﬁcation; if a misclassiﬁcation is not predicted, subsequent layers are not computed [13, 14, 15, 16, 17].”