Early Exiting BERT for Efficient Document Ranking

By Ji Xin, Rodrigo Nogueira, Yaoliang Yu, Jimmy J. Lin. 2020

49 citations recorded by the source corpus.

Pre-trained language models such as BERT have shown their effectiveness in various tasks. Despite their power, they are known to be computationally intensive, which hinders real-world applications. In this paper, we introduce early exiting BERT for document ranking. With a slight modification, BERT becomes a model with multiple output paths, and each inference sample can exit early from these paths. In this way, computation can be effectively allocated among samples, and overall system latency is significantly reduced while the original quality is maintained. Our experiments on two document ranking datasets demonstrate up to 2.5x inference speedup with minimal quality degradation. The source code of our implementation can be found at https://github.com/castorini/earlyexiting-monobert.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

R2-D2: A Modular Baseline for Open-Domain Question Answering (2021, methodology): “Yang et al. (2019) also combined BM25 ranker and reader scores via linear combination.”
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, background): “, 2021) as well as early exit strategies for inference (Liu et al., 2020; Schwartz et al., 2020b; Xin et al., 2020).”
Consistent Accelerated Inference via Confident Adaptive Transformers (2021, methodology): “We ﬁx F rather than train it jointly with the new components of G to avoid any reduction in F ’s performance (Xin et al., 2020b).”
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression (2021, methodology): “Many dynamic inference acceleration methods (Xin et al., 2020b; Schwartz et al., 2020b; Liu et al., 2020; Xin et al., 2020a; Li et al., 2020) use entropy or the maximum value of the predicted probability distribution as the signal for early exiting.”
A Survey on Dynamic Neural Networks for Natural Language Processing (2022, methodology): “DeeBERT (Xin et al., 2020b) two-stage; sum of CE loss entropy < θ RightTool (Schwartz et al., 2020) joint; sum of CE loss calibrated max class probability > θ FastBERT (Liu et al., 2020) two-stage; self-distillation entropy < θ RomeBERT (Geng et al., 2021) joint; self-distillation + GR entropy < θ…”
Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space (2022, unknown): “Elbayad et al., 2020; Hou et al., 2020; Xin et al., 2020, 2021; Li et al., 2021; Schuster et al., 2021).”
Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges (2021, background): “Specifically, these studies [24, 67, 87, 109, 132, 140, 141, 148, 150, 155, 157, 158, 168, 172] define a loss function for each of the classifier, and minimize the weighted sum of cross entropy losses per sample as follows: where 3) by treating the final classifier (last exit) as teacher model and…”
BERxiT: Early Exiting for BERT with Better Fine-Tuning and Extension to Regression (2021, methodology): “Early exiting (Schwartz et al., 2020; Xin et al., 2020a; Liu et al., 2020) has been proposed to accelerate the inference of BERT and models with similar architecture, i.e., those comprising multiple transformer layers (Vaswani et al., 2017) with a classiﬁer at the top.”