BERxiT: Early Exiting for BERT with Better Fine-Tuning and Extension to Regression

By Ji Xin, Raphael Tang, Yaoliang Yu, Jimmy J. Lin. 2021

165 citations recorded by the source corpus.

The slow speed of BERT has motivated much research on accelerating its inference, and the early exiting idea has been proposed to make trade-offs between model quality and efficiency. This paper aims to address two weaknesses of previous work: (1) existing fine-tuning strategies for early exiting models fail to take full advantage of BERT; (2) methods to make exiting decisions are limited to classification tasks. We propose a more advanced fine-tuning strategy and a learning-to-exit module that extends early exiting to tasks other than classification. Experiments demonstrate improved early exiting for BERT, with better trade-offs obtained by the proposed fine-tuning strategy, successful application to regression tasks, and the possibility to combine it with other acceleration methods. Source code can be found at https://github.com/castorini/berxit.

Subfields: Artificial Intelligence, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yi Tay
2. Mostafa Dehghani
3. Donald Metzler
4. Tommi S. Jaakkola
5. Regina Barzilay
6. Dara Bahri
7. Luke S. Zettlemoyer
8. Tal Schuster

In Large Language Models

1. Julian McAuley
2. Yi Tay
3. Canwen Xu
4. Mostafa Dehghani
5. Dara Bahri
6. Jai Gupta
7. Donald Metzler
8. Regina Barzilay

In Efficiency & Compression

1. Julian McAuley
2. Canwen Xu
3. Dara Bahri
4. Yi Tay
5. Donald Metzler
6. Regina Barzilay
7. Vinh Q. Tran
8. Adam Fisch

Citations in the corpus

• Consistent Accelerated Inference via Confident Adaptive Transformers (2021, background): “[45] propose a meta confidence classifier similar to ours.”
• A Survey on Dynamic Neural Networks for Natural Language Processing (2022, methodology): “…et al., 2021) joint; weighted sum of CE + imitation learning entropy < θ PCEE-BERT (2022a) joint; weighted sum of CE patience (#consistent IC confidence > θ ) BERxiT (Xin et al., 2021) alternate; sum of CE loss estimated confidence > θ CAT (Schuster et al., 2021) joint; avg. of CE loss estimated…”
• Confident Adaptive Language Modeling (2022, methodology): “, based on the softmax distribution), while others try to predict the routing in advance [46; 70], or train a small early-exit classifier [57; 84], as we also examine here.”
• Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space (2022, unknown): “Elbayad et al., 2020; Hou et al., 2020; Xin et al., 2020, 2021; Li et al., 2021; Schuster et al., 2021).”