RomeBERT: Robust Training of Multi-Exit BERT

By Shijie Geng, Peng Gao, Zuohui Fu, Yongfeng Zhang. 2021

32 citations recorded by the source corpus.

BERT has achieved superior performances on Natural Language Understanding (NLU) tasks. However, BERT possesses a large number of parameters and demands certain resources to deploy. For acceleration, Dynamic Early Exiting for BERT (DeeBERT) has been proposed recently, which incorporates multiple exits and adopts a dynamic early-exit mechanism to ensure efficient inference. While obtaining an efficiency-performance tradeoff, the performances of early exits in multi-exit BERT are significantly worse than late exits. In this paper, we leverage gradient regularized self-distillation for RObust training of Multi-Exit BERT (RomeBERT), which can effectively solve the performance imbalance problem between early and late exits. Moreover, the proposed RomeBERT adopts a one-stage joint training strategy for multi-exits and the BERT backbone while DeeBERT needs two stages that require more training time. Extensive experiments on GLUE datasets are performed to demonstrate the superiority of our approach. Our code is available at https://github.com/romebert/RomeBERT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tommi S. Jaakkola
2. Regina Barzilay
3. Julian McAuley
4. Tal Schuster
5. Adam Fisch
6. Luke S. Zettlemoyer
7. Percy Liang
8. Caiming Xiong

In Computer Vision and Pattern Recognition

1. Julian McAuley
2. Regina Barzilay
3. Adam Fisch
4. Tal Schuster
5. Tommi S. Jaakkola
6. Canwen Xu
7. Percy Liang
8. Yongfeng Zhang

In Large Language Models

1. Julian McAuley
2. Canwen Xu
3. Regina Barzilay
4. Adam Fisch
5. Tal Schuster
6. Tommi S. Jaakkola

Citations in the corpus

• Consistent Accelerated Inference via Confident Adaptive Transformers (2021, methodology): “As discussed in §1, our method is inspired by the approach of Schwartz et al. (2020) and others (Liu et al., 2020; Geng et al., 2021; Zhou et al., 2020), where they preempt computation if the softmax value of any early clas-sifier is above a predefined threshold.”
• A Survey on Dynamic Neural Networks for Natural Language Processing (2022, background): “, 2020) two-stage; self-distillation entropy < θ RomeBERT (Geng et al., 2021) joint; self-distillation + GR entropy < θ SkipBERT (2022) joint; weighted sum of CE + KD max class probability > θ”