BERT-of-Theseus: Compressing BERT by Progressive Module Replacing

By Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, Ming Zhou. 2020

228 citations recorded by the source corpus.

In this paper, we propose a novel model compression approach to effectively compress BERT by progressive module replacing. Our approach first divides the original BERT into several modules and builds their compact substitutes. Then, we randomly replace the original modules with their substitutes to train the compact modules to mimic the behavior of the original modules. We progressively increase the probability of replacement through the training. In this way, our approach brings a deeper level of interaction between the original and compact models. Compared to the previous knowledge distillation approaches for BERT compression, our approach does not introduce any additional loss function. Our approach outperforms existing knowledge distillation approaches on GLUE benchmark, showing a new perspective of model compression.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Efficiency & Compression

Citations in the corpus

TinyBERT: Distilling BERT for Natural Language Understanding (2019, neutral): “, 2020), BERT-of-Theseus (Xu et al., 2020) and MiniLM (Wang et al.”
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables (2020, neutral): “(e.g., DistilBERT [35], BERT-of-Theseus [36], DeeBERT [37], PABEE [38], etc.)”
A Primer in BERTology: What We Know About How BERT Works (2020, methodology): “Other techniques include decomposing BERT’s embedding matrix into smaller matrices (Lan et al., 2019) and progressive model replacing (Xu et al., 2020).”
Representation Compensation Networks for Continual Semantic Segmentation (2022, background): “During training, similar to [33, 90], these frozen teacher layers will impose regularization to trainable parameters, encouraging trainable layers to behave like the teacher model.”
Pre-trained models for natural language processing: A survey (2020, methodology): “[196] proposed Theseus Compression motivated by a famous thought experiment called “Ship of Theseus”, which progressively substitutes modules from the source model with modules of fewer parameters.”
Compressing Visual-linguistic Model via Knowledge Distillation (2021, background): “, BERT [14]), and these works show that mimicking the distribution of self-attention and intermediate representations of transformer blocks increases performances [57, 33, 63, 75] for downstream tasks.”
I-BERT: Integer-only BERT Quantization (2021, background): “, 2017); (2) knowledge distillation (Hinton et al., 2014; Jiao et al., 2019; Mishra & Marr, 2017; Polino et al., 2018; Romero et al., 2014; Sanh et al., 2019; Sun et al., 2019; 2020; Tang et al., 2019; Turc et al., 2019; Wang et al., 2020; Xu et al., 2020); (3) efficient neural architecture design (”
A Survey on Vision Transformer (2020, unknown): “94 of-Theseus [190] design Q-BERT [191] Quantization 12 Q8BERT [192] 12 TinyBERT [45]”
BERT Loses Patience: Fast and Robust Inference with Early Exit (2020, methodology): “2 BERT-PKD [23] 66M 1.”
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2020, unknown): “Otherwise, a smaller model of ﬁxed depth is obtained (Sajjad et al., 2020; Xu et al., 2020).”
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 (2021, unknown): “[26, 58], quantization [118, 153], factorized embedding parameterization and cross-layer parameter sharing [70], and progressive module replacing [146].”
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression (2021, methodology): “For downstream KD and BERTof-Theseus, we initialize the model by truncating the first (bottom) 6 layers of the finetuned teacher, following the original papers (Sun et al., 2019; Xu et al., 2020).”
SqueezeBERT: What can computer vision teach NLP about efficient neural networks? (2020, background): “And, on the GLUE benchmark, which consists of 9 natural language understanding (NLU) datasets [16], MobileBERT achieves higher accuracy than other efﬁcient networks such as DistilBERT [17], PKD [18], and several others [19, 20, 21, 22].”
Adversarial Self-Supervised Data Free Distillation for Text Classification (2020, background): “Considering that the embedding layer is separated from transformer blocks when training, we strongly recommend sharing the ﬁrst layer’s parameters of the teacher with the student, which is also suggested in Xu et al. (2020).”
Pre-trained Summarization Distillation (2020, background): “Bert-of-Theseus (Xu et al., 2020) randomly replaces multiple teacher layers with a single student layer during ﬁne-tuning with probability r , such that each student layer learns to replicate 2 teacher layers.”
Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT (2022, methodology): “, 2015) is a powerful technique that has been successfully deployed to train efficient "student" BERT models from larger "teacher" models (Sanh et al., 2019; Jiao et al., 2020; Sun et al., 2019; Xu et al., 2020).”
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers (2020, methodology): “…can also be utilized to initialize task-speciﬁc distillation (Sun et al., 2019a; Turc et al., 2019; Aguilar et al., 2019; Mukher-jee and Awadallah, 2020; Xu et al., 2020; Hou et al., 2020; Li et al., 2020), which uses a ﬁne-tuned teacher model to guide the training of the student on speciﬁc tasks.”
A Survey on Dynamic Neural Networks for Natural Language Processing (2022, background): “…static model acceleration, which is often referred to as model compression , including knowledge distillation, weight sharing, pruning and quantization (Sanh et al., 2019; Xu et al., 2020; Lan et al., 2020; Zafrir et al., 2019; Xu et al., 2021) (etc., see another survey (Xu and McAuley, 2022)).”