LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression

By Yihuan Mao, Yujing Wang, Chufan Wu, Chen Zhang, Yang Wang, Yaming Yang, Quanlu Zhang, Yunhai Tong, Jing Bai. 2020

81 citations recorded by the source corpus.

BERT is a cutting-edge language representation model pre-trained by a large corpus, which achieves superior performances on various natural language understanding tasks. However, a major blocking issue of applying BERT to online services is that it is memory-intensive and leads to unsatisfactory latency of user requests, raising the necessity of model compression. Existing solutions leverage the knowledge distillation framework to learn a smaller model that imitates the behaviors of BERT. However, the training procedure of knowledge distillation is expensive itself as it requires sufficient training data to imitate the teacher model. In this paper, we address this issue by proposing a tailored solution named LadaBERT (Lightweight adaptation of BERT through hybrid model compression), which combines the advantages of different model compression methods, including weight pruning, matrix factorization and knowledge distillation. LadaBERT achieves state-of-the-art accuracy on various public datasets while the training overheads can be reduced by an order of magnitude.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Efficiency & Compression

Citations in the corpus

ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Model compression has been explored from different aspects [25, 37, 39, 34, 43, 20, 24, 50, 18, 74, 40, 26, 55, 59, 28, 60, 68, 33, 14, 38, 31].”
TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020, methodology): “Instead of directly being used for compression, knowledge distillation can also be used in combination with other compression methods like pruning (McCarley, 2019; Mao et al., 2020), low-rank approximation (Mao et al.”
Utility Is in the Eye of the User: A Critique of NLP Leaderboard Design (2020, background): “There are many others like it as well (Zadeh and Moshovos, 2020; Hou et al., 2020; Mao et al., 2020).”
Block Pruning For Faster Transformers (2021, methodology): “While Mao et al. (2020) combine SVD-based matrix factorization with un-structured pruning, Wang et al. (2019) use structured pruning in order to reduce the rank.”
Extreme Compression for Pre-trained Transformers Made Simple and Efficient (2022, background): “Many research efforts have been made to compress these huge transformer models including knowledge distillation [20, 54, 53, 61], pruning [57, 43, 8], and low-rank decomposition [30].”
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2020, unknown): “The importance of weights can be judged by their absolute values, gradients, or a custom-designed measurement (Gordon et al., 2020; Mao et al., 2020; Guo et al., 2019; Sanh et al., 2020; Chen et al., 2020).”