DynaBERT: Dynamic BERT with Adaptive Width and Depth

By Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Qun Liu. 2020

390 citations recorded by the source corpus.

The pre-trained language models like BERT and RoBERTa, though powerful in many natural language processing tasks, are both computational and memory expensive. To alleviate this problem, one approach is to compress them for specific tasks before deployment. However, recent works on BERT compression usually reduce the large BERT model to a fixed smaller size, and can not fully satisfy the requirements of different edge devices with various hardware performances. In this paper, we propose a novel dynamic BERT model (abbreviated as DynaBERT), which can run at adaptive width and depth. The training process of DynaBERT includes first training a width-adaptive BERT and then allows both adaptive width and depth, by distilling knowledge from the full-sized model to small sub-networks. Network rewiring is also used to keep the more important attention heads and neurons shared by more sub-networks. Comprehensive experiments under various efficiency constraints demonstrate that our proposed dynamic BERT (or RoBERTa) at its largest size has comparable performance as BERT (or RoBERTa), while at smaller widths and depths consistently outperforms existing BERT compression methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Efficiency & Compression

Citations in the corpus

TinyBERT: Distilling BERT for Natural Language Understanding (2019, neutral): “, 2020), pruning (Cui et al., 2019; McCarley, 2019; F. et al., 2020; Elbayad et al., 2020; Gordon et al., 2020; Hou et al., 2020) or quantization (Shen et al.”
AdapterDrop: On the Efficiency of Adapters in Transformers (2020, background): “(2) Robustly trained transformers in which the model depth can be reduced at run-time, thereby decreasing inference time dynamically (Fan et al., 2020; Elbayad et al., 2020; Xin et al., 2020; Hou et al., 2020).”
TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020, methodology): “…used for compression, knowledge distillation can also be used in combination with other compression methods like pruning (McCarley, 2019; Mao et al., 2020), low-rank approximation (Mao et al., 2020) and dynamic networks (Hou et al., 2020), to fully leverage the knowledge of the teacher BERT model.”
BinaryBERT: Pushing the Limit of BERT Quantization (2020, methodology): “Following (Jiao et al., 2020; Hou et al., 2020; Zhang et al., 2020), we adopt data augmentation with one training epoch in each stage on all GLUE tasks except for MNLI and QQP.”
TILDE: Term Independent Likelihood moDEl for Passage Re-ranking (2021, neutral): “…have attempted to dynamically identify appropriate points for early exciting for accelerating the BERT inference step, at the expense of small degradation of model quality [11, 26, 32]; yet, the up to 40% savings in runtime [26] still render these methods impractical for deployed search engines.”
A Survey on Vision Transformer (2020, background): “[51] propose to adaptively reduce the width and depth of the pre-defined transformer models, and obtains multiple models with different sizes simultaneously.”
IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers (2021, background): “These adaptive computation strategies help speed up the inference time of convolutional neural networks (CNNs) [33], recurrent neural network (RNNs) [15], and also self-attention based methods (BERT) [25].”
FastFormers: Highly Efficient Transformer Models for Natural Language Understanding (2020, methodology): “Voita et al. (2019); Michel et al. (2019); Hou et al. (2020) proposed methods to prune some of the heads of Multi-Head Attention (MHA) in Trans-former architecture.”
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers (2020, background): “…can also be utilized to initialize task-speciﬁc distillation (Sun et al., 2019a; Turc et al., 2019; Aguilar et al., 2019; Mukher-jee and Awadallah, 2020; Xu et al., 2020; Hou et al., 2020; Li et al., 2020), which uses a ﬁne-tuned teacher model to guide the training of the student on speciﬁc tasks.”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “Second, structured pruning prunes structured blocks of weights or even complete components of PLMs by reducing and simplifying certain modules such as attention heads [90] and Transformer layers [52].”
LLM-Pruner: On the Structural Pruning of Large Language Models (2023, methodology): “There are several ways to remove the structure, such as l1-dependent pruning [15, 64], first-order importance estimation [17], hessian-based estimation [20, 51] or the optimal brain surgeon [23, 20].”
Few-Shot Learning With a Strong Teacher (2021, background): “Distilling knowledge from one model to another has also been investigated for model interpretability [81], compression [75], [82], incremental learning [83], [84], etc.”
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2020, positive): “Earlier solutions following this methodology used recurrent neural networks (RNNs) as the generic base model, e.g. , ULMFiT (Howard and Ruder, 2018) and ELMo (Peters et al., 2018).”
The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation (2021, background): “To-date, evaluating the merits and tradeoffs incurred by compression have overwhelmingly centered on settings where the data is relatively abundant (Gale et al., 2019; Li et al., 2020a; Hou et al., 2020; Chen et al., 2021; Bai et al., 2020; ab Tessera et al., 2021).”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “While similarity-based approaches are common in encoder-based LMs (Sun et al., 2019, 2020; Jiao et al., 2020; Hou et al., 2020; Zuo et al., 2022; Liang et al., 2021), their application in LLM knowledge distillation is not as widespread.”
Utility Is in the Eye of the User: A Critique of NLP Leaderboard Design (2020, background): “There are many others like it as well (Zadeh and Moshovos, 2020; Hou et al., 2020; Mao et al., 2020).”
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (2023, background): “…often overparameterized and can be pruned significantly with minimal impact on performance (Han et al., 2016; Wen et al., 2016; Liu et al., 2017; Luo et al., 2017; Cai et al., 2019; Deng et al., 2020; Hou et al., 2020; Wang et al., 2020; Lagunas et al., 2021; Xia et al., 2022; Kurtic et al., 2023).”
Intriguing Properties of Compression on Multilingual Models (2022, background): “imal impact on aggregate performance numbers (Gale et al., 2019; Li et al., 2020; Hou et al., 2020; Chen et al., 2021; Bai et al., 2020; ab Tessera et al., 2021), the impact on individual sub-populations in the data, such as low-resource languages, can”