Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning

By Mitchell A. Gordon, Kevin Duh, Nicholas Andrews. 2020

378 citations recorded by the source corpus.

Pre-trained universal feature extractors, such as BERT for natural language processing and VGG for computer vision, have become effective methods for improving deep learning models without requiring more labeled data. While effective, feature extractors like BERT may be prohibitively large for some deployment scenarios. We explore weight pruning for BERT and ask: how does compression during pre-training affect transfer learning? We find that pruning affects transfer learning in three broad regimes. Low levels of pruning (30-40%) do not affect pre-training loss or transfer to downstream tasks at all. Medium levels of pruning increase the pre-training loss and prevent useful pre-training information from being transferred to downstream tasks. High levels of pruning additionally prevent models from fitting downstream datasets, leading to further degradation. Finally, we observe that fine-tuning BERT on a specific task does not improve its prunability. We conclude that BERT can be pruned once during pre-training rather than separately for each task without affecting performance.

Subfields: Artificial Intelligence, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

Citations in the corpus

TinyBERT: Distilling BERT for Natural Language Understanding (2019, neutral): “, 2020), pruning (Cui et al., 2019; McCarley, 2019; F. et al., 2020; Elbayad et al., 2020; Gordon et al., 2020; Hou et al., 2020) or quantization (Shen et al.”
On the Predictability of Pruning Across Scales (2020, methodology): “In practice, pruning can reduce the parameter-counts of contemporary models by 2x (Gordon et al., 2020) to 5x (Renda et al., 2020) with no increase in error.”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Model compression has been explored from diﬀerent aspects [25, 37, 39, 34, 43, 20, 24, 50, 18, 74, 40, 26, 55, 59, 28, 60, 68, 33, 14, 38, 31].”
Pre-Trained Models: Past, Present and Future (2021, background): “Gordon et al. (2020) show that low levels of pruning (30 – 40”
Pre-trained models for natural language processing: A survey (2020, neutral): “[103] explored the timing of pruning (e.”
When BERT Plays the Lottery, All Tickets Are Winning (2020, neutral): “ subnetwork type is shown inFigure 2.The main LTH prediction is validated: the “good” subnetworks can be successfully retrained alone. Our m-pruning results are consistent with contemporaneous work byGordon et al. (2020) andChen et al.(2020). We observe the following differences between the two prun”
Movement Pruning: Adaptive Sparsity by Fine-Tuning (2020, background): “As observed by Gordon et al. [2020], ﬁne-tuned weights stay close in absolute value to their initial pre-trained values.”
I-BERT: Integer-only BERT Quantization (2021, background): “These techniques can be broadly categorized into: (1) pruning (Fan et al., 2019; Gordon et al., 2020; Han et al., 2015; LeCun et al., 1990; Li et al., 2016b; Mao et al., 2017; 2020; Michel et al., 2019; Molchanov et al., 2016; Raganato et al., 2020; Sanh et al., 2020; Yang et al., 2017); (2) knowled”
Outliers Dimensions that Disrupt Transformers Are Driven by Frequency (2022, methodology): “The current Transformer-based language models are heavily overparametrized, which explains why it is possible to prune these models by up to 30-40% (Gordon et al., 2020; Sanh et al., 2020; Prasanna et al., 2020; Chen et al., 2020, inter alia) without a significant drop in performance.”
BERxiT: Early Exiting for BERT with Better Fine-Tuning and Extension to Regression (2021, background): “Pruning (Han et al., 2015; Fan et al., 2020; Gordon et al., 2020) removes unim-portant parts of the neural model, from individual weights to layers and blocks.”
Analyzing Redundancy in Pretrained Transformer Models (2020, background): “While the work on pretrained model compression (Cao et al., 2020; Shen et al., 2020; Sanh et al., 2019; Turc et al., 2019; Gordon et al., 2020; Guyon and Elisseeff, 2003) indirectly shows that models exhibit redundancy, little has been done to explore the redundancy in the network.”
A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development (2023, methodology): “Distillation-based approaches, such as those proposed in [49], [115]−[117], involve training a smaller student model using extra training data or soft labels generated by a larger teacher model (LLM or ChatGPT).”
Block Pruning For Faster Transformers (2021, methodology): “When targeting transformer models, it is typical to select the weights to prune based on their magnitude (Gordon et al., 2020), or by computing an importance score using a ﬁrst-order method (Sanh et al., 2020).”
BERT Busters: Outlier Dimensions that Disrupt Transformers (2021, background): “Some of the recent ﬁndings are that the lottery ticket hypothesis (Frankle and Carbin, 2019) holds for BERT: its largest weights do form subnetworks that can be retrained alone to reach the performance close to that of the full model (Prasanna et al., 2020; Chen et al., 2020; Gordon et al., 2020).”