Compressing Large-Scale Transformer-Based Models: A Case Study on BERT

By Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Y. Yang, Deming Chen, Marianne S. Winslett, Hassan Sajjad, Preslav Nakov. 2020

234 citations recorded by the source corpus.

Abstract Pre-trained Transformer-based models have achieved state-of-the-art performance for various Natural Language Processing (NLP) tasks. However, these models often have billions of parameters, and thus are too resource- hungry and computation-intensive to suit low- capability devices or applications with strict latency requirements. One potential remedy for this is model compression, which has attracted considerable research attention. Here, we summarize the research in compressing Transformers, focusing on the especially popular BERT model. In particular, we survey the state of the art in compression for BERT, we clarify the current best practices for compressing large-scale Transformer models, and we provide insights into the workings of various methods. Our categorization and analysis also shed light on promising future research directions for achieving lightweight, accurate, and generic NLP models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Systems

In Efficiency & Compression

Citations in the corpus

Pretrained Transformers Improve Out-of-Distribution Robustness (2020, methodology): “This highlights that testing model compression methods for BERT (Shen et al., 2020; Ganesh et al., 2020; Li et al., 2020) on only in-distribution examples gives”
Few-Shot Self-Rationalization with Natural Language Prompts (2021, background): “Thus, any efforts to reduce required memory such as compression (Ganesh et al., 2021) may be valuable for few-shot self-rationalization.”
A Primer in BERTology: What We Know About How BERT Works (2020, methodology): “In unstructured, the weights in the entire model are pruned irrespective of their location, as in magnitude pruning (Chen et al., 2020) or movement pruning (Sanh et al.”
Pre-trained models for natural language processing: A survey (2020, background): “There are four common ways to compress PTMs [42]: (1) pruning, which removes less important parameters, (2) weight quantization [36], which uses fewer bits to represent the parameters, (3) parameter sharing across similar model units, and (4) knowledge distillation [57], which trains a smaller stude”
The Lottery Ticket Hypothesis for Pre-trained BERT Networks (2020, background): “This includes pruning (in which parts of a model are removed) [27–31], quantization (in which parameters are represented with fewer bits) [32, 33], parameter-sharing (in which the same parameters are used in multiple parts of a model) [34–36], and distilliation (in which a smaller student model is t”
Intriguing Properties of Compression on Multilingual Models (2022, methodology): “IMP is widely used and competitive with far more compute inten-parameters that replicate the prior results. sive approaches (Gale et al., 2019; Gordon et al., 2020; Du et al., 2021b; Ganesh et al., 2021), while allowing us to sparsify to an exact level.”
Poor Man's BERT: Smaller and Faster Transformer Models (2020, methodology): “To this end people have explored various methods (Ganesh et al., 2020) to achieve this goal.”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “Pruning refers to identifying and removing redundant and/or less importantweights [50].”
BERT Busters: Outlier Dimensions that Disrupt Transformers (2021, unknown): “See overviews by Ganesh et al. (2020) and Rogers et al. (2020).”
GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference (2020, methodology): “Knowledge distillation approaches for BERT can be categorized to three groups based on what the student learns from the teacher [13].”