Compression of Deep Learning Models for Text: A Survey

By Manish Gupta, Puneet Agrawal. 2020

148 citations recorded by the source corpus.

In recent years, the fields of natural language processing (NLP) and information retrieval (IR) have made tremendous progress thanks to deep learning models like Recurrent Neural Networks (RNNs), Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTMs) networks, and Transformer [ 121 ] based models like Bidirectional Encoder Representations from Transformers (BERT) [ 24 ], Generative Pre-training Transformer (GPT-2) [ 95 ], Multi-task Deep Neural Network (MT-DNN) [ 74 ], Extra-Long Network (XLNet) [ 135 ], Text-to-text transfer transformer (T5) [ 96 ], T-NLG [ 99 ], and GShard [ 64 ]. But these models are humongous in size. On the other hand, real-world applications demand small model size, low response times, and low computational power wattage. In this survey, we discuss six different types of methods (Pruning, Quantization, Knowledge Distillation (KD), Parameter Sharing, Tensor Decomposition, and Sub-quadratic Transformer-based methods) for compression of such models to enable their deployment in real industry NLP projects. Given the critical need of building applications with efficient and small models, and the large amount of recently published work in this area, we believe that this survey organizes the plethora of work done by the “deep learning for NLP” community in the past few years and presents it as a coherent story.

Subfields: Artificial Intelligence, Computational Mathematics, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tianyi Zhou
2. Saleh Ashkboos
3. Can Xu
4. Tao Shen
5. Dacheng Tao
6. Ajit Rajasekharan
7. Torsten Hoefler
8. Marcelo Gennari Do Nascimento

In Computational Mathematics

1. Tao Shen
2. Dacheng Tao
3. Tianyi Zhou
4. Ming Li
5. Jinyang Li
6. Xiaohan Xu
7. Reynold Cheng
8. Can Xu

In Natural Language Processing

1. Tao Shen
2. Dacheng Tao
3. Ajit Rajasekharan
4. Torsten Hoefler
5. Tianyi Zhou
6. Ming Li
7. Jinyang Li
8. Xiaohan Xu

In Efficiency & Compression

1. Saleh Ashkboos
2. Can Xu
3. Dacheng Tao
4. Ajit Rajasekharan
5. Torsten Hoefler
6. Marcelo Gennari Do Nascimento
7. Maximilian L. Croci
8. James Hensman

Citations in the corpus

• Efficiently Scaling Transformer Inference (2022, neutral): “Several approaches (Gupta and Agrawal, 2020) to improve the inference efficiency of Trans-former models focus on model architecture improvements, for example efficient attention layers (Roy et al., 2020; Choromanski et al., 2020; Kitaev et al., 2020; Sukhbaatar et al., 2019; Child et al., 2019),…”
• Sparse is Enough in Scaling Transformers (2021, neutral): “Numerous techniques improve the efficiency of Transformer models, and Gupta and Agrawal [11] divide them into several classes, including pruning, knowledge distillation, quantization, parameter sharing, efficient attention, and efficient feedforward.”
• A Survey on Knowledge Distillation of Large Language Models (2024, background): “Recognizing the disparities between proprietary and open-source LLMs, the field of NLP has witnessed a surge in the adoption of knowledge distillation techniques (Gou et al., 2021; Gupta and Agrawal, 2022) as a means to bridge this performance gap.”
• SliceGPT: Compress Large Language Models by Deleting Rows and Columns (2024, background): “A majority of model compression techniques fall into one of four categories: distillation, tensor decomposition (which includes low-rank factorization), pruning and quantization (Hoefler et al., 2021; Gholami et al., 2021; Zhu et al., 2023; Gupta & Agrawal, 2021).”
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “As these models have many layers through which the input has to pass through to get the model prediction, latency is high [162].”