A Survey of Model Compression and Acceleration for Deep Neural Networks

By Yu Cheng, Duo Wang, Pan Zhou, Zhang Tao. 2017

1,247 citations recorded by the source corpus.

Deep neural networks (DNNs) have recently achieved great success in many visual recognition tasks. However, existing deep neural network models are computationally expensive and memory intensive, hindering their deployment in devices with low memory resources or in applications with strict latency requirements. Therefore, a natural thought is to perform model compression and acceleration in deep networks without significantly decreasing the model performance. During the past five years, tremendous progress has been made in this area. In this paper, we review the recent techniques for compacting and accelerating DNN models. In general, these techniques are divided into four categories: parameter pruning and quantization, low-rank factorization, transferred/compact convolutional filters, and knowledge distillation. Methods of parameter pruning and quantization are described first, after that the other techniques are introduced. For each category, we also provide insightful analysis about the performance, related applications, advantages, and drawbacks. Then we go through some very recent successful methods, for example, dynamic capacity networks and stochastic depths networks. After that, we survey the evaluation matrices, the main datasets used for evaluating the model performance, and recent benchmark efforts. Finally, we conclude this paper, discuss remaining the challenges and possible directions for future work.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

DISCO: Dynamic and Invariant Sensitive Channel Obfuscation for deep neural networks (2020, background): “work compression to minimize computational complexity and accelerate inference [41].”
VEGA: Towards an End-to-End Configurable AutoML Pipeline (2020, background): “There are many surveys (He et al., 2019; Z¨oller & Huber, 2019; Elsken et al., 2019; Cheng et al., 2017; Shorten & Khoshgoftaar, 2019) that have summarized classical algorithms on these topics.”
Differentiable Neural Input Search for Recommender Systems (2020, positive): “Network pruning, as one kind of model compression techniques, improves the efficiency of over-parameterized deep neural networks by removing redundant neurons or connections without damaging model performance [7, 14, 26].”
Pre-trained models for natural language processing: A survey (2020, background): “Although model compression is widely studied for CNNs in CV [18], compression for PTMs for NLP is just beginning.”
On Knowledge distillation from complex networks for response prediction (2019, background): “We now quickly review existing works which use the idea of knowledge distillation to build compact models (Cheng et al., 2017).”
Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2019, methodology): “general neural network structures, existing optimization methods can be divided into four categories [128]: 1) parameter pruning and sharing [129], [130], including also weights quantization [131]–[133]; 2) low-rank factorization [124]; 3) transferred/compact convolution filters [107], [134], [135];”
Reborn Filters: Pruning Convolutional Neural Networks with Limited Data (2020, background): “There are always strict limits on memory and inference speed of deep convolutional neural networks (CNNs) deployed on various low-end edge devices ( e.g. , mobile phones, tablets and wearable gadgets) (Cheng et al. 2017; Wang, Li, and Smola 2019).”
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect (2024, background): “We speculate that it may be due to significant cumulative errors in generative tasks.”
An Adversarial Feature Distillation Method for Audio Classification (2019, methodology): “The knowledge distillation method has a wide range of applications in model compression [17] and semi-supervised tasks [18].”
Fast Sparse ConvNets (2019, background): “Weight sparsity is generally known to lead [3] to theoretically smaller and more computationally efficient (in terms of number of floating-point operations) models, but it is often disregarded as a practical means of accelerating models because of the misconception that sparse operations cannot be f”
Parameter Prediction for Unseen Deep Architectures (2021, methodology): “Besides the works discussed in § 6, our work is also loosely related to other parameter prediction methods [113, 98, 114], analysis of graph structure of neural networks [46], knowledge distillation from multiple teachers [115], compression methods [116] and optimization-based initialization [117– 1”
FedVision: An Online Visual Object Detection Platform Powered by Federated Learning (2020, background): “Thus, neural network compression can be performed to reduce the sizes of the network by pruning less useful weight values while preserving model performance (Cheng et al. 2017).”
Compact and Computationally Efficient Representation of Deep Neural Networks (2018, background): “25 This fact motivated an entire research field of model compression [1] that aimed to reduce the 26 complexity of inference of deep neural networks.”
GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference (2020, background): “Another approach trains a student based on the output logits (input of the last layer’s softmax) [27].”
AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search (2020, neutral): “…Language Model Compression Existing efforts to compress pre-trained language models such as BERT can be broadly categorized into four lines: knowledge distillation (Hinton et al., 2015), parameter sharing (Ullrich et al., 2017), pruning (Cheng et al., 2017) and quantization (Han et al., 2016).”