Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding
By Song Han, Huizi Mao, William (Bill) J. Dally. 2015
10,226 citations recorded by the source corpus.
Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems with limited hardware resources. To address this limitation, we introduce "deep compression", a three stage pipeline: pruning, trained quantization and Huffman coding, that work together to reduce the storage requirement of neural networks by 35x to 49x without affecting their accuracy. Our method first prunes the network by learning only the important connections. Next, we quantize the weights to enforce weight sharing, finally, we apply Huffman coding. After the first two steps we retrain the network to fine tune the remaining connections and the quantized centroids. Pruning, reduces the number of connections by 9x to 13x; Quantization then reduces the number of bits that represent each connection from 32 to 5. On the ImageNet dataset, our method reduced the storage required by AlexNet by 35x, from 240MB to 6.9MB, without loss of accuracy. Our method reduced the size of VGG-16 by 49x from 552MB to 11.3MB, again with no loss of accuracy. This allows fitting the model into on-chip SRAM cache rather than off-chip DRAM memory. Our compression method also facilitates the use of complex neural networks in mobile applications where application size and download bandwidth are constrained. Benchmarked on CPU, GPU and mobile GPU, compressed network has 3x to 4x layerwise speedup and 3x to 7x better energy efficiency.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Quoc V. Le
- Luke S. Zettlemoyer
- Dacheng Tao
- Percy Liang
- Sergey Levine
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Yi Yang
- Jitendra Malik
- Alan L. Yuille
In Efficiency & Compression
- Song Han
- Dan Alistarh
- Kurt Keutzer
- Georgios Georgiadis
- Tijmen Blankevoort
- Jonathan Frankle
- Yunhe Wang
- Ji Lin
Citations in the corpus
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, background): “Many of those previous studies [19, 20, 43, 27] however mostly focus on CNN models with relatively few layers.”
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, methodology): “For example, model compression can reduce inference latency to enable real-time applications like simultaneous MT (See et al., 2016) or reduce memory usage to save energy for mobile devices (Han et al., 2016).”
- Sequence-Level Knowledge Distillation (2016, methodology): “…Mariet and Sra, 2016), zero-out weights based on an importance criterion: LeCun et al. (1990) use (a diagonal approximation to) the Hessian to identify weights whose removal minimally impact the objective function, while Han et al. (2016) remove weights based on thresholding their absolute values.”
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, background): “Model compression (Han et al., 2016; He et al., 2018; Yang et al., 2018) is a common way to reduce model size by trading accuracy for efficiency.”
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, methodology): “[37] Song Han, Huizi Mao, and William J Dally.”
- Billion-Scale Similarity Search with GPUs (2017, background): “It turns out that accepting a minimal accuracy loss results in orders of magnitude of compression [21].”
- SGDR: Stochastic Gradient Descent with Warm Restarts (2016, methodology): “…whether our results for individual models and ensembles can be further improved by using their networks instead of WRNs. Deep compression methods (Han et al., 2015) can be used to reduce the time and memory costs of DNNs and their ensembles. images from all 1000 classes downsampled to 32 × 32…”
- FastText.zip: Compressing text classification models (2016, background): “Recently, several research efforts have been conducted to compress the parameters of architectures involved in computer vision, namely for state-of-the-art Convolutional Neural Networks (CNNs) (Han et al., 2016; Lin et al., 2015).”
- Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (2022, methodology): “Magnitude pruning [18, 29] can achieve arbitrary levels of sparsity “by design” since it removes exactly the proportion of parameters with lowest magnitudes in order to match the desired density.”
- Measuring the Intrinsic Dimension of Objective Landscapes (2018, background): “( v ) There is a beautiful array of papers on compressing networks such that they also achieve computational savings during the forward pass (Wen et al., 2016; Han et al., 2016; Yang et al., 2015); subspace training does not speed up execution time during inference.”
- The case for 4-bit precision: k-bit Inference Scaling Laws (2022, background): “…Rusci et al., 2020), centering (Krishnamoorthi, 2018; Jacob et al., 2017), learned data types that are found through clustering (Gong et al., 2014; Han et al., 2015; Choi et al., 2016; Park et al., 2017), or direct codebook optimization (Rastegari et al., 2016; Hou et al., 2016; Leng et al.,…”
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (2022, background): “It proves to be effective for various convolutional neural works (CNNs) (Han et al., 2016; Jacob et al., 2018; Nagel et al., 2019; Wang et al., 2019; Lin et al., 2020) and transformers (Shen et al., 2020; Kim et al., 2021; Liu et al., 2021; Wang et al., 2020; Bondarenko et al., 2021).”
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, methodology): “Previous approaches include pruning near zero weights (Gong et al., 2014; Han et al., 2015a) using matrix factorization techniques (Zhang et al., 2015), quantizing the weights (Gupta et al., 2015), using shared weights (Chen et al., 2015) and applying Huffman codes (Han et al., 2015a) among others.”
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications (2017, methodology): “XIW +SSKPI( SSHPIF] 7EVEL, EVVMWSR Figure 1. MobileNet models can be applied to various recognition tasks for efficient on device intelligence. [2], and pruning, vector quantization and Huffman coding [5] have been proposed in the literature. Additionally various factorizations have been proposed to ”
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, methodology): “Techniques for eliminating unnecessary weights from neural networks (pruning) (LeCun et al., 1990; Hassibi & Stork, 1993; Han et al., 2015; Li et al., 2016) can reduce parameter-counts by more than 90% while maintaining accuracy.”
- Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions (2021, methodology): “, ResNet, ResNeXt), there are still some specific modules and operations designed for CNNs and not considered in this work, such as SE [22], SK [35], dilated convolution [73], model pruning [19], and NAS [60].”