To prune, or not to prune: exploring the efficacy of pruning for model compression

By Michael Zhu, Suyog Gupta. 2017

1,473 citations recorded by the source corpus.

Model pruning seeks to induce sparsity in a deep neural network's various connection matrices, thereby reducing the number of nonzero-valued parameters in the model. Recent reports (Han et al., 2015; Narang et al., 2017) prune deep networks at the cost of only a marginal loss in accuracy and achieve a sizable reduction in model size. This hints at the possibility that the baseline models in these experiments are perhaps severely over-parameterized at the outset and a viable alternative for model compression might be to simply reduce the number of hidden units while maintaining the model's dense connection structure, exposing a similar trade-off in model size and accuracy. We investigate these two distinct paths for model compression within the context of energy-efficient inference in resource-constrained environments and propose a new gradual pruning technique that is simple and straightforward to apply across a variety of models/datasets with minimal tuning and can be seamlessly incorporated within the training process. We compare the accuracy of large, but pruned models (large-sparse) and their smaller, but dense (small-dense) counterparts with identical memory footprint. Across a broad range of neural network architectures (deep CNNs, stacked LSTM, and seq2seq LSTM models), we find large-sparse models to consistently outperform small-dense models and achieve up to 10x reduction in number of non-zero parameters with minimal loss in accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

The State of Sparsity in Deep Neural Networks (2019, methodology): “Our main contributions: (1) We perform a comprehensive evaluation of variational dropout (Molchanov et al., 2017), l0 regularization (Louizos et al., 2017b), and magnitude pruning (Zhu & Gupta, 2017) on Transformer trained on WMT 2014 English-to-German and ResNet-50 trained on ImageNet.”
Efficient Neural Audio Synthesis (2018, methodology): “We use a pruning scheme based on the weight magnitude that increases sparsity as training proceeds (Narang et al., 2017a; Zhu & Gupta, 2017).”
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (2019, methodology): “This agrees with the observations made in works on model compression: sparse architectures learned through pruning cannot be trained from scratch to the same test set performance as a model trained with joint sparsification and optimization (Zhu and Gupta, 2017; Gale et al., 2019).”
Advances and Open Problems in Federated Learning (2019, background): “This topic has received a lot of attention in the broader ML community [194, 120, 436, 270, 312], but these methods either do not have a straightforward mapping to federated learning, or make the training process more complex which makes it difficult to adopt.”
Dynamic Model Pruning with Feedback (2020, methodology): “Lastly, our algorithm gradually increases the sparsity s t of the mask from 0 to the desired sparsity using the same scheduling as in (Zhu & Gupta, 2017); see Appendix A.2.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “Further to this, some works have focused on other computer vision problems, such as image generation [170], action recognition [171], multi-modal reasoning [172], label-to-image translation [173], and multi-task”
Compacting, Picking and Growing for Unforgetting Continual Learning (2019, methodology): “Set an accuracy goal for task 1; Alternatively remove small weights and re-train the remaining weights for task 1 via gradual pruning [51], whenever the accuracy goal is still hold; Let the model weights preserved for task 1 be W 1 (referred to as task-1 weights), and those that are removed by the i”
Generative Speech Coding with Predictive Variance Regularization (2021, methodology): “For other hidden layers, we applied iterative magnitude pruning to remove 92% of the model weights [28].”
Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning (2020, methodology): “This procedure is conveniently implemented in a Tensorﬂow (Abadi et al., 2016) package 4 , which we use (Zhu and Gupta, 2017).”
Sparse Networks from Scratch: Faster Training without Losing Performance (2019, methodology): “Extensions to this work include: Compressing recurrent neural networks and other models (Narang et al., 2017; Zhu and Gupta, 2018; Dai et al., 2018), continuous pruning and re-training (Guo et al., 2016), joint loss/pruning-cost optimization (Carreira-Perpinán and Idelbayev, 2018), layer-by-layer…”
Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization (2019, methodology): “As in (Zhu & Gupta, 2017), the pruning schedule we used was s(t) = s− s ( 1− t T )3 , (2) where t = 1, · · · , T indexes pruning steps, and s the target sparsity reached at the end of training.”
Movement Pruning: Adaptive Sparsity by Fine-Tuning (2020, methodology): “We follow Zhu and Gupta [2018] and use a cubic sparsity scheduling for Magnitude Pruning (MaP), Movement Pruning (MvP), and Soft Movement Pruning (SMvP).”
Sparse GPU Kernels for Deep Learning (2020, methodology): “1) Experimental Setup: We introduce sparsity into the 1 × 1 convolutions of MobileNetV1 using magnitude pruning [5].”
Characterising Bias in Compressed Models (2020, methodology): “We evaluate three different compression approaches: magnitude pruning (Zhu & Gupta, 2017), ﬁxed point 8-bit quantization (Jacob et al., 2017) and hybrid 8-bit quantization with dynamic range (Williamson, 1991).”