Pruning Filters for Efficient ConvNets
By Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf. 2016
4,125 citations recorded by the source corpus.
The success of CNNs in various applications is accompanied by a significant increase in the computation and parameter storage costs. Recent efforts toward reducing these overheads involve pruning and compressing the weights of various layers without hurting original accuracy. However, magnitude-based pruning of weights reduces a significant number of parameters from the fully connected layers and may not adequately reduce the computation costs in the convolutional layers due to irregular sparsity in the pruned networks. We present an acceleration method for CNNs, where we prune filters from CNNs that are identified as having a small effect on the output accuracy. By removing whole filters in the network together with their connecting feature maps, the computation costs are reduced significantly. In contrast to pruning weights, this approach does not result in sparse connectivity patterns. Hence, it does not need the support of sparse convolution libraries and can work with existing efficient BLAS libraries for dense matrix multiplications. We show that even simple filter pruning techniques can reduce inference costs for VGG-16 by up to 34% and ResNet-110 by up to 38% on CIFAR10 while regaining close to the original accuracy by retraining the networks.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Dacheng Tao
- Mohit Bansal
- Yoshua Bengio
- Quoc V. Le
- Jianfeng Gao
- Trevor Darrell
- Luke S. Zettlemoyer
- Percy Liang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Yi Yang
- Xiaogang Wang
- Ming-Hsuan Yang
- Trevor Darrell
- Jitendra Malik
- Alan L. Yuille
In Efficiency & Compression
Citations in the corpus
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, methodology): “…factor that our larger models 7 Since the reduction in memory from storing sparse matrices is highly dependent on the data structure used, we follow past work and report the number of nonzero model parameters (Luo et al., 2017; Li et al., 2017). are also less converged on the pretraining task.”
- AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “Structure pruning and growing were proposed for different goals, such as efficient inference [7, 8, 11, 13, 14, 18, 19, 24, 25, 27, 38–40], lifelong learning [41] and model adaptation [10, 29].”
- Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (2022, methodology): “We compare with layer-wise structured magnitude pruning [29] †† .”
- Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “Some works prune neurons/channels individually based on their input or output weights norm (He et al., 2014; Li et al., 2017), or activations entropy (He et al.”
- ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Model compression has been explored from different aspects [25, 37, 39, 34, 43, 20, 24, 50, 18, 74, 40, 26, 55, 59, 28, 60, 68, 33, 14, 38, 31].”
- Reducing Transformer Depth on Demand with Structured Dropout (2019, methodology): “However, it has been more widely adopted in computer vision and applied to convolutional network to remove filters (Li et al., 2016; Wen et al., 2016), channels (He et al.”
- Are Sixteen Heads Really Better than One? (2019, background): “, 2015) and structured pruning (Anwar et al., 2017; Li et al., 2016; Molchanov et al., 2017), wherein entire parts of the model are pruned.”
- MobileNetV2: Inverted Residuals and Linear Bottlenecks (2018, methodology): “methods of network pruning [12, 13, 14, 15, 16, 17] and connectivity learning [18, 19].”
- Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT (2019, methodology): “[18] H.”
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “Techniques for eliminating unnecessary weights from neural networks (pruning) (LeCun et al., 1990; Hassibi & Stork, 1993; Han et al., 2015; Li et al., 2016) can reduce parameter-counts by more than 90% while maintaining accuracy.”
- Distilling Task-Specific Knowledge from BERT into Simple Neural Networks (2019, background): “Thus, others explore pruning entire filters (Li et al., 2016; Liu et al., 2017), with some even targeting device-centric metrics, such as floating-point operations (Tang et al.”
- What is the State of Neural Network Pruning? (2020, methodology): “Other methods consider parameters in groups (structured pruning), removing entire neurons, filters, or channels to exploit hardware and software optimized for dense computation (Li et al., 2016; He et al., 2017).”
- Weight Agnostic Neural Networks (2019, background): “Network Pruning By removing connections with small weight values from a trained neural network, pruning approaches [36, 40, 44, 63, 68, 72, 73, 75, 81] can produce sparse networks that keep only a small fraction of the connections, while maintaining similar performance on image classification tasks ”
- Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware (2018, methodology): “Our work is also orthogonal and complementary to techniques that modify DNNs to speed up inference [21, 30, 42, 27, 15].”
- Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks (2018, methodology): “We note that pruning has been proposed in prior work for non-security reasons, specifically, to reduce the computational expense of evaluating a DNN [19,48,25,4,33].”
- To prune, or not to prune: exploring the efficacy of pruning for model compression (2017, methodology): “Within the context of reducing model size by removing redundant connections, several recent works (Anwar et al., 2015; Lebedev and Lempitsky, 2015; Li et al., 2016; Changpinyo et al., 2017) propose techniques to prune and induce sparsity in a structured way, motivated primarily by the desire to…”
- Zero-shot Knowledge Transfer via Adversarial Belief Matching (2019, background): “, 2015), network pruning (Li et al., 2016; Han et al., 2016) and quantization (Gupta et al.”
- AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, neutral): “Quantitative comparison with the structural pruning (Li et al., 2016) and CEC (Shu et al.”
- Differentiable Neural Input Search for Recommender Systems (2020, methodology): “pruning [17, 23, 30] generally performed iterative pruning and finetuning over certain pretrained over-parameterized deep network.”