Structured Pruning of Deep Convolutional Neural Networks
By Sajid Anwar, Kyuyeon Hwang, Wonyong Sung. 2015
824 citations recorded by the source corpus.
Real-time application of deep learning algorithms is often hindered by high computational complexity and frequent memory accesses. Network pruning is a promising technique to solve this problem. However, pruning usually results in irregular network connections that not only demand extra representation efforts but also do not fit well on parallel computation. We introduce structured sparsity at various scales for convolutional neural networks: feature map-wise, kernel-wise, and intra-kernel strided sparsity. This structured sparsity is very advantageous for direct computational resource savings on embedded computers, in parallel computing environments, and in hardware-based systems. To decide the importance of network connections and paths, the proposed method uses a particle filtering approach. The importance weight of each particle is assigned by assessing the misclassification rate with a corresponding connectivity pattern. The pruned network is retrained to compensate for the losses due to pruning. While implementing convolutions as matrix products, we particularly show that intra-kernel strided sparsity with a simple constraint can significantly reduce the size of the kernel and feature map tensors. The proposed work shows that when pruning granularities are applied in combination, we can prune the CIFAR-10 network by more than 70% with less than a 1% loss in accuracy.
Subfields: Artificial Intelligence, Computational Mathematics, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computational Mathematics
- Gaurav Menghani
- Suyog Gupta
- Michael Zhu
- Mark Anderson
- Sidak Pal Singh
- Carlos Gómez-Rodríguez
- Martin Jaggi
- Ari S. Morcos
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Yi Yang
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Alan L. Yuille
- Jitendra Malik
In Model Architectures
- Jiahui Yu
- Thomas S. Huang
- Martin Jaggi
- Gaurav Menghani
- Sidak Pal Singh
- Kumar Shridhar
- Felix Laumann
- Suyog Gupta
In Efficiency & Compression
Citations in the corpus
- A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, background): “Other similar work [2, 38, 6] to our work that reduces or removed the redundant connections or induces sparsity are motivated by the desire to speed up computation.”
- Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks (2018, methodology): “We note that pruning has been proposed in prior work for non-security reasons, specifically, to reduce the computational expense of evaluating a DNN [19,48,25,4,33].”
- To prune, or not to prune: exploring the efficacy of pruning for model compression (2017, methodology): “Within the context of reducing model size by removing redundant connections, several recent works (Anwar et al., 2015; Lebedev and Lempitsky, 2015; Li et al., 2016; Changpinyo et al., 2017) propose techniques to prune and induce sparsity in a structured way, motivated primarily by the desire to…”
- Model Fusion via Optimal Transport (2019, methodology): “Structured pruning [26–28] is an approach to model compression that aims to remove entire neurons or channels, resulting in an out-of-the-box reduction in inference costs, while affecting the performance minimally.”
- Channel Pruning for Accelerating Very Deep Neural Networks (2017, neutral): “Inference-time channel pruning is challenging, as reported by previous works [2, 40].”
- Pruning Convolutional Neural Networks for Resource Efficient Transfer Learning (2016, positive): “Our approach is in line with existing work in structured pruning [1], while largely orthogonal to fine-grained pruning of fully-connected layers [3] and speedup by reduced precision [2] or tensor decomposition [6].”
- On the importance of single directions for generalization (2018, methodology): “Interestingly, this corresponds to the ablation strategies generally used in the model pruning literature (Li et al., 2017, Anwar et al., 2015, Molchanov et al., 2017).”
- Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers (2018, methodology): “He et al. (2017) and Zhang et al. (2016), while others use a data-dependent approach through typically a joint optimization in generating pruning decisions, e.g. , Han et al. (2015) and Anwar et al. (2017).”
- Slimmable Neural Networks (2018, background): “uire specific software and hardware accelerators to speedup. Driven by this fact, Molchanov et al. (2016), Wen et al. (2016), Li et al. (2016a), Liu et al. (2017), He et al. (2017), Luo et al. (2017), Anwar et al. (2017), Kim et al. (2017) and Ye et al. (2018) encourage structured sparsity by pruning”
- Recovering from Random Pruning: On the Plasticity of Deep Convolutional Neural Networks (2018, methodology): “Lastly, in [1] authors has used particle filtering to prune out the filters.”
- Model Pruning Enables Efficient Federated Learning on Edge Devices (2019, methodology): “Besides these unstructured pruning methods, structured pruning was also studied [20], which, however, often requires specific network architectures and does not conform to the lottery ticket hypothesis.”
- LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS (2023, background): “Un-structured [37] and structured pruning [1, 27] remove components at weight-level and neuron (mostly channel) levels to provide a smaller network with a smaller or more efficient network architecture.”
- Accelerating Sparse Deep Neural Networks (2021, methodology): “Furthermore, the pattern used to prune models may adhere to a specific structure [10,17–19] or follow no structure at all [7,11].”
- A review of convolutional neural network architectures and their optimizations (2022, background): “were the first to propose the idea of kernel-level pruning in 2017, where the concept of intra kernel strided sparsity was introduced (Anwar et al. 2017).”
- Variational Convolutional Neural Network Pruning (2019, background): “However, most of these pruning approaches [1, 7, 16] need re-training stage to restore comparable accuracy, which increases computational complexity and is not friendly for operation.”
- Deep Autoencoder based Energy Method for the Bending, Vibration, and Buckling Analysis of Kirchhoff Plates (2020, methodology): “Moreover, the number of neurons on each hidden layer and the number of hidden layers can be arbitrarily chosen and are invariably determined through a trial and error procedure or a pruning technique [46].”
- Exploring the Regularity of Sparse Structure in Convolutional Neural Networks (2017, background): “Some literature attempts to explore one or a few possibilities among all choices, such as the investigattion of intrakernel strided pruning[12].”
- A comprehensive survey on model compression and acceleration (2020, background): “In addition, Anwar et al. (2017) introduces structured pruning of the convolutional layer (channel-wise, kernel-wise and intra-kernel strided sparsity) at different scales.”