Compact Deep Convolutional Neural Networks With Coarse Pruning

By Sajid Anwar, Wonyong Sung. 2016

58 citations recorded by the source corpus.

The learning capability of a neural network improves with increasing depth at higher computational costs. Wider layers with dense kernel connectivity patterns furhter increase this cost and may hinder real-time inference. We propose feature map and kernel level pruning for reducing the computational complexity of a deep convolutional neural network. Pruning feature maps reduces the width of a layer and hence does not need any sparse representation. Further, kernel pruning converts the dense connectivity pattern into a sparse one. Due to coarse nature, these pruning granularities can be exploited by GPUs and VLSI based implementations. We propose a simple and generic strategy to choose the least adversarial pruning masks for both granularities. The pruned networks are retrained which compensates the loss in accuracy. We obtain the best pruning ratios when we prune a network with both granularities. Experiments with the CIFAR-10 dataset show that more than 85% sparsity can be induced in the convolution layers with less than 1% increase in the missclassification rate of the baseline network.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Song Han
2. Quoc V. Le
3. Ji Lin
4. Dacheng Tao
5. Junjie Yan
6. Yi Yang
7. Chuang Gan
8. Zhijian Liu

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Song Han
4. Yi Yang
5. Trevor Darrell
6. Xiaogang Wang
7. Alan L. Yuille
8. Jitendra Malik

Citations in the corpus

• AMC: AutoML for Model Compression and Acceleration on Mobile Devices (2018, background): “Channel pruning [40, 24, 1, 38] removes the redundant channels from feature maps.”
• Channel Pruning for Accelerating Very Deep Neural Networks (2017, background): “[3] select channels via over 100 random trials, however it need long time to evaluate each trial on a deep network, which makes it infeasible to work on very deep models and large datasets.”
• Rethinking the Value of Network Pruning (2018, background): “Mittal et al. (2018) shows that random channel pruning (Anwar & Sung, 2016) can perform on par with a variety of more sophisticated pruning criteria, demonstrating the plasticity of network models.”
• Quantization Mimic: Towards Very Tiny CNN for Object Detection (2018, background): “[28,29] focus on the importance of different filter weights and do pruning operation according to weights importance.”
• Protecting Intellectual Property of Deep Neural Networks with Watermarking (2018, background): “The goal of model pruning is to reduce redundant parameters, but still keep the performance of original deep neural networks [8, 21, 41, 49, 51].”
• Edge Machine Learning for AI-Enabled IoT Devices: A Review (2020, background): “[75] selected pruning candidate by hundreds of random evaluations.”
• Structured Probabilistic Pruning for Convolutional Neural Network Acceleration (2017, background): “rows and columns of weight matrix), so that it can accelerate CNN computation without specific implementation and hardware modification [1, 26].”
• ADC: Automated Deep Compression and Acceleration with Reinforcement Learning (2018, background): “Channel pruning [47, 25, 2, 45] removed redundant channels on feature maps.”
• NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm (2017, methodology): “This method reduces FLOPs but incurs performance degradation [28].”