Pruning Convolutional Neural Networks for Resource Efficient Transfer Learning

By Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, Jan Kautz. 2016

439 citations recorded by the source corpus.

We propose a new framework for pruning convolutional kernels in neural networks to enable efficient inference, focusing on transfer learning where large and potentially unwieldy pretrained networks are adapted to specialized tasks. We interleave greedy criteria-based pruning with fine-tuning by backpropagation - a computationally efficient procedure that maintains good generalization in the pruned network. We propose a new criterion based on an efficient first-order Taylor expansion to approximate the absolute change in training cost induced by pruning a network component. After normalization, the proposed criterion scales appropriately across all layers of a deep CNN, eliminating the need for per-layer sensitivity analysis. The proposed criterion demonstrates superior performance compared to other criteria, such as the norm of kernel weights or average feature map activation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

The State of Sparsity in Deep Neural Networks (2019, methodology): “…recent work has achieved comparable compression levels with more computationally efficient first-order loss approximations, and further refinements have related this work to efficient empirical estimates of the Fisher information of the model parameters (Molchanov et al., 2016; Theis et al., 2018).”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “…2016), redundancy (Mariet & Sra, 2016; Srinivas & Babu, 2015a), per-layer second derivatives (Dong et al., 2017), and energy/computation efficiency (Yang et al., 2017) (e.g., pruning convolutional filters (Li et al., 2016; Molchanov et al., 2016; Luo et al., 2017) or channels (He et al., 2017)).”
Efficient Structured Pruning and Architecture Searching for Group Convolution (2018, background): “Pruning is a promising approach that removes parameters contributing little to test accuracy, and its success has been demonstrated in numerous prior works [6, 5, 29, 24].”
AMC: AutoML for Model Compression and Acceleration on Mobile Devices (2018, methodology): “Channel pruning [40, 24, 1, 38] removes the redundant channels from feature maps.”
MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks (2017, background): “FLOPs-based constraints have been studied previously [22, 30, 17], although we believe our work is the ﬁrst to tackle the issue via cleverly designed sparsifying regularizers.”
TSViz: Demystification of Deep Learning Models for Time-Series Analysis (2018, neutral): “the loss [29], [30].”
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, methodology): “However, all such methods have a significant constraint – they assume that either the previously used training dataset is available [8, 29, 45, 57], or some real images representative of the prior training dataset distribution are available [25, 26, 34, 56].”
ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression (2017, methodology): “The core is to evaluate neuron importance, which has been widely studied in the community [34, 27, 21, 14, 23].”
NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications (2018, background): “[6, 14, 16] aim to remove individual redundant weights from DNNs.”
NISP: Pruning Networks Using Neuron Importance Score Propagation (2017, methodology): “Some previous works [8, 9, 10] approximate it using 2nd-order Taylor expansion.”
Rigging the Lottery: Making All Tickets Winners (2019, neutral): “Saliency has been used as a criterion to remove connections (Molchanov et al., 2016), however it has been shown to produce inferior results compared to magnitude based removal, especially when used to remove multiple connections at once (Evci, 2018).”
Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers (2018, methodology): “Based on this discussion, many existing works which claim to use Lasso, group Lasso (e.g. Wen et al. (2016); Anwar et al. (2017)), or thresholding (e.g. Molchanov et al. (2017)) to enforce parameter sparsity have some theoretical gaps to bridge.”
Importance Estimation for Neural Network Pruning (2019, methodology): “We focus our attention on extending previously proposed methods [2, 22, 27] with a new pruning criterion and a method that iteratively removes the least important set of neurons (typically filters) from the trained model.”
Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks (2018, background): “[ Molchanov et al. , 2017 ] proposes a Taylor expansion based pruning criterion to approximate the change in the cost function induced by pruning.”
Balanced Sparsity for Efficient DNN Inference on GPU (2018, positive): “Driven by the great potentials to reduce the model sizes for accelerating DNNs, a series of work (Han et al. 2015; Guo, Yao, and Chen 2016; Molchanov et al. 2016; LeCun, Denker, and Solla 1990; Engelbrecht 2001) identify and zero out the unimportant weights at a high compression ratio.”
Progressive Blockwise Knowledge Distillation for Neural Network Acceleration (2018, methodology): “, 2017]; 2) parameter pruning and sharing [Courbariaux et al., 2015; Hu et al., 2016; Li et al., 2017a; Luo et al., 2017; Molchanov et al., 2017; Li et al., 2017c]; 3) matrix factorization [Tai et al.”
Fast Neural Architecture Construction using EnvelopeNets (2018, methodology): “(28) also use pruning to remove duplicates and to reduce model size.”
Exploring the Regularity of Sparse Structure in Convolutional Neural Networks (2017, neutral): “Second-order derivative[13], absolute value[7], lossapproximating Taylor expansion[10] and output sensitivity[14] are examples of importance metrics used for pruning.”
Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration (2018, methodology): “Some filter pruning approaches [23, 25, 16, 27, 7, 33, 39, 37, 46, 14, 18, 22] need to utilize training data to determine the pruned filters.”
Learning a Wavelet-like Auto-Encoder to Accelerate Deep Neural Networks (2017, methodology): “Comparison with state-of-the-art methods ThiNet (Luo, Wu, and Lin 2017) and Taylor 3 (Molchanov et al. 2016) are two newest methods that also focus on accelerating deep CNNs, and they also conduct experiments on VGG16-Net.”