Importance Estimation for Neural Network Pruning

By Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, Jan Kautz. 2019

1,174 citations recorded by the source corpus.

Structural pruning of neural network parameters reduces computational, energy, and memory transfer costs during inference. We propose a novel method that estimates the contribution of a neuron (filter) to the final loss and iteratively removes those with smaller scores. We describe two variations of our method using the first and second-order Taylor expansions to approximate a filter's contribution. Both methods scale consistently across any network layer without requiring per-layer sensitivity analysis and can be applied to any kind of layer, including skip connections. For modern networks trained on ImageNet, we measured experimentally a high (>93%) correlation between the contribution computed by our methods and a reliable estimate of the true importance. Pruning with the proposed methods led to an improvement over state-of-the-art in terms of accuracy, FLOPs, and parameter reduction. On ResNet-101, we achieve a 40% FLOPS reduction by removing 30% of the parameters, with a loss of 0.02% in the top-1 accuracy on ImageNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

DISCO: Dynamic and Invariant Sensitive Channel Obfuscation for deep neural networks (2020, methodology): “While most methods interleave pruning with the training phase [42, 43, 44], there has been recent focus on pruning at inference [45].”
Network Pruning via Transformable Architecture Search (2019, neutral): “1.8% FPGM [19] 68.41% 1.87% 88.48% 1.15% 1.06E9 41.8% TAS 69.15% 1.50% 89.19% 0.68% 1.21E9 33.3% ResNet-50 SFP [18] 74.61% 1.54% 92.06% 0.81% 2.38E9 41.8% CP [20] - - 90.80% 1.40% 2.04E9 50.0% Taylor [38] 74.50% 1.68% - - 2.25E9 44.9% AutoSlim [45] 76.00% - - - 3.00E9 26.6% FPGM [19] 75.50% 0.65% 92”
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, methodology): “For example, one might want to use a smaller or more resource-efﬁcient network architecture while deploying on edge inference devices [44, 66, 76], or to adapt the network to the inference hardware [9, 63, 71], or for continuously learning to classify new image classes [29, 34], etc .”
Overcoming Catastrophic Forgetting by Neuron-level Plasticity Control (2019, methodology): “(Molchanov et al. 2019) Furthermore, the Taylor criterion is more computationally efficient than other methods such as (Yu et al. 2018; Luo and Wu 2017; Luo, Wu, and Lin 2017), since Taylor criterion is computed from the gradient of the loss function with respect to the neurons, which is computed…”
AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (2023, background): “If the removal of a parameter has a large influence, then the model is sensitive to it and we should retain it (Molchanov et al., 2019; Liang et al., 2021; Zhang et al., 2022).”
DepGraph: Towards Any Structural Pruning (2023, background): “It is also worth noting that in structural pruning, grouped layers are pruned simultaneously, which expects the parameters in the removed group to be consistently unimportant, which brings certain difficulties to existing importance criteria designed for a single layer [20,26,28,39].”
Pruning neural networks without any data by iteratively conserving synaptic flow (2020, background): “The resulting metric is also closely related to ∂ L ∂θ θ (cid:12)(cid:12) the score used in SNIP [13], − H ∂ L ∂θ θ the score used in GraSP, and ∂ L ∂θ θ 2 the score used in the pruning after training algorithm Taylor-FO [27].”
GhostNet: More Features From Cheap Operations (2019, neutral): “1 Taylor-FO-BN-ResNet-50 [41] 7.”
FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout (2021, methodology): “The majority of pruning methods [21, 19, 40, 38, 53] aim to generate a single pruned model and require access to labelled data in order to perform a costly ﬁne-tuning/calibration for each pruned variant.”
Group Fisher Pruning for Practical Network Compression (2021, methodology): “…2.33 - DCP (Zhuang et al., 2018) 74.95 76.01 1.81 - AOFP (Ding et al., 2019b) 75.11 75.34 1.77 - FPGM (He et al., 2019) 74.83 76.15 1.90 1.62 IE (Molchanov et al., 2019) 74.50 76.18 2.25 - C-SGD (Ding et al., 2019a) 74.54 75.33 2.20 - Meta (Liu et al., 2019) 75.40 76.60 2.0 - GBN (You et al.,…”
OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networks (2022, methodology): “• Model compression methods [6] prune [53, 29, 56, 24, 37] or share [19] weights to improve the efficiency of the model parameterization.”
Neural Pruning via Growing Regularization (2020, methodology): “…Mozer & Smolensky (1989); Reed (1993), the central problem of neural network pruning has been (arguably) how to choose weights to discard, i.e., the weight importance scoring problem LeCun et al. (1990); Hassibi & Stork (1993); Molchanov et al. (2017b; 2019); Wang et al. (2019a); He et al. (2020).”
Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers (2020, methodology): “For the dynamic strategy, a single global pruning threshold (Molchanov et al. (2019); Lin et al. (2018)) or layer-wise greedy algorithms (Luo et al. (2017); He et al. (2017)) are applied to achieve different layer-wise sparsity.”
Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training (2021, methodology): “The criteria used for pruning includes but are not limited to magnitude (Mozer & Smolensky, 1989; Han et al., 2016; Guo et al., 2016), Hessian (LeCun et al., 1990; Hassibi & Stork, 1993), mutual information (Dai et al., 2018a), Taylor expansion (Molchanov et al., 2016; 2019).”
A Low Effort Approach to Structured CNN Design Using PCA (2018, background): “Methods like [6], [26] and [29] that have only loop1 require N = 4 number”