NISP: Pruning Networks Using Neuron Importance Score Propagation

By Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I. Morariu, Xintong Han, Mingfei Gao, Ching-Yung Lin, Larry S. Davis. 2017

883 citations recorded by the source corpus.

To reduce the significant redundancy in deep Convolutional Neural Networks (CNNs), most existing methods prune neurons by only considering the statistics of an individual layer or two consecutive layers (e.g., prune one layer to minimize the reconstruction error of the next layer), ignoring the effect of error propagation in deep networks. In contrast, we argue that for a pruned network to retain its predictive power, it is essential to prune neurons in the entire neuron network jointly based on a unified goal: minimizing the reconstruction error of important responses in the "final response layer" (FRL), which is the second-to-last layer before classification. Specifically, we apply feature ranking techniques to measure the importance of each neuron in the FRL, formulate network pruning as a binary integer optimization problem, and derive a closed-form solution to it for pruning neurons in earlier layers. Based on our theoretical analysis, we propose the Neuron Importance Score Propagation (NISP) algorithm to propagate the importance scores of final responses to every neuron in the network. The CNN is pruned by removing neurons with least importance, and it is then fine-tuned to recover its predictive power. NISP is evaluated on several datasets with multiple CNN models and demonstrated to achieve significant acceleration and compression with negligible accuracy loss.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

What is the State of Neural Network Pruning? (2020, methodology): “…iterations constant, many methods produce pruned models that outperform re-training from scratch with the same sparsity pattern (Zhang et al., 2015; Yu et al., 2018; Louizos et al., 2017; He et al., 2017; Luo et al., 2017; Frankle & Carbin, 2019) (at least with a large enough amount of pruning…”
Similarity-Preserving Knowledge Distillation (2019, methodology): “Weight pruning methods [11, 20, 22, 24, 33, 35, 39] remove unimportant weights from the network, sparsifying the network connectivity structure.”
Efficient Structured Pruning and Architecture Searching for Group Convolution (2018, background): “[47] considers the importance as a global score.”
Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generation (2021, background): “There are many effective strategies for improving the parameter efficiency of trained networks: for example, pruning algorithms learn to remove redundant or less important parameters [12,40], and quantization algorithms learn to represent parameters using fewer bits while maintaining task performanc”
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, neutral): “Many techniques have been proposed to successfully compress neural networks including [17, 28, 32, 35, 44, 45, 69, 72].”
Overcoming Catastrophic Forgetting by Neuron-level Plasticity Control (2019, methodology): “Although there are other methods in network pruning that claim to have better performance(Yu et al., 2018; Luo and Wu, 2017; Luo et al., 2017), we chose the Taylor criterion due to its computational efficiency.”
Rethinking the Value of Network Pruning (2018, neutral): “Therefore, this is usually believed, and reported to be superior to directly training a smaller network from scratch (Li et al., 2017; Luo et al., 2017; He et al., 2017b; Yu et al., 2018) – a commonly used baseline approach.”
M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning (2019, background): “[29] compress the weights according to their scores.”
Improved Knowledge Distillation via Teacher Assistant (2019, background): “There has been an interesting line of research that compresses a large network by reducing the connections based on weight magnitudes (Han, Mao, and Dally 2016; Li et al. 2016) or importance scores (Yu et al. 2018).”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Pruning methods can be broadly categorized into unstructured pruning [49, 85, 137, 141, 186, 249], and structured pruning [90, 105, 153, 162, 266, 267, 271].”
Extreme Network Compression via Filter Group Approximation (2018, background): “[39] proposed a Neuron Importance Score Propagation algorithm to propagate the importance scores to every weight.”
BlockDrop: Dynamic Inference Paths in Residual Networks (2017, background): “The need to deploy top-performing deep neural network models on mobile devices motivates techniques that can effectively reduce the storage and computational costs of such networks, including knowledge distillation [19, 40, 4], low-rank factorization [25, 47, 41], filter pruning [30, 36, 32, 57], qu”
Variational Convolutional Neural Network Pruning (2019, methodology): “[29, 47] transform network pruning into an optimization problem and perform channel selection in a layer-wise manner via greedy algorithms.”
DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures (2019, neutral): “62 × NISP-110 (Yu et al., 2018) N/A -0.18% 1 .”
Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection (2020, neutral): “Backward Elimination Many of the popular conventional network pruning methods are based on backward elimination of redundant neurons (e.g. Liu et al., 2017; Li et al., 2017; Yu et al., 2018), and fewer algorithms are based forward selection like our method (e.”
Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolation (2020, background): “A common approach to this problem is to prune weights and neurons that are not needed to maintain the networks performance [20,11,10,36,14,21,29,40].”
Choose Your Neuron: Incorporating Domain Knowledge through Neuron-Importance (2018, background): “been proposed [33,15] in various contexts; however, this simple gradient-based importance measure has some notable properties which we leverage.”