Pruning Convolutional Neural Networks for Resource Efficient Inference

By Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, Jan Kautz. 2016

2,288 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “• A CT G RAD [Molchanov et al., 2017]: prunes neurons/channels with the lowest (activations ⇥ gradients), averaged over the training data, with layerwise ` 2 -normalization.”
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (2021, methodology): “more parameters than needed in inference (Buciluǎ et al., 2006; Hinton et al., 2015; Urban et al., 2017; Karnin, 1990; Reed, 1993; Augasta and Kathirvalavakumar, 2013; Liu et al., 2014; Han et al., 2015; Molchanov et al., 2017).”
Green AI (2019, methodology): “FPO has been used in the past to quantify the energy footprint of a model [26, 42, 12, 41], but is not widely adopted in AI.”
Are Sixteen Heads Really Better than One? (2019, methodology): “4 As recommended by Molchanov et al. (2017) we normalize the importance scores by layer (using the (cid:96) 2 norm).”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “…2016), redundancy (Mariet & Sra, 2016; Srinivas & Babu, 2015a), per-layer second derivatives (Dong et al., 2017), and energy/computation efficiency (Yang et al., 2017) (e.g., pruning convolutional filters (Li et al., 2016; Molchanov et al., 2016; Luo et al., 2017) or channels (He et al., 2017)).”
What is the State of Neural Network Pruning? (2020, background): “, (Molchanov et al., 2016) vs (Wang & Cheng, 2016)), though most often papers entirely omit their formula for computing FLOPs.”
Weight Agnostic Neural Networks (2019, neutral): “Network Pruning By removing connections with small weight values from a trained neural network, pruning approaches [35, 39, 41, 58, 62, 66, 67, 69, 75] can produce sparse networks that keep only a small fraction of the connections, while maintaining similar performance on image classiﬁcation tasks…”
Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks (2018, methodology): “We note that pruning has been proposed in prior work for non-security reasons, specifically, to reduce the computational expense of evaluating a DNN [19,48,25,4,33].”
BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, methodology): “The learned structure is 5− 10− 65− 25 with 1.00% test error and 0.57 FLOPS.”
Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks (2018, methodology): “…model compression methods, structural pruning has been widely adopted to approach the resource-constrained deep neural network optimization problem (Molchanov et al., 2017; He et al., 2017; Luo et al., 2017; He et al., 2018b; Gordon et al., 2018) since it provides solution that works directly on…”
Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation (2020, methodology): “In the parameter analyzing experiments, we erase parameters according to their importance which is evaluated by the Taylor expansion-based method (Molchanov et al., 2017) .”
Overcoming Catastrophic Forgetting by Neuron-level Plasticity Control (2019, methodology): “The importance value of a convolution filter is computed by averaging the importance values of the neurons on the corresponding feature map before computing its absolute value, following the original paper(Molchanov et al. 2016).”
Uncertainty-guided Continual Learning with Bayesian Neural Networks (2019, methodology): “In this section, we introduce a variant of our method, UCB-P , which is related to recent efforts in weight pruning in the context of reducing inference computation and network compression (Liu et al., 2017; Molchanov et al., 2016).”
PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning (2017, methodology): “The method of [20] ranks all filters in a network based on their importance to the current dataset, as measured by a metric related to the Taylor expansion of the loss function.”
Rethinking the Value of Network Pruning (2018, methodology): “As a remedy, network pruning (LeCun et al., 1990; Hassibi & Stork, 1993; Han et al., 2015; Molchanov et al., 2016; Li et al., 2017) has been identiﬁed as an effective technique to improve the efﬁciency of deep networks for applications with limited computational budget.”
M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning (2019, methodology): “Other methods [15, 22, 29] explore compression for fast inference.”
N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning (2017, methodology): “We perform pruning based on Molchanov et al. (2016), which removes filters using a greedy criteria based approach and then finetunes the network.”