What is the State of Neural Network Pruning?

By Davis W. Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, John V. Guttag. 2020

1,270 citations recorded by the source corpus.

Neural network pruning---the task of reducing the size of a network by removing parameters---has been the subject of a great deal of work in recent years. We provide a meta-analysis of the literature, including an overview of approaches to pruning and consistent findings in the literature. After aggregating results across 81 papers and pruning hundreds of models in controlled conditions, our clearest finding is that the community suffers from a lack of standardized benchmarks and metrics. This deficiency is substantial enough that it is hard to compare pruning techniques to one another or determine how much progress the field has made over the past three decades. To address this situation, we identify issues with current practices, suggest concrete remedies, and introduce ShrinkBench, an open-source framework to facilitate standardized evaluations of pruning methods. We use ShrinkBench to compare various pruning techniques and show that its comprehensive evaluation can prevent common pitfalls when comparing pruning methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “In this work, we follow the network pruning approach, where the redundant units (weights, neurons or ﬁlters/channels) of a pre-trained NN are removed; see [Kuzmin et al., 2019, Blalock et al., 2020, Hoeﬂer et al., 2021] for recent surveys.”
On the Predictability of Pruning Across Scales (2020, methodology): “, 2015), and produces stateof-the-art tradeoffs between density and error (Gale et al., 2019; Blalock et al., 2020; Renda et al., 2020).”
Initialization and Regularization of Factorized Neural Layers (2021, background): “guessing “lottery tickets” (Frankle & Carbin, 2019)—with a prevailing trend of viewing low-rank methods as uncompetitive for compression (Blalock et al., 2020; Zhang et al., 2020; Idelbayev & Carreira-Perpiñán, 2020; Su et al., 2020).”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Training and inference with high levels of pruning/sparsity, while maintaining state-of-the-art performance, has remained an open problem [16].”
The Lottery Ticket Hypothesis for Pre-trained BERT Networks (2020, methodology): “In the past decade, computer vision has been the most common application area for neural network pruning research [40].”
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints (2022, methodology): “Pruning is typically employed as a post-training architecture search to construct smaller and faster models from larger models (LeCun et al., 1989; Gale et al., 2019; Blalock et al., 2020).”
Knowledge distillation: A good teacher is patient and consistent (2021, background): “Balanced and fair discussion of this topic goes beyond the scope of this paper, so we refer interested reader to recent overviews as a starting point [3, 44].”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, background): “Though pruning sometimes introduce little improvement of the accuracy as they remove some of the local minima [12], the techniques can not perform better than switch to a better architecture [24], e.”
The Computational Limits of Deep Learning (2020, methodology): “What is the State of Neural Network Pruning? (2020) [14] ✘ ✘ ✘ ✘”
Recursive Least Squares for Training and Pruning Convolutional Neural Networks (2022, methodology): “In detail, network pruning algorithms prune redundant and uninformative channels or zero out unimportant weights [5, 6].”
A TinyML Platform for On-Device Continual Learning With Quantized Latent Replays (2021, methodology): “The inference task can be strongly optimized by quantizing [4] or pruning [5] the trained model.”