Pruning algorithms-a survey

By Russell Reed. 1993

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

On the Predictability of Pruning Across Scales (2020, background): “For decades, neural network pruning—eliminating unwanted parts of a network—has been a popular approach for reducing network sizes or the computational demands of inference (LeCun et al., 1990; Reed, 1993; Han et al., 2015).”
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (2021, background): “more parameters than needed in inference (Buciluǎ et al., 2006; Hinton et al., 2015; Urban et al., 2017; Karnin, 1990; Reed, 1993; Augasta and Kathirvalavakumar, 2013; Liu et al., 2014; Han et al., 2015; Molchanov et al., 2017).”
The Graph Neural Network Model (2009, neutral): “Thus, second-order learning algorithms [62], pruning [63], and growing learning algorithms [64]–[66] designed for static networks cannot be directly applied to GNNs.”
What is the State of Neural Network Pruning? (2020, background): “Ignoring Pre-2010s Methods There was already a rich body of work on neural network pruning by the mid 1990s (see, e.g., Reed’s survey (Reed, 1993)), which has been almost completely ignored except for Lecun’s Optimal Brain Damage (LeCun et al., 1990) and Hassibi’s Optimal Brain Surgeon (Hassibi et…”
Early Stopping without a Validation Set (2017, methodology): “Finally, a widely-used provision against overfitting is to add regularization terms to the objective function that penalize the parameter vector w, typically measured by the l1 or l2 norm [9].”
Learning the Number of Neurons in Deep Networks (2016, methodology): “This trend was started by [LeCun et al., 1990, Hassibi et al., 1993] to cancel out individual parameters, and by [Mozer and Smolensky, 1988, Ji et al., 1990, Reed, 1993], and more recently [Liu et al., 2015], when it comes to removing entire neurons.”
Memory Bounded Deep Convolutional Networks (2014, methodology): “These were grouped into “pruning” methods [18], and include both regularization penalties and techniques for determining the importance of a given weight to the network’s accuracy.”
Pruning Convolutional Neural Networks for Resource Efficient Transfer Learning (2016, methodology): “Optimal Brain Damage [9] and Optimal Brain Surgeon [4] leverage a second-order Taylor expansion to select parameters for deletion, using pruning as regularization to improve training and generalization.”
Linear Mode Connectivity and the Lottery Ticket Hypothesis (2019, background): “We have long known that it is possible to prune neural networks after training, often removing 90% of connections or more with no reduction in accuracy after small amount of additional training (e.g., LeCun et al., 1990; Reed, 1993; Han et al., 2015; Gale et al., 2019; He et al., 2018).”
Designing Energy-Efficient Convolutional Neural Networks Using Energy-Aware Pruning (2016, background): “ods evaluate whether to prune each weight independently and do not account for correlation between weights [11].”
SNIP: Single-shot Network Pruning based on Connection Sensitivity (2018, background): “Essentially, early works in network pruning can be categorized into two groups (Reed (1993)): 1) those that utilize sparsity enforcing penalties; and 2) methods that prune the network based on some saliency criterion.”
Structured Pruning of Deep Convolutional Neural Networks (2015, methodology): “These works [9]-[12] utilize unstructured sparsity in feed forward neural networks.”
Compression-aware Training of Deep Networks (2017, background): “In this context, early works have proposed to analyze the saliency of individual parameters [LeCun et al., 1990, Hassibi et al., 1993] or units [Mozer and Smolensky, 1988, Ji et al., 1990, Reed, 1993, Liu et al., 2015], so as to measure their impact on the output.”
Review of surrogate modeling in water resources (2012, methodology): “…have been developed to determine appropriate ANN structures for a given problem, including the methods based on the growing or pruning strategies [Reed, 1993] such as the methods presented in the work of Teoh et al. [2006] and Xu et al. [2006], methods based on the network geometrical 9 of…”
A New Adaptive Merging and Growing Algorithm for Designing Artificial Neural Networks (2009, background): “Most pruning algorithms start the removal of irrelevant parameters (connections and/or hidden neurons) after training an oversized ANN to a minimum error (see the review paper [3]).”
Iteratively Pruned Deep Learning Ensembles for COVID-19 Detection in Chest X-rays (2020, methodology): “Reed [16] performed a neural model pruning to decrease computational complexity.”
Feature selection with neural networks (2002, background): “Feature selection with neural nets can be thought of as a special case of architecture pruning (Reed, 1993), where input features are pruned, rather than hidden neurons or weights.”
Comparing Rewinding and Fine-tuning in Neural Network Pruning (2020, methodology): “Pruning is a set of techniques for removing weights, ﬁlters, neurons, or other structures from neural networks (e.g., Le Cun et al., 1990; Reed, 1993; Han et al., 2015; Li et al., 2017; Liu et al., 2019).”