Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment

By Michael C. Mozer, Paul Smolensky. 1988

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

What is the State of Neural Network Pruning? (2020, methodology): “…with Han et al. (2015) reintroducing the magnitude-based pruning of Janowsky (1989), Lee et al. (2019b) reintroducing the saliency heuristic of Mozer & Smolensky (1989a), and He et al. (2018a) reintroducing the practice of “reviving” previously pruned weights described in Tresp et al. (1997).”
Dynamic Model Pruning with Feedback (2020, methodology): “For instance, one-shot pruning strategies ﬁnd a suitable sparsity mask by inspecting the weights of a pretrained network (Mozer & Smolensky, 1989; LeCun et al., 1990; Han et al., 2017).”
Deep learning in neural networks: An overview (2014, background): “…5.3, 5.11), input pruning (Moody, 1992; Refenes, Zapranis, & Francis, 1994), unit pruning (e.g., Ivakhnenko, 1968, 1971; Levin, Leen, & Moody, 1994; Mozer & Smolensky, 1989; White, 1989), weight pruning, e.g., optimal brain damage (LeCun, Denker, & Solla, 1990), and optimal brain surgeon (Hassibi…”
Learning the Number of Neurons in Deep Networks (2016, methodology): “This trend was started by [LeCun et al., 1990, Hassibi et al., 1993] to cancel out individual parameters, and by [Mozer and Smolensky, 1988, Ji et al., 1990, Reed, 1993], and more recently [Liu et al., 2015], when it comes to removing entire neurons.”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “As the history grows, to find an M with small bitsM + bitsH , step 4 uses sequential network construction: it regularly changes M ’s size by adding or pruning units and connections [111, 112, 8, 168, 59, 107, 304, 176, 141, 92, 143, 204, 53, 296, 106, 31, 57, 185, 283].”
Rigging the Lottery: Making All Tickets Winners (2019, background): “Mozer & Smolensky (1989), Le-Cun et al. (1990) and Hassibi & Stork (1993) are some early examples, but impractical for modern neural networks as they use information from the Hessian to prune a trained network.”
Sparse Networks from Scratch: Faster Training without Losing Performance (2019, methodology): “Earlier work focused on pruning via second-order derivatives (LeCun et al., 1989; Karnin, 1990; Hassibi and Stork, 1992) and heuristics which ensure efﬁcient training of networks after pruning (Chauvin, 1988; Mozer and Smolensky, 1988; Ishikawa, 1996).”
Importance Estimation for Neural Network Pruning (2019, methodology): “This belief was investigated as early as 1988 by Mozer [28] who proposed adding a gating function after each layer to be pruned.”
SNIP: Single-shot Network Pruning based on Connection Sensitivity (2018, methodology): “On the other hand, classical saliency criteria include the sensitivity of the loss with respect to the neurons (Mozer & Smolensky (1989)) or the weights (Karnin (1990)) and Hessian of the loss with respect to the weights (LeCun et al. (1990); Hassibi et al. (1993)).”
A hybrid neural network‐first principles approach to process modeling (1992, methodology): “…of freedom) may result in poor models, one approach has been to decrease the redundancy of the neural network model by developing algorithms that " prune " the weights that have no significant effect on the network's performance (McAvoy and Bhat, 1990; Karnin, 1990; Mozer and Smolensky, 1989).”
Evolving Artiicial Neural Networks (2007, background): “Research on constructive and destructive algorithms represents an e ort towards the automatic design of architectures [141, 142, 143, 144, 145, 146, 147, 148].”
Constructive algorithms for structure learning in feedforward neural networks for regression problems (1997, methodology): “3 Typically, approximation involves using up to the first [19], [20] or second [21], [22] term in the Taylor series expansion for the change in error.”
Extracting refined rules from knowledge-based neural networks (2004, result): “…equal the accuracy of MoFN on problems whose solution matches their bias towards sets of Conjunctive rules. third piece of evidence in support of the over fitting hypothesis comes from the work on pruning neural networks to eliminate superfluous parts (Mozer & Smolensky, 1988; Le Cun et al., 1989).”
DENFIS: dynamic evolving neural-fuzzy inference system and its application for time-series prediction (2002, unknown): “At the same time, several models that have elements of adaptive, online learning or structure and knowledge adaptation, have been developed that include connectionist models [1]–[4], [10]–[12], [16], [19], [21], [23], [25], [26], [30], [31], [33], [34], [45]–[49], [54], [57], [61], [64], [65], fuzzy”
Neural Pruning via Growing Regularization (2020, background): “Since its early debut Mozer & Smolensky (1989); Reed (1993), the central problem of neural network pruning has been (arguably) how to choose weights to discard, i.e., the weight importance scoring problem LeCun et al. (1990); Hassibi & Stork (1993); Molchanov et al. (2017b; 2019); Wang et al.…”
Neural networks for classification: a survey (2000, methodology): “Mozer and Smolensky [119] describe a node pruning method based on a saliency measure that is the difference of the error between when the node is removed and when the node is present.”
Data-Independent Structured Pruning of Neural Networks via Coresets (2020, background): “For example, if the algorithm is tuned to a specific data (as in [4], [26], [43], [56], and [57]), the compressed network would fail to approximate the output of the original network on less typical unseen examples.”