A simple procedure for pruning back-propagation trained neural networks

By E. Karnin. 1990

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (2021, background): “more parameters than needed in inference (Buciluǎ et al., 2006; Hinton et al., 2015; Urban et al., 2017; Karnin, 1990; Reed, 1993; Augasta and Kathirvalavakumar, 2013; Liu et al., 2014; Han et al., 2015; Molchanov et al., 2017).”
What is the State of Neural Network Pruning? (2020, methodology): “Pruning has been used since the late 1980s (Janowsky, 1989; Mozer & Smolensky, 1989a;b; Karnin, 1990), but has seen an explosion of interest in the past decade thanks to the rise of deep neural networks.”
Sparse Networks from Scratch: Faster Training without Losing Performance (2019, methodology): “Earlier work focused on pruning via second-order derivatives (LeCun et al., 1989; Karnin, 1990; Hassibi and Stork, 1992) and heuristics which ensure efﬁcient training of networks after pruning (Chauvin, 1988; Mozer and Smolensky, 1988; Ishikawa, 1996).”
A hybrid approach of neural network and memory-based learning to data mining (2000, background): “To measure the sensitivity, feeding zero input to the network is suggested instead of actually removing the weights and directly calculating the sensitivity [ 17 ].”
Neural networks for the prediction and forecasting of water resource variables: a review of modelling issues and applications (2000, methodology): “Pruning algorithms have been proposed by many authors, including Sietsma and Dow (1991); Karnin (1990); Chung and Lee (1992); Setiono (1997).”
A new strategy for adaptively constructing multilayer feedforward neural networks (2003, background): “Karnin [14] de:ned a sensitivity of the error function being minimized with respect to the removal of each connection and pruned the connections that have low sensitivities.”
Neural networks in geophysical applications (2000, methodology): “Other pruning techniques can be found in Karnin (1990), Pellilo and Fanelli (1993), and Cottrell et al. (1995).”
Nonlinear principal component analysis using autoassociative neural networks (1991, background): “Karnin (1990) estimates the sensitivity of the error to the exclusion of each connection and eliminates the connections with lowest sensitivities.”
Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning (2024, background): “Weight utility can be defined as the change in loss when setting the weight to zero, essentially removing its connection (Mozer & Smolensky 1988, Karnin 1990).”
Pruning neural networks without any data by iteratively conserving synaptic flow (2020, background): “Popular scoring metrics include weight magnitudes [4, 6], its generalization to multi-layers [25], first- [1, 26, 27, 28] and second-order [2, 3, 28] Taylor coefficients of the training loss with respect to the parameters, and more sophisticated variants [29, 30, 31].”
A hybrid neural network‐first principles approach to process modeling (1992, methodology): “…of freedom) may result in poor models, one approach has been to decrease the redundancy of the neural network model by developing algorithms that " prune " the weights that have no significant effect on the network's performance (McAvoy and Bhat, 1990; Karnin, 1990; Mozer and Smolensky, 1989).”