Second Order Derivatives for Network Pruning: Optimal Brain Surgeon

By Babak Hassibi, David G. Stork. 1992

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

The State of Sparsity in Deep Neural Networks (2019, methodology): “3 https://bit.ly/2ExE8Yj Some of the earliest techniques for sparsifying neural networks make use of second-order approximation of the loss surface to avoid damaging model quality (LeCun et al., 1989; Hassibi & Stork, 1992).”
Learning both Weights and Connections for Efficient Neural Network (2015, background): “After an initial training phase, we remove all connections whose weight is lower than a threshold.”
Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding (2015, background): “In early work, network pruning proved to be a valid way to reduce the network complexity and over-ﬁtting (LeCun et al., 1989; Hanson & Pratt, 1989; Hassibi et al., 1993; Str¨om, 1997).”
Are Sixteen Heads Really Better than One? (2019, background): “There is a rich literature on pruning trained neural networks, going back to LeCun et al. (1990) and Hassibi and Stork (1993) in the early 90s and reinvigorated after the advent of deep learning, with two orthogonal approaches: ﬁne-grained “weight-by-weight” pruning (Han et al., 2015) and…”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “vices). Although pruning is central to our experiments, we aim to gain insight into why training needs the overparameterized networks that make pruning necessary. LeCun et al. [31] and Hassibi et al. [18] ﬁrst explored pruning based on second derivatives. More recently, Han et al. [17, 15, 14] showe”
Pruning Filters for Efficient ConvNets (2016, background): “The early work by Le Cun et al. (1989) introduces Optimal Brain Damage, which prunes weights with a theoretically justified saliency measure.”
Weight Agnostic Neural Networks (2019, neutral): “Network Pruning By removing connections with small weight values from a trained neural network, pruning approaches [35, 39, 41, 58, 62, 66, 67, 69, 75] can produce sparse networks that keep only a small fraction of the connections, while maintaining similar performance on image classiﬁcation tasks…”
Deep learning in neural networks: An overview (2014, neutral): “…pruning (Moody, 1992; Refenes, Zapranis, & Francis, 1994), unit pruning (e.g., Ivakhnenko, 1968, 1971; Levin, Leen, & Moody, 1994; Mozer & Smolensky, 1989; White, 1989), weight pruning, e.g., optimal brain damage (LeCun, Denker, & Solla, 1990), and optimal brain surgeon (Hassibi & Stork, 1993).”
Network Pruning via Transformable Architecture Search (2019, methodology): “Unstructured pruning methods [26, 7, 5, 9] usually enforce the convolutional weights [26, 11] or feature maps [5, 7] to be sparse.”
You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization (2018, background): “The early works of pruning focus on removing unimportant connections (LeCun et al. (1990); Hassibi & Stork (1993); Han et al. (2015); Guo et al. (2016)).”
MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks (2017, background): “Optimal Brain Damage [17, 9] proposed pruning the weights of a fully trained DNN based on their contribution to the objective function.”
Data-free Parameter Pruning for Deep Neural Networks (2015, background): “A few years later, Hassibi et al . came up with Optimal Brain Surgeon (OBS) [9], which was shown to perform much better than OBD, but was much more computationally intensive.”
EIE: Efficient Inference Engine on Compressed Deep Neural Network (2016, background): “In early work, network pruning proved to be a promising approach to reducing the network complexity and over-ﬁtting [34]–[36].”