Optimal Brain Surgeon and general network pruning

By Babak Hassibi, David G. Stork, G. Wolff. 1993

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “Pruning methods date back to at least the 1990s with the works of (LeCun et al., 1990; Hassibi et al., 1993).”
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022, background): “Finally, Optimal Brain Quantization (OBQ) (Frantar et al., 2022) generalizes the classic Optimal Brain Surgeon (OBS) second-order weight pruning framework (Hassibi et al., 1993; Singh & Alistarh, 2020; Frantar et al., 2021) to apply to quantization.”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, methodology): “ing. LeCun et al. ﬁrst explored pruning as a way to reduce the size of neural networks [7]; they pruned based on the second derivative of the loss function with respect to each weight. Hassibi et al. [5] build on this approach. More recently, Han et al. [4, 3, 2] showed that these techniques could b”
What is the State of Neural Network Pruning? (2020, methodology): “…Methods There was already a rich body of work on neural network pruning by the mid 1990s (see, e.g., Reed’s survey (Reed, 1993)), which has been almost completely ignored except for Lecun’s Optimal Brain Damage (LeCun et al., 1990) and Hassibi’s Optimal Brain Surgeon (Hassibi et al., 1993).”
BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, methodology): “…as their modern variations; 3) close relationship between computation of Hessian and Network Compression using Hessian metric (LeCun et al., 1990; Hassibi et al., 1993); 4) acceleration effect to training convergence by secondorder optimization algorithm (Botev et al., 2017) to which it is…”
Efficient Structured Pruning and Architecture Searching for Group Convolution (2018, background): “, L1, L2 norms [27, 5, 20, 26], firstorder [29, 3] or second-order [19, 7, 2] gradients, average percentage of zero [26, 11], singular values [32, 28].”
CondenseNet: An Efficient DenseNet Using Learned Group Convolutions (2017, background): “CondenseNets are closely related to approaches that improve the inference efficiency of (convolutional) networks via weight pruning [11, 14, 27, 29, 32] and/or weight quantization [21, 36].”
Learning the Number of Neurons in Deep Networks (2016, neutral): “This trend was started by [LeCun et al., 1990, Hassibi et al., 1993] to cancel out individual parameters, and by [Mozer and Smolensky, 1988, Ji et al., 1990, Reed, 1993], and more recently [Liu et al., 2015], when it comes to removing entire neurons.”
Exploring Sparsity in Recurrent Neural Networks (2017, neutral): “Optimal Brain Surgeon is another work in the same vein that prunes weights using the inverse of a Hessian matrix with the additional advantage of no re-training after pruning (Hassibi et al., 1993).”
N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning (2017, methodology): “Additionally, the models identified by our policy are smaller and perform better than those reported by Knowledge Distillation methods that use hand designed student models such as FitNets (Romero et al. 2014) (#P: 2.5M, A: 91.61”
Data-Free Knowledge Distillation for Deep Neural Networks (2017, background): “Further work showed that not only is it a valid way to decrease the over-all memory proﬁle of a network, but that it is also a good way to deal with overﬁtting [7] and can help a network better generalize.”
Domain-Adaptive Deep Network Compression (2017, background): “This process can be conducted while training [2, 38, 14, 20], or by analyzing the inﬂuence of each parameter once the network has been trained [22].”
A Survey of Modern Deep Learning based Object Detection Models (2021, background): “Utilization of efﬁcient components and compression techniques like pruning ( [85], [86]), quantization ( [87], [88]), hashing [89], etc. have improved the efﬁciency of deep learning models.”
Block-Sparse Recurrent Neural Networks (2017, methodology): “LeCun et al. (1989) and Hassibi et al. (1993) both use Hessian-based approaches to prune weights below a certain threshold.”
EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis (2019, methodology): “By contrast, Optimal Brain Damage (OBD) (LeCun et al., 1990) and Optimal Brain Surgeon (OBS) (Hassibi et al., 1993) prune weights based on the Hessian of the loss function; the advantage is that both criteria reﬂect the sensitivity of the cost to the weight.”
The Lottery Ticket Hypothesis: Training Pruned Neural Networks (2018, methodology): “ing. LeCun et al. ﬁrst explored pruning as a way to reduce the size of neural networks [7]; they pruned based on the second derivative of the loss function with respect to each weight. Hassibi et al. [5] build on this approach. More recently, Han et al. [4, 3, 2] showed that these techniques could b”
Compact and Computationally Efficient Representation of Deep Neural Networks (2018, methodology): “For instance, some of the most popular techniques 27 include sparsification of the network [3, 8, 7, 13] or quantization of the weight elements [17, 11].”