Optimal Brain Damage

By Yann LeCun, John S. Denker, Sara A. Solla. 1989

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “…with reduced precision support and (2) reduce overall memory footprint (Han et al., 2016; Dong et al., 2019 Effect of MT Model Size weights to zero to (1) remove operations and (2) reduce the memory footprint when models are stored in sparse matrix formats (LeCun et al., 1990; Han et al., 2015).”
Sequence-Level Knowledge Distillation (2016, methodology): “…et al., 1990; Has-sibi and Stork, 1993; He et al., 2014; Han et al., 2016; Mariet and Sra, 2016), zero-out weights based on an importance criterion: LeCun et al. (1990) use (a diagonal approximation to) the Hessian to identify weights whose removal minimally impact the objective function, while…”
The State of Sparsity in Deep Neural Networks (2019, methodology): “3 https://bit.ly/2ExE8Yj Some of the earliest techniques for sparsifying neural networks make use of second-order approximation of the loss surface to avoid damaging model quality (LeCun et al., 1989; Hassibi & Stork, 1992).”
Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters (2018, background): “An early approach is Optimal Brain Damage (LeCun et al., 1990) which employs the Hessian of the network weights in order to determine whether weights can be pruned without signiﬁcantly impacting training performance.”
FastText.zip: Compressing text classification models (2016, positive): “Denil et al. (2013) show that such classiﬁcation models are easily compressed because they are over-parametrized, which concurs with early observations by LeCun et al. (1990).”
Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (2022, methodology): “Popular non-magnitude based techniques include [26, 17, 34].”
Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “Pruning methods date back to at least the 1990s with the works of (LeCun et al., 1990; Hassibi et al., 1993).”
On the Predictability of Pruning Across Scales (2020, background): “For decades, neural network pruning—eliminating unwanted parts of a network—has been a popular approach for reducing network sizes or the computational demands of inference (LeCun et al., 1990; Reed, 1993; Han et al., 2015).”
Initialization and Regularization of Factorized Neural Layers (2021, methodology): “Best of (LeCun et al., 1990; Zeng & Urtasun, 2019), obtained from Wang et al.”
Predicting Parameters in Deep Learning (2013, methodology): “An early approach is the technique of “Optimal Brain Damage” [18] which uses approximate second derivative information to remove parameters from an already trained network.”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Model compression has been explored from diﬀerent aspects [25, 37, 39, 34, 43, 20, 24, 50, 18, 74, 40, 26, 55, 59, 28, 60, 68, 33, 14, 38, 31].”
Reducing Transformer Depth on Demand with Structured Dropout (2019, background): “As opposed to weight pruning (LeCun et al., 1990), structured pruning removes coherent groups of weights to preserve the original structure of the network.”
The Right Tool for the Job: Matching Model and Instance Complexities (2020, background): “Pruning (LeCun et al., 1990) removes some of the weights in the network, resulting in a smaller, potentially faster network.”
Learning both Weights and Connections for Efficient Neural Network (2015, methodology): “To achieve this goal, we present a method to prune network connections in a manner that preserves the original accuracy.”
Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding (2015, background): “In early work, network pruning proved to be a valid way to reduce the network complexity and over-ﬁtting (LeCun et al., 1989; Hanson & Pratt, 1989; Hassibi et al., 1993; Str¨om, 1997).”
Are Sixteen Heads Really Better than One? (2019, background): “There is a rich literature on pruning trained neural networks, going back to LeCun et al. (1990) and Hassibi and Stork (1993) in the early 90s and reinvigorated after the advent of deep learning, with two orthogonal approaches: ﬁne-grained “weight-by-weight” pruning (Han et al., 2015) and…”
MobileNetV2: Inverted Residuals and Linear Bottlenecks (2018, methodology): “methods of network pruning [12, 13, 14, 15, 16, 17] and connectivity learning [18, 19].”
Progressive Neural Networks (2016, methodology): “The actor-mimic approach [14] applied these principles to reinforcement learning, by fine-tuning a DQN multi-task network on new Atari games and showing that some responded with faster learning, while others did not.”
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT (2019, methodology): “Efforts in this area could be broadly categorized as follows: (i) new architectures that are compact by design [11, 13]; (ii) automated neural architecture search (NAS) with reward function set as latency or model size [32, 35]; (iii) pruning based methods to reduce model size of existing architectu”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “Techniques for eliminating unnecessary weights from neural networks (pruning) (LeCun et al., 1990; Hassibi & Stork, 1993; Han et al., 2015; Li et al., 2016) can reduce parameter-counts by more than 90% while maintaining accuracy.”