Comparing Biases for Minimal Network Construction with Back-Propagation

By Stephen Jose Hanson, L. Pratt. 1988

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Fixing Weight Decay Regularization in Adam (2017, methodology): “In the weight decay described by Hanson & Pratt (1988), the weights x decay exponentially as where w t deﬁnes the rate of the weight decay at time-step t and ∇ f t ( x t ) is the t -th batch gradient multiplied by a learning rate α t .”
Learning both Weights and Connections for Efficient Neural Network (2015, background): “An early approach to pruning was biased weight decay [17].”
Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding (2015, background): “In early work, network pruning proved to be a valid way to reduce the network complexity and over-ﬁtting (LeCun et al., 1989; Hanson & Pratt, 1989; Hassibi et al., 1993; Str¨om, 1997).”
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks (2016, methodology): “Weight decay [17] was an early method for pruning a network.”
Deep learning in neural networks: An overview (2014, background): “For example, weight decay (Hanson and Pratt, 1989; Weigend et al., 1991; Krogh and Hertz, 1992) prefers small weights and can be derived from Gaussian or Laplace weight priors (Hinton and van Camp, 1993); see also (Murray and Edwards, 1993).”
Recent advances in convolutional neural networks (2015, background): “It reduces the number of parameters and operations in CNNs by permanently dropping less important connections [155], which enables smaller networks to inherit knowledge from the large predecessor networks and maintains comparable of performance.”
Using Videos to Evaluate Image Model Robustness (2019, background): “We examine the robustness of 12 model architectures: VGG16, VGG19 (Simonyan & Zisserman (2014)), ResNet-V1-50, ResNet-V1-101, ResNet-V1-152 (He et al. (2016)), MobileNet-V1 (Howard et al. (2017)), NASNet-Mobile (Zoph et al. (2018)), Inception-V1 (Szegedy et al. (2015)), Inception-V2, Inception-V3 (S”
Incremental Learning Through Deep Adaptation (2017, methodology): “pactly [26], [27], gain speedups [28] or avoid over-fitting [29], using combinations of coding, quantization, pruning and tensor decomposition.”
EIE: Efficient Inference Engine on Compressed Deep Neural Network (2016, background): “In early work, network pruning proved to be a promising approach to reducing the network complexity and over-ﬁtting [34]–[36].”
Exploring Sparsity in Recurrent Neural Networks (2017, neutral): “One method is to use several bias techniques to decay weights (Hanson & Pratt, 1989).”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, background): “That is, large absolute weight values are considered unlikely and cost more bits than small ones [91, 294, 135, 97].”
Importance Estimation for Neural Network Pruning (2019, methodology): “One of the first works that used regularization, by Hanson and Pratt [9], used weight decay along with other energy minimization functions to reduce the complexity of the neural network.”
Biological Sequence Modeling with Convolutional Kernel Networks (2017, neutral): “Correctly regularizing neural networks to avoid overﬁtting is another open issue and involves various heuristics such as dropout [Srivastava et al., 2014], weight decay [Hanson and Pratt, 1989], and early stopping.”
Going Deeper with Embedded FPGA Platform for Convolutional Neural Network (2016, background): “In early work, network pruning proved to be a valid way to reduce the network complexity and over-ﬁtting [17, 18, 19].”
Rethinking the Hyperparameters for Fine-tuning (2020, unknown): “L 2 regularization or weight decay is widely used for constraining the model capacity (Hanson & Pratt, 1989; Krogh & Hertz, 1992).”
"Learning-Compression" Algorithms for Neural Net Pruning (2018, background): “iw 2 i /(A+ w 2 i ) [16, 31] that encourage weights to be either large or small.”
Further Advantages of Data Augmentation on Convolutional Neural Networks (2018, methodology): “…to our knowledge, a systematic analysis of the implicit regularization effect of data augmentation on deep neural networks compared to the most popular regularization techniques, such as weight decay Hanson & Pratt (1989) and dropout Srivastava et al. (2014), which are typically used all together.”
What Matters for Adversarial Imitation Learning? (2021, methodology): “Apart from the above regularizers, we also run experiments with regularizers commonly used in Supervised Learning, namely dropout [10], the weight decay [1] variant from AdamW [33] as well as the entropy bonus of the discriminator output treated as a Bernoulli distribution.”
Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges (2018, methodology): “An early approach to pruning was the Biased Weight Decay [19],”