A Simple Weight Decay Can Improve Generalization

By Anders Krogh, John A. Hertz. 1991

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks (2018, methodology): “Weight decay [29] is a traditional approach to combat overtraining.”
Train faster, generalize better: Stability of stochastic gradient descent (2015, methodology): “Weight decay is a simple and effective method that often improves generalization [20].”
Revisiting Few-sample BERT Fine-tuning (2020, neutral): “1 Overview Pre-trained Weight Decay Weight decay (WD) is a common regularization technique [21].”
Visualizing the Loss Landscape of Neural Nets (2017, background): “However, this sharpness balance can be flipped simply by turning on weight decay [25].”
Deep learning in neural networks: An overview (2014, background): “For example, weight decay(Hanson and Pratt, 1989; Weigend et al., 1991; Krogh and Hertz, 1992) encourages near-zero weights, by penalizi g large weights.”
A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, background): “To avoid this issue, a regularization term can be added to the ob-jective function which causes a weight decay [61].”
Style transfer-based image synthesis as an efficient regularization technique in deep learning (2019, background): “various parts of the model (dropout [3]), to limit the growth of the weights (weight decay [4]), to select the right moment to”
Early Stopping without a Validation Set (2017, methodology): “Finally, a widely-used provision against overfitting is to add regularization terms to the objective function that penalize the parameter vector w, typically measured by the l1 or l2 norm [8].”
Residual Continual Learning (2020, methodology): “Since a weight decay loss is widely used as a regularization term to improve generalization of neural networks (Krogh and Hertz 1992), we will also use a weight decay loss for not only Wt but also α.”
Learning the Number of Neurons in Deep Networks (2016, methodology): “While group sparsity lets us effectively remove some of the neurons, exploiting standard regularizers on the individual parameters has proven effective in the past for generalization purpose [Bartlett, 1996, Krogh and Hertz, 1992, Theodoridis, 2015, Collins and Kohli, 2014].”
Shakedrop Regularization for Deep Residual Learning (2018, methodology): “Widely used regularization methods include data augmentation [18], stochastic gradient descent (SGD) [37], weight decay [20], batch normalization (BN) [15], label smoothing [28], adversarial training [6], mixup [29]–[31], [38], and dropout [26], [32].”
Memory Bounded Deep Convolutional Networks (2014, methodology): “However, these and other approaches [12] proposed in the literature fail to address the issue of high memory cost, which is particularly problematic in models that rely on multiple fully connected layers.”
Spectral Norm Regularization for Improving the Generalizability of Deep Learning (2017, background): “It is known that, in some ideal cases, weight decay improves generalizability by preventing neural networks from ﬁtting noises [21].”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, background): “That is, large absolute weight values are considered unlikely and cost more bits than small ones [91, 294, 135, 97].”
Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes (2018, methodology): “Weight decay [16] is a commonly-used strategy to achieve better model generalization by adding a regularization term to the loss function as follows:”
Three Mechanisms of Weight Decay Regularization (2018, methodology): “For MNIST, we used simple fully-connected networks with different depth and width.”
Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks (2017, neutral): “We augment the data using the same procedure as CIFAR-10/100 mentioned above.”
Biological Sequence Modeling with Convolutional Kernel Networks (2017, neutral): “In particular, correctly regularizing neural networks to avoid overfitting is still an open issue and involves various heuristics such as dropout [40] and weight decay [13, 20].”
ReSeg: A Recurrent Neural Network-Based Model for Semantic Segmentation (2015, methodology): “In all our experiments, we used L2 regularization [25], also known as weight decay, set to 0.”