The Effects of Adding Noise During Backpropagation Training on a Generalization Performance

By Guozhong An. 1996

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Adaptive Computation Time for Recurrent Neural Networks (2016, background): “Secondly, neural networks have been successfully trained under a wide range of adversarial regularisation constraints, including sparse internal states [23], stochastically masked units [28] and randomly perturbed weights [1].”
Shake-Shake regularization (2017, background): “is work is to repeat this coefﬁcient update operation before each backward pass. This results in a stochastic blend of forward and backward ﬂows during training. Related to this idea are the works of An (1996) and Neelakantan et al. (2015). These authors showed that adding noise to the gradient duri”
Deep Learning with Limited Numerical Precision (2015, background): “Moreover, the addition of noise during training has been shown to improve the neural network’s performance (Murray & Edwards, 1994; Bishop, 1995; An, 1996; Audhkhasi et al., 2013).”
Deep learning in neural networks: An overview (2014, methodology): “Dropout is closely related to older, biologically plausibl e techniques of adding noise to neurons or synapses during training (e.g., Murray and Edwards, 1993; Schuster, 1992; Nadal and Parga, 1994; Jim et al., 1995; An, 1996), which in turn are closely related to finding pertur bation-resistant low”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, methodology): “There is classic work on adding noise to inputs for improved generalization (Matsuoka, 1992; An, 1996).”
Adding Gradient Noise Improves Learning for Very Deep Networks (2015, background): “Adding random noise to the weights, gradient, or the hidden units has been a known technique amongst neural network practitioners for many years (e.g., An (1996)).”
Stabilizing Training of Generative Adversarial Networks through Regularization (2017, background): “Instead, the underlying assumption we make is that optimizing the gradient-norm regularized objective F (P,Q; ) makes converge to ⇤ +O( ), for which we know that the Laplacian terms cancel [5, 2].”
Shake-Shake regularization of 3-branch residual networks (2017, background): “, 2016) (see appendix for details on similarities and differences) and to work which has shown that adding noise to activations or weights reduces overfit (An, 1996; Blundell et al., 2015; Neelakantan et al., 2015).”
Does Unsupervised Architecture Representation Learning Help Neural Architecture Search? (2020, methodology): “Using random noise injection mechanism has been proved to be effective on the regularization of neural networks [49, 50, 36].”
Towards Text Generation with Adversarially Learned Neural Outlines (2018, background): “The injection of noise into neural models has been explored in many contexts [2, 55, 40, 50, 41] and has been shown to have important implications in unsupervised representation learning.”
Model-Based Planning in Discrete Action Spaces (2017, background): “The work of [3] shows that adding independent noise to the weights of a neural network during training induces an additional term on the loss function.”
Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion (2010, background): “…of training a neural network with noisy input (Scalettar and Zee, 1988; von Lehman et al., 1988)—or training with jitter as it is sometimes called—has been proposed to enhance generalization performance for supervised learning tasks (Sietsma and Dow, 1991; Holmstrm and Koistinen, 1992; An, 1996).”
Noise as a Resource for Learning in Knowledge Distillation (2020, methodology): “Noise has been used as a common regularization technique to improve the generalization performance of overparameterized deep neural networks (DNNs) by adding it to the input data, the weights, or the hidden units [1, 5, 17, 50, 51, 56].”
Bagging, Boosting and the Random Subspace Method for Linear Classifiers (2002, background): “One way is to stabilise the decision of a weak classifier (as weak classifiers are often unstable) by regularisation [2] or noise injection [3].”
Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting (2020, background): “Related Work One related line of research is injecting weight noise into deep networks during training (An, 1996; Neelakantan et al., 2015; Zhou et al., 2019).”
Behavior recognition for humanoid robots using long short-term memory (2016, methodology): “The noise robustness accuracy score can be further improved by injecting noise into the training data, a well-known technique to achieve better performance in neural networks.(45)”