Adding Gradient Noise Improves Learning for Very Deep Networks

By Arvind Neelakantan, Luke Vilnis, Quoc V. Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, James Martens. 2015

597 citations recorded by the source corpus.

Deep feedforward and recurrent networks have achieved impressive results in many perception and language processing applications. Recently, more complex architectures such as Neural Turing Machines and Memory Networks have been proposed for tasks including question answering and general computation, creating a new set of optimization challenges. In this paper, we explore the low-overhead and easy-to-implement optimization technique of adding annealed Gaussian noise to the gradient, which we find surprisingly effective when training these very deep architectures. Unlike classical weight noise, gradient noise injection is complementary to advanced stochastic optimization algorithms such as Adam and AdaGrad. The technique not only helps to avoid overfitting, but also can result in lower training loss. We see consistent improvements in performance across an array of complex models, including state-of-the-art deep networks for question answering and algorithm learning. We observe that this optimization strategy allows a fully-connected 20-layer deep network to escape a bad initialization with standard stochastic gradient descent. We encourage further application of this technique to additional modern neural architectures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets (2022, background): “…data efﬁciency curves on a particular dataset S 5 for a variety of interventions: full-batch gradient descent, stochastic gradient descent, large or small learning rates, residual dropout Srivastava et al. (2014), weight decay Loshchilov & Hutter (2017) and gradient noise Neelakantan et al. (2015).”
Do Transformer Modifications Transfer Across Implementations and Applications? (2021, background): “, 2012; Kingma & Ba, 2014; Shazeer & Stern, 2018b), regularization schemes (Srivastava et al., 2014; Zhang et al., 2017; Neelakantan et al., 2015), and model architectures (He et al.”
Neural Programmer: Inducing Latent Programs with Gradient Descent (2015, background): “We find that training the model is difficult, but it can be greatly improved by injecting random Gaussian noise to the gradient (Welling & Teh, 2011; Neelakantan et al., 2016) which enhances the generalization ability of the Neural Programmer.”
Shake-Shake regularization (2017, neutral): “Related to this idea are the works of An (1996) and Neelakantan et al. (2015).”
Neural Random Access Machines (2015, neutral): “The details can be found in Neelakantan et al. (2015).”
Differential Privacy Has Disparate Impact on Model Accuracy (2019, background): “Clipping and noise are (separately) standard regularization techniques [30, 31], but their combination in DP-SGD disproportionately impacts underrepresented classes.”
A Little Is Enough: Circumventing Defenses For Distributed Learning (2019, background): “This last assumption is advocated by the fact that SGD better converges with a little random noise [21, 25, 13].”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “It has also been argued that the poor performance of GD in non-convex problems can be improved by adding some isotropic white noise to the gradients, which helps it to escape poor local minima and avoid over-fitting (Neelakantan et al., 2015).”
Neural Optimizer Search with Reinforcement Learning (2017, neutral): “• annealed noise: t ∼ N(0, 1/(1 + t)) inspired by Neelakantan et al. (2016).”
QSGD: Communication-Optimal Stochastic Gradient Descent, with Applications to Training Neural Networks (2016, positive): “These results are consistent with recent work [27] noting beneﬁts of added noise in training deep networks.”
End-to-End Incremental Learning (2018, positive): “We apply L(2)-regularization and random noise [21] (with parameters η = 0.”
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning (2017, background): “Although the bound is also stronger, Gaussian noise encourages active exploration of parameter space and improves accuracy as was empirically studied in [34].”
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients (2016, background): “The credit may belong to the regularization effect introduced by the noise in gradients, similar to previous works in adding noise to gradients (Neelakantan et al., 2015).”
Neural Networks with Few Multiplications (2015, background): “Similarly, Published as a conference paper at ICLR 2016 Neelakantan et al. (2015) adds noise into gradients, which makes the optimizer prefer large-basin areas and forces it to ﬁnd broad minima.”
An overview of gradient descent optimization algorithms (2016, positive): “unch”16. You should thus always monitor error on a validation set during training and stop (with some patience) if your validation error does not improve enough. 6.4 Gradient noise Neelakantan et al. [13] add noise that follows a Gaussian distribution N(0;˙2 t ) to each gradient update: g t;i= g t;i”
Three Mechanisms of Weight Decay Regularization (2018, background): “Since higher learning rates lead to larger gradient noise, which has been shown to act as a stochastic regularizer (Neelakantan et al., 2015; Keskar et al., 2016; Jastrzębski et al., 2017; Hoffer et al., 2017), this means weight decay can indirectly exert a regularizing effect through the effective ”
TerpreT: A Probabilistic Programming Language for Program Induction (2016, methodology): “We remain optimistic about extensions to the TERPRET framework which allow differentiable interpreters to handle problems involving perceptual data [Gaunt et al., 2016a], and about using machine learning techniques to guide search-based techniques [Balog et al., 2016].”