The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Minima and Regularization Effects

By Zhanxing Zhu, Jingfeng Wu, Ting Yu, Lei Wu, Jin Ma. 2018

43 citations recorded by the source corpus.

Understanding the behavior of stochastic gradient descent (SGD) in the context of deep neural networks has raised lots of concerns recently. Along this line, we theoretically study a general form of gradient based optimization dynamics with unbiased noise, which unifies SGD and standard Langevin dynamics. Through investigating this general optimization dynamics, we analyze the behavior of SGD on escaping from minima and its regularization effects. A novel indicator is derived to characterize the efficiency of escaping from minima through measuring the alignment of noise covariance and the curvature of loss function. Based on this indicator, two conditions are established to show which type of noise structure is superior to isotropic noise in term of escaping efficiency. We further show that the anisotropic noise in SGD satisfies the two conditions, and thus helps to escape from sharp and poor minima effectively, towards more stable and flat minima that typically generalize well. We verify our understanding through comparing this anisotropic diffusion with full gradient descent plus isotropic diffusion (i.e. Langevin dynamics) and other types of position-dependent noise.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Don't Use Large Mini-Batches, Use Local SGD (2018, background): “Given the positive effect of adding well-structured stochastic noise to SGD dynamics for non-convex problems (Smith & Le, 2018; Chaudhari & Soatto, 2018; Zhu et al., 2018; Xing et al., 2018), the communication efficient post-local SGD can generalize better and lead to flatter minima than mini-batch ”
On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent (2018, background): “…in order for convergence rates to improve linearly [9]; the other regards the SGD update equation as the Euler-Maruyama discretization of a stochastic differential equation [26, 30], and attempts to maintain a constant level of mini-batch noise to help SGD explore the loss landscape [3, 35, 29].”
Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning (2020, methodology): “[34] proved that Σ̄θ∗ of SGD on deep neural networks well aligns the Hessian matrix H(θ∗), namely the top eigenvectors associated with large eigenvalues in Σθ∗ have similar directions in those in H(θ∗).”
A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks (2019, methodology): “A popular approach for investigating the behavior SGD is based on considering SGD as a discretization of a continuous-time process (Mandt et al., 2016; Jastrzebski et al., 2017; Li et al., 2017a; Hu et al., 2017; Zhu et al., 2018; Chaudhari & Soatto, 2018).”
The Break-Even Point on Optimization Trajectories of Deep Neural Networks (2020, background): “This is inspired by recent work showing training can escape a region with too large curvature compared to the learning rate (Zhu et al., 2018; Wu et al., 2018; Jastrzebski et al., 2018).”
Limitations of the Empirical Fisher Approximation (2019, methodology): “It has also been employed to explain certain characteristics of SGD [Zhu et al., 2018, Jastrzebski et al., 2018] or as a diagnostic tool during training [Liao et al., 2018].”
The Implicit and Explicit Regularization Effects of Dropout (2020, background): “Jastrzębski et al. (2017); Zhu et al. (2018); Wen et al. (2019) study how the noise distribution in SGD aﬀects generalization.”
An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise (2019, background): “A similar observation has been made in Zhu et al. (2018) where the authors studied the effects of how using Fisher covariance noise relates to the efficiency of escaping from local minima and compare it to isotropic noise.”
Interplay Between Optimization and Generalization of Stochastic Gradient Descent with Covariance Noise (2019, background): “More direct approaches of studying the covariance structure of mini-batch gradients have also been explored (Jastrzebski et al., 2017; Xing et al., 2018; Zhu et al., 2018; Li et al., 2015).”