Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks

By Pratik Chaudhari, Stefano Soatto. 2017

339 citations recorded by the source corpus.

Stochastic gradient descent (SGD) is widely believed to perform implicit regularization when used to train deep neural networks, but the precise manner in which this occurs has thus far been elusive. We prove that SGD minimizes an average potential over the posterior distribution of weights along with an entropic regularization term. This potential is however not the original loss function in general. So SGD does perform variational inference, but for a different loss than the one used to compute the gradients. Even more surprisingly, SGD does not even converge in the classical sense: we show that the most likely trajectories of SGD for deep networks do not behave like Brownian motion around critical points. Instead, they resemble closed loops with deterministic components. We prove that such “out-of-equilibrium” behavior is a consequence of highly nonisotropic gradient noise in SGD; the covariance matrix of mini-batch gradients for deep networks has a rank as small as 1% of its dimension. We provide extensive empirical validation of these claims. This article summarizes the findings in [1]. See the longer version for background, detailed results and proofs.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Statistics and Probability, Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Statistics and Probability

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

Don't Use Large Mini-Batches, Use Local SGD (2018, background): “[5] Pratik Chaudhari and Stefano Soatto.”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, neutral): “…gradient noise covariance and thus the optimal learning rates are nearly constant; however, the total variance is much too small to induce effective learning rates, probably due to over-parameterization effects inducing non full rank gradient covariances as was found in Chaudhari and Soatto [8].”
An Alternative View: When Does SGD Escape Local Minima? (2018, positive): “With the help of noise, SGD can also be viewed as doing approximate Bayesian inference [22] or variational inference [3].”
Three Factors Influencing Minima in SGD (2017, background): “Chaudhari & Soatto (2017) explored the stationary non-equilibrium solution for the SDE for non-isotropic gradient noise.”
Understanding Batch Normalization (2018, methodology): “The effect of the normalization in (1) is similarly investigated by removing those components, results in the absence of any afﬁne transformation are shown to the right in Figure 13.”
Fantastic Generalization Measures and Where to Find Them (2019, methodology): “…we consider some optimization based measures based on the speed of the optimization algorithm as motivated by the work of (Hardt et al., 2015) and (Wilson et al., 2017a), and the magnitude of the gradient noise as motivated by the work of (Chaudhari and Soatto, 2018) and (Smith and Le, 2017).”
On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent (2018, methodology): “date equation as the Euler-Maruyama discretization of a stochastic differential equation [26, 30], and attempts to maintain a constant level of mini-batch noise to help SGD explore the loss landscape [3, 35, 29]. Both justiﬁcations for the linear scaling rule implicitly impose strong conditions on t”
Emergence of Invariance and Disentanglement in Deep Representations (2017, methodology): “oned by Dziugaite and Roy (2017), who point out that minima that are at in the sense of having a large volume, toward which stochastic gradient descent algorithms are implicitly or explicitly biased (Chaudhari and Soatto, 2018), naturally relates to the PAC-Bayes loss for the choice of a normal prio”
Inductive biases for deep learning of higher-level cognition (2020, background): “, 2018), implicit effects of the choice of the optimization method (Jastrzkebski et al., 2017; Smith and Le, 2017; Chaudhari and Soatto, 2018), self-supervised learning or self-supervised pre-training (Hinton et al.”
Stochastic Mirror Descent on Overparameterized Nonlinear Models (2019, background): “There have been many efforts in the past few years to study deep learning from an optimization perspective, e.g., (Achille & Soatto, 2017; Chaudhari & Soatto, 2018; Shwartz-Ziv & Tishby, 2017; Allen-Zhu et al., 2019; Oymak & Soltanolkotabi, 2019; Azizan & Hassibi, 2019; Ma et al., 2018b; Du et al.,…”
Optimal SGD Hyperparameters for Fully Connected Networks (2018, background): “As discussed in [1, 2, 9, 5, 11], the continuum limit of stochastic gradient descent is described by the stochastic differential equation dwi = −∂iLτdt+ dξi , E[dξidξj ]|w,t = 2gFij(w)dt , (4) where wi is used to denote the parameters of the neural network, and dξi is random noise.”
The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Minima and Regularization Effects (2018, methodology): “Typically, in the scenario of modern deep neural networks, due to the over-parameterization, Hessian and the gradient covariance are usually ill-conditioned and anistropic near minima, as shown by [18] and [3].”
Escaping Saddles with Stochastic Gradients (2018, background): “Recent empirical observations show that this is not the case for training neural networks (Chaudhari & Soatto, 2017).”
Neural networks with late-phase weights (2020, background): “Following Liu & Theodorou (2019) and Chaudhari & Soatto (2018), the corresponding continuous-time dynamics are: dxt = −F (xt)dt+ √ 2β−1D(xt)dWt (20) with Wt Brownian motion and β , 2Bη the inverse temperature.”
A Walk with SGD (2018, background): “…of ﬂatter minima generalize better Hochreiter and Schmidhuber (1997); Keskar et al. (2016); Wu et al. (2017), and that SGD ﬁnds such minima when used with small batch size and large learning rate Keskar et al. (2016); Jastrz˛ebski et al. (2017); Smith et al. (2017); Chaudhari and Soatto (2017).”