Stochastic Gradient Descent as Approximate Bayesian Inference

By Stephan Mandt, Matthew D. Hoffman, David M. Blei. 2017

758 citations recorded by the source corpus.

Stochastic Gradient Descent with a constant learning rate (constant SGD) simulates a Markov chain with a stationary distribution. With this perspective, we derive several new results. (1) We show that constant SGD can be used as an approximate Bayesian posterior inference algorithm. Specifically, we show how to adjust the tuning parameters of constant SGD to best match the stationary distribution to a posterior, minimizing the Kullback-Leibler divergence between these two distributions. (2) We demonstrate that constant SGD gives rise to a new variational EM algorithm that optimizes hyperparameters in complex probabilistic models. (3) We also propose SGD with momentum for sampling and show how to adjust the damping coefficient accordingly. (4) We analyze MCMC algorithms. For Langevin Dynamics and Stochastic Gradient Fisher Scoring, we quantify the approximation errors due to finite learning rates. Finally (5), we use the stochastic process perspective to give a short proof of why Polyak averaging is optimal. Based on this idea, we propose a scalable approximate MCMC algorithm, the Averaged Stochastic Gradient Sampler.

Subfields: Artificial Intelligence, Statistics and Probability, Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

An Empirical Model of Large-Batch Training (2018, background): “[MHB17] derived a sampling distribution for SGD, motivating our deﬁnition of ‘temperature’.”
The large learning rate phase of deep learning: the catapult mechanism (2020, background): “According to this suggestion, training with a large learning rate (or with a small batch size) can improve performance because it leads to more stochasticity during training (Mandt et al., 2017; Smith et al., 2017; Smith & Le, 2018; Smith et al., 2018).”
Regularizing and Optimizing LSTM Language Models (2017, background): “ASGD has been analyzed in depth theoretically and many surprising results have been shown including its asymptotic second-order convergence (Polyak & Juditsky, 1992; Mandt et al., 2017).”
Don't Use Large Mini-Batches, Use Local SGD (2018, methodology): “The variance matrix Σ p w q , following Hoffer et al. (2017) and Hu et al. (2017), for uniform sampling of mini-batch indices can be approximated as Recent works (Jastrz˛ebski et al., 2018; Li et al., 2017; Mandt et al., 2017; Zhu et al., 2019) interpret eq.”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, methodology): “[34] proposed to use the iterates of averaged SGD as an MCMC sampler, after analyzing the dynamics of SGD using tools from stochastic calculus.”
Averaging Weights Leads to Wider Optima and Better Generalization (2018, methodology): “Following this explanation from [Mandt et al., 2017], we can interpret points proposed by SGD as being constrained to the surface of a sphere, since they come from a high dimensional Gaussian distribution.”
Improving Neural Language Modeling via Adversarial Training (2019, positive): “We also improve the AWD-LSTM-MoS baseline by an amount of 1.18/1.17 and 2.14/2.03 in perplexity for both datasets.”
An Alternative View: When Does SGD Escape Local Minima? (2018, background): “With the help of noise, SGD can also be viewed as doing approximate Bayesian inference [22] or variational inference [3].”
Deep Neural Networks as Gaussian Processes (2017, background): “Whether gradient-based stochastic optimization implements an approximate Bayesian computation is an interesting question (Mandt et al., 2017).”
Three Factors Influencing Minima in SGD (2017, methodology): “This is a similar assumption to previous papers Mandt et al. (2017); Poggio & et al. (2018). Assumption 2 The covariance of the gradients and the Hessian of the loss approximation are approximately equal, i.”
There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average (2018, background): “A.7 INCLUDING HIGH LEARNING RATE ITERATES INTO SWA As discussed in Mandt et al. (2017), under certain assumptions SGD samples from a Gaussian distribution centered at the optimum of the loss w0 with covariance proportional to the learning rate.”
A Bayesian Perspective on Generalization and Stochastic Gradient Descent (2017, positive): “To achieve this, Bayesians add noise to the gradient updates (Mandt et al., 2017; Welling & Teh, 2011).”
Practical Deep Learning with Bayesian Principles (2019, background): “Many other approaches have recently been proposed to compute posterior approximations by training deterministic networks [46, 37, 38].”
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam (2018, background): “Another related work by Mandt et al. (2017) views SG descent as VI but requires additional effort to obtain posterior approximations, while in our approach the approximation is automatically obtained within an adaptive method.”
Gaussian Process Behaviour in Wide Deep Neural Networks (2018, neutral): “The latter comparison to SGD is suggestive that this optimization method mimics Bayesian inference – an idea that has been receiving increasing attention (Welling and Teh, 2011; Mandt et al., 2017; Smith and Le, 2018).”
Deep Ensembles: A Loss Landscape Perspective (2019, methodology): “One could also compute an (exponential moving) average of the weights along the trajectory, inspired by Polyak-Ruppert averaging in convex optimization, (see also [Mandt et al., 2017] for a Bayesian view on iterate averaging).”
Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks (2017, methodology): “…regularization: The idea that SGD is related to variational inference has been seen in machine learning before (Duvenaud et al., 2016; Mandt et al., 2016) under assumptions such as quadratic steady-states; for instance, see Mandt et al. (2017) for methods to approximate steady-states using SGD.”