Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks

By Chunyuan Li, Changyou Chen, David Edwin Carlson, Lawrence Carin. 2015

368 citations recorded by the source corpus.

Effective training of deep neural networks suffers from two main issues. The first is that the parameter space of these models exhibit pathological curvature. Recent methods address this problem by using adaptive preconditioning for Stochastic Gradient Descent (SGD). These methods improve convergence by adapting to the local geometry of parameter space. A second issue is overfitting, which is typically addressed by early stopping. However, recent work has demonstrated that Bayesian model averaging mitigates this problem. The posterior can be sampled by using Stochastic Gradient Langevin Dynamics (SGLD). However, the rapidly changing curvature renders default SGLD methods inefficient. Here, we propose combining adaptive preconditioners with SGLD. In support of this idea, we give theoretical properties on asymptotic convergence and predictive risk. We also provide empirical results for Logistic Regression, Feedforward Neural Nets, and Convolutional Neural Nets, demonstrating that our preconditioned SGLD method gives state-of-the-art performance on these models.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling (2018, methodology): “h a manner that each update results in an approximate sample from the posterior. Different strategies have been developed for augmenting the gradients and noise according to a preconditioning matrix. Li et al. (2016) show that a preconditioner based on the RMSprop algorithm performs well on deep neu”
Adding Gradient Noise Improves Learning for Very Deep Networks (2015, neutral): “Similar concerns motivated the development of very recent algorithms for preconditioned SGLD in the Bayesian setting (Li et al., 2016).”
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam (2018, background): “Many other works have explored variety of approximation methods, e.g., Gal & Ghahramani (2016) use dropout for VI, Hernandez-Lobato & Adams (2015); Hasenclever et al. (2017) use expectation propagation, Li et al. (2016); Balan et al. (2015) use stochastic-gradient Langevin dynamics.”
Learning Weight Uncertainty with Stochastic Gradient MCMC for Shape Classification (2016, methodology): “Algorithms SG-MCMC Optimization Basic SGLD SGD Preconditioning pSGLD [2] RMSprop/Adagrad/Adam Momentum SGHMC momentum SGD Thermostat SGNHT Santa [3]”
Bayesian Recurrent Neural Networks (2017, methodology): “Others have also considered applying SGLD to neural networks (Li et al., 2015) and Gan et al. (2016) more recently used SGLD for LSTMs (we compare to these results in our experiments).”
Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization (2015, methodology): “We increase accuracy by implementing the symmetric splitting scheme (SSS) (Chen et al., 2015; Li et al., 2016b).”
Bayesian Inference for Large Scale Image Classification (2019, background): “One approach is to estimate the covariance of the stochastic gradient noise B explicitly and use it correct and pre-condition the sampling dynamics [6, 25].”
Deep learning classifiers for hyperspectral imaging: A review (2019, methodology): “For instance Chen et al. (2016) and Haut et al. (2019a) train a CNN2D with one principal component (PC), while Liang and Li (2016) employ three PCs to train the CNN2D and post-process the extracted spatial-features with sparse coding (SC) (Charles et al., 2011; Song et al., 2014) to create a sparse…”
Adv-BNN: Improved Adversarial Defense through Robust Bayesian Neural Network (2018, methodology): “For SGLD and its extension (e.g. (Li et al., 2016)), since the algorithms are essentially SGD updates with extra Gaussian noise, they are very easy to implement.”
Stochastic natural gradient descent draws posterior samples in function space (2018, methodology): “Patterson & Teh (2013) combined SGLD with Riemannian Langevin dynamics while Li et al. (2016) combined SGLD with RMSprop.”
Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning (2019, methodology): “We simply use the prescribed temperature of Li et al. (2016a) without tuning, but better results of the sampling methods can be achieved by tuning the temperature.”
Generalizing Hand Segmentation in Egocentric Videos With Uncertainty-Guided Model Adaptation (2020, background): “Many other works have also been proposed to enable scalable variational inference in large Bayesian deep learning problems [18, 21, 29, 1].”
Bayesian Optimization with Robust Bayesian Neural Networks (2016, neutral): “[21] and to derive a hybrid between SGD optimization and HMC sampling in Chen et al.”
Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors (2018, methodology): “Recent advances in stochastic MCMC meth-ods (Li et al., 2016; Welling & Teh, 2011) and stochastic variational methods (Blundell et al., 2015; Rezende et al., 2014), black-box variational and alpha-divergence minimization (Hernandez-Lobato et al., 2016; Ranganath et al., 2014), and probabilistic…”
Continuous-Time Flows for Efficient Inference and Density Estimation (2017, methodology): “In fact, CTF has been applied for scalable Bayesian sampling (Ding et al., 2014; Li et al., 2016a; Chen et al., 2016; Li et al., 2016b; Zhang et al., 2017).”
Towards Better Representations with Deep/Bayesian Learning (2018, methodology): “A popular method is to use Stochastic Gradient Markov Chain Monte Carlo (SG-MCMC) methods to generate posterior samples (Welling and Teh, 2011; Chen et al., 2014; Ding et al., 2014; Li et al., 2016a; Zhang et al., 2017c).”