Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization

By Changyou Chen, David Edwin Carlson, Zhe Gan, Chunyuan Li, Lawrence Carin. 2015

94 citations recorded by the source corpus.

Stochastic gradient Markov chain Monte Carlo (SG-MCMC) methods are Bayesian analogs to popular stochastic optimization methods; however, this connection is not well studied. We explore this relationship by applying simulated annealing to an SGMCMC algorithm. Furthermore, we extend recent SG-MCMC methods with two key components: i) adaptive preconditioners (as in ADAgrad or RMSprop), and ii) adaptive element-wise momentum weights. The zero-temperature limit gives a novel stochastic optimization method with adaptive element-wise momentum weights, while conventional optimization methods only have a shared, static momentum weight. Under certain assumptions, our theoretical analysis suggests the proposed simulated annealing approach converges close to the global optima. Experiments on several deep neural network models show state-of-the-art results compared to related stochastic optimization algorithms.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Entropy-SGD: biasing gradient descent into wide valleys (2016, methodology): “We follow the experiments in Welling & Teh (2011) and Chen et al. (2015) and set the learning rate schedule to be η / ( 1 + t ) b where the initial learning rate η and b are hyper-parameters.”
Stochastic Gradient Descent as Approximate Bayesian Inference (2017, background): “Finally, Chen et al. (2016) also draw analogies between SGD and scalable MCMC.”
Deep Rewiring: Training very sparse deep networks (2017, background): “Chen et al. (2016) proposed a momentum scheme and temperature annealing (for the temperature T in our notation) for stochastic gradient updates, leading to a stochastic optimization method.”
Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks (2017, background): “In fact, our results show that since j = 0 for deep networks due to non-isotropic gradient noise, very simple algorithms such as SGLD by [54] also benefit from the acceleration that their sophisticated counterparts aim for [55, 56].”
A Variational Analysis of Stochastic Gradient Algorithms (2016, methodology): “Chen et al. (2015b) analyze stochastic gradient MCMC and studied their convergence properties using stochastic di ﬀ erential equations.”
Towards Better Representations with Deep/Bayesian Learning (2018, methodology): “A lot of work in Bayesian community is how to compute the posteriors and this expectation efficiently and accurately (Chen et al., 2016b; Zhang et al., 2017c, 2018a,b).”
Asynchronous Stochastic Quasi-Newton MCMC for Non-Convex Optimization (2018, methodology): “In other recent studies, SGLD has also been extended to synchronous (Ahn et al., 2014) and asynchronous (Chen et al., 2016b; Springenberg et al., 2016) distributed MCMC settings.”
Continuous-Time Flows for Efficient Inference and Density Estimation (2017, methodology): “In fact, CTF has been applied for scalable Bayesian sampling (Ding et al., 2014; Li et al., 2016a; Chen et al., 2016; Li et al., 2016b; Zhang et al., 2017).”
Bayesian Optimization with Robust Bayesian Neural Networks (2016, unknown): “Following these update equations, the distribution of (θ, r) is the one in Equation (6), and θ is guaranteed to be distributed according to p(θ | D).”
Learning Weight Uncertainty with Stochastic Gradient MCMC for Shape Classification (2016, background): “neal the system temperature of the Itô diffusion to zero, in the limit SG-MCMC becomes stochastic optimization [13].”
Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning (2019, methodology): “It is interesting to note that our approach inverts steps of the Santa algorithm (Chen et al., 2016a) for optimization.”