Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning

By Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, Andrew Gordon Wilson. 2019

298 citations recorded by the source corpus.

The posteriors over neural network weights are high dimensional and multimodal. Each mode typically characterizes a meaningfully different representation of the data. We develop Cyclical Stochastic Gradient MCMC (SG-MCMC) to automatically explore such distributions. In particular, we propose a cyclical stepsize schedule, where larger steps discover new modes, and smaller steps characterize each mode. We also prove non-asymptotic convergence of our proposed algorithm. Moreover, we provide extensive experimental results, including ImageNet, to demonstrate the scalability and effectiveness of cyclical SG-MCMC in learning complex multimodal distributions, especially for fully Bayesian inference with modern deep neural networks.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors (2020, methodology): “Zhang et al. (2020) use 9 MCMC samples and report 77.1% accuracy and 0.888 NLL; and Heek & Kalchbrenner (2019) use 30 MCMC samples and report 77.5% accuracy and 0.883 NLL. Rank-1 BNNs have a similar parameter count to deterministic ResNet-50 instead of incurring a 9-30x memory cost and use a single…”
Deep Ensembles: A Loss Landscape Perspective (2019, methodology): “Due to space constraints, we do not consider Markov Chain Monte Carlo (MCMC) methods in this work; Zhang et al. [2020] show that popular stochastic gradient MCMC (SGMCMC) methods may not explore multiple modes and propose cyclic SGMCMC.”
Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision (2019, methodology): “To mitigate this problem, Zhang et al. [53] proposed to use a cyclical stepsize schedule to help escaping local modes in p ( θ”
Hyperparameter Ensembles for Robustness and Uncertainty Quantification (2020, background): “For instance, cyclical learning-rate schedules can explore several local minima [36, 76] where ensemble members can be snapshot.”
Understanding Uncertainty in Bayesian Neural Networks (2019, background): “, 2016), step size scheduling (Zhang et al., 2019) and preconditioning (Ma et al.”
Laplace Redux - Effortless Bayesian Deep Learning (2021, methodology): “CSGHMC We use the publicly available code provided by the original authors [68].”
Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty (2020, methodology): “Among themost popular are variational inference [20, 57, 71] and stochastic gradient MCMC [32, 184, 194].”
Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing (2019, methodology): “…been used to train deep neural networks (Smith, 2017), warm restart stochastic gradient descent (Loshchilov and Hutter, 2017), improve convergence rates (Smith and Topin, 2017), obtain model ensembles (Huang et al., 2017) and explore multimodal distributions in MCMC sampling (Zhang et al., 2019).”
Drug discovery with explainable artificial intelligence (2020, methodology): “Many of these methods treat neural networks as Bayesian models, by considering a prior distribution over its learnable weights, and then performing inference over their posterior distribution with various methods (Figure 4b), e.g., Markov Chain Monte Carlo 151 or variational inference.”
Bayesian Inference for Large Scale Image Classification (2019, result): “Similar to earlier work [21] we found that many fewer samples are needed when a cyclic step size with cycle length n is used.”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, methodology): “[75] developed a cyclical SG-MCMC (cSGMCMC) to compute the posterior over the weights of neural networks.”
Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning (2020, methodology): “Finally, the results of cyclical Stochastic Gradient Hamiltonian Monte Carlo (SGHMC), (Zhang et al., 2019) which reportedly has marginally better performance compared with cyclical SGLD, could not be reproduced with any value of SGD momentum term.”
What Are Bayesian Neural Network Posteriors Really Like? (2021, methodology): “…literature for Bayesian neural networks has mainly focused on stochastic gradient-based methods (Welling & Teh, 2011; Ahn et al., 2014; Chen et al., 2014; Ma et al., 2015; Ahn et al., 2012; Ding et al., 2014; Zhang et al., 2020b; Garriga-Alonso & Fortuin, 2021) for computational efficiency reasons.”