Bayesian Sampling Using Stochastic Gradient Thermostats

By Nan Ding, Youhan Fang, Ryan Babbush, Changyou Chen, Robert D. Skeel, Hartmut Neven. 2014

243 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Entropy-SGD: biasing gradient descent into wide valleys (2016, methodology): “Conventional MCMC techniques such as SGLD or Nose’-Hoover thermostats (Ding et al., 2014) can only trade energy for entropy via the temperature parameter which does not allow the direct use of the geometric information of the energy landscape and does not help with narrow minima.”
Stochastic Gradient Descent as Approximate Bayesian Inference (2017, methodology): “Other research suggests improvements to this issue, using Hamiltonian Monte-Carlo (Chen et al., 2014) or thermostats (Ding et al., 2014).”
Privacy for Free: Posterior Sampling and Stochastic Gradient Monte Carlo (2015, methodology): “While SGNHT is able to adaptively adjust the temperature so that the samples that it produces remain “unbiased” in some sense as η → 0.”
Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks (2017, background): “In fact, our results show that since j = 0 for deep networks due to non-isotropic gradient noise, very simple algorithms such as SGLD by [54] also benefit from the acceleration that their sophisticated counterparts aim for [55, 56].”
A Variational Analysis of Stochastic Gradient Algorithms (2016, methodology): “Other research suggests improvements to this issue, using Hamiltonian Monte-Carlo (Chen et al., 2014) or thermostats (Ding et al., 2014).”
Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning (2019, methodology): “Stochastic gradient Markov Chain Monte Carlo (SG-MCMC) methods (Welling & Teh, 2011; Chen et al., 2014; Ding et al., 2014; Li et al., 2016a) provide a promising direction for a sampling based approach to inference in Bayesian deep learning.”
Provable Bayesian Inference via Particle Mirror Descent (2015, methodology): “For the experiments on logistic regression, sparse GP and LDA on real-world datasets, we use indirect criteria which are widely used (Chen et al., 2014; Ding et al., 2014; Hensman et al., 2013; Patterson and Teh, 2013; Hoffman et al., 2013) because of the intractability of the posterior.”
Adversarial Distillation of Bayesian Neural Network Posteriors (2018, methodology): “Since the introduction of SGLD, a suite of stochastic-gradient MCMC algorithms have been proposed (Ahn et al., 2012; 2014; Balan et al., 2015; Chen et al., 2014; Ding et al., 2014), drawing from the wealth of knowledge behind general MCMC techniques.”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, unknown): “The stochastic gradient MCMC (SG-MCMC) [68], [69]”
Bayesian Inference for Large Scale Image Classification (2019, methodology): “We also report the results of sampling with a sampler based Nos´e-Hoover thermostats (SGNHT) [7, 14] applied to the ResNet++ architecture.”
Continuous-Time Flows for Efficient Inference and Density Estimation (2017, methodology): “In fact, CTF has been applied for scalable Bayesian sampling (Ding et al., 2014; Li et al., 2016a; Chen et al., 2016; Li et al., 2016b; Zhang et al., 2017).”
Score-Based Generative Modeling with Critically-Damped Langevin Diffusion (2021, methodology): “Methods leveraging auxiliary variables similar to our velocities are also used in statistics—such as Hamiltonian Monte Carlo (Neal, 2011)—and have found applications, for instance, in Bayesian machine learning (Chen et al., 2014; Ding et al., 2014; Shang et al., 2015).”
A survey of uncertainty in deep neural networks (2021, background): “To do so, the ”thermostat” ideas [176], [177], [178] are proposed so that a prescribed constant temperature distribution is maintained with the parameter dependent noise.”
What Are Bayesian Neural Network Posteriors Really Like? (2021, background): “…literature for Bayesian neural networks has mainly focused on stochastic gradient-based methods (Welling & Teh, 2011; Ahn et al., 2014; Chen et al., 2014; Ma et al., 2015; Ahn et al., 2012; Ding et al., 2014; Zhang et al., 2020b; Garriga-Alonso & Fortuin, 2021) for computational efficiency reasons.”
Deep Poisson Factor Modeling (2015, methodology): “Other alternatives for scaling up inference in Bayesian models such as the parameter server [13, 19], conditional density filtering [9] and stochastic gradient-based approaches [4, 5, 28] are left as interesting future work.”
Hamiltonian ABC (2015, methodology): “After an overview of HMC, we will briefly describe stochastic gradient Hamiltonian dynamics (SGHD), starting with using Langevin dynamics [25], then HMC with friction [5], and finally HMC with thermostats [6].”
Towards Better Representations with Deep/Bayesian Learning (2018, methodology): “, 2014) and stochastic gradient Nose-Hoover Thermostats (SGNHT) (Ding et al., 2014).”