Bayesian Learning via Stochastic Gradient Langevin Dynamics

By Max Welling, Yee Whye Teh. 2011

3,086 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Don't Decay the Learning Rate, Increase the Batch Size (2017, background): “Intuitively, equation 1 ensures we can reach the minimum, no matter how far away our parameters are initialized, while equation 2 ensures that the learning rate decays sufﬁciently quickly that we converge to the minimum, rather than bouncing around it due to gradient noise (Welling & Teh, 2011).”
Improved Techniques for Training Score-Based Generative Models (2020, methodology): “When α is sufficiently small and T is sufficiently large, the distribution of xT will be close to p(x) under some regularity conditions [6, 7].”
Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, methodology): “Following the practice in [65] we separately set the standard deviation of the Gaussian noise as 0 .”
Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, methodology): “G.1 IMPROPER SGLD Recall the transition kernel of SGLD: x0 ∼ p0(x) xi+1 = xi − α 2 ∂Eθ(xi) ∂θ + , ∼ N (0, α) In the proper formulation of this sampler (Welling & Teh, 2011), the step-size and the variance of the Gaussian noise are related Var( ) = α.”
Generative Modeling by Estimating Gradients of the Data Distribution (2019, background): “The distribution of x̃T equals p(x) when → 0 and T →∞, in which case x̃T becomes an exact sample from p(x) under some regularity conditions [62].”
Variational Inference with Normalizing Flows (2015, neutral): “This approach has been explored for sampling from complex densities by Welling & Teh (2011); Ahn et al. (2012); Suykens et al. (1998).”
A note on the evaluation of generative models (2015, methodology): “These issues can be addressed by better image models (e.g., Kingma et al., 2014), better optimization and inference procedures, or a more Bayesian treatment of the parameters (e.g., Lacoste-Julien et al., 2011; Welling & Teh, 2011).”
Neural Programmer: Inducing Latent Programs with Gradient Descent (2015, methodology): “We use a schedule inspired from Welling & Teh (2011), where at every step we sample a Gaussian of 0 mean and variance = curr step − 0 .”
Stochastic variational inference (2012, methodology): “Another is the stochastic gradient Langevin method of Welling and Teh (2011), which enjoys asymptotic convergence guarantees and also takes advantage of stochastic optimization.”
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, methodology): “…(Blundell et al., 2015; Graves, 2011; Louizos and Welling, 2016), assumed density ﬁltering (Hern´andez-Lobato and Adams, 2015), expectation propagation (Hasenclever et al., 2015; Li et al., 2015) or stochastic gradient Langevin diffusion methods (Korattikara et al., 2015; Welling and Teh, 2011).”
Variational Inference: A Review for Statisticians (2016, methodology): “Some MCMC methods can also exploit these innovations (Welling and Teh 2011; Ahmed et al. 2012).”
Wireless Network Intelligence at the Edge (2018, methodology): “The work [131] resolves this problem, by first adding an extra random noise to entropy SGD’s weight updates, as done in SGLD [130].”
Data Poisoning Attacks on Factorization-Based Collaborative Filtering (2016, background): “A less investigated problem is how to synthesize malicious data points that are hard for a defender to detect.”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, methodology): “Finally, while the empirical focus of our paper is classiﬁcation calibration, we also compare to additional approximate BNN inference methods which perform well on smaller architectures, including deterministic variational inference (DVI) [60], single-layer deep GPs (DGP) with expectation propagatio”
Entropy-SGD: biasing gradient descent into wide valleys (2016, methodology): “We follow the experiments in Welling & Teh (2011) and Chen et al. (2015) and set the learning rate schedule to be η / ( 1 + t ) b where the initial learning rate η and b are hyper-parameters.”
A Kernel Test of Goodness of Fit (2016, methodology): “For the purpose of illustration we use a simple generative model from Gorham & Mackey (2015); Welling & Teh (2011), θ1 ∼ N (0, 10); θ2 ∼ N (0, 1) Xi ∼ 1 2 N (θ1, 4) + 1 2 N (θ2 + θ1, 4).”
Kernel Adaptive Metropolis-Hastings (2013, methodology): “In this case, a “burn-in” phase is used to get a rough sketch of the shape of the distribution: the initial samples need not come from a converged or even valid MCMC chain, and it suffices to have a scheme with good exploratory properties, e.g., Welling & Teh (2011).”