Stochastic Gradient Riemannian Langevin Dynamics on the Probability Simplex

By Sam Patterson, Yee Whye Teh. 2013

274 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Stochastic Gradient Hamiltonian Monte Carlo (2014, methodology): “One attempt at applying such methods in a sampling context is the recently proposed stochastic gradient Langevin dynamics (SGLD) (Welling and Teh, 2011; Ahn et al., 2012; Patterson and Teh, 2013).”
Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling (2018, methodology): “Patterson & Teh (2013) suggested using the Fisher information matrix as a preconditioner in SGLD.”
Bayesian dark knowledge (2015, neutral): “…Ther-mostat (SG-NHT) [DFB + 14] (which improves upon SGHMC), stochastic gradient Fisher scoring (SGFS) [AKW12] (which uses second order information), stochastic gradient Riemannian Langevin Dynamics (for inferring parameters in the probability simplex) [PT13], distributed SGLD [ASW14], etc.”
Deep Rewiring: Training very sparse deep networks (2017, background): “Patterson & Teh (2013) considered the problem of stochastic gradient dynamics constrained to the probability simplex.”
A Complete Recipe for Stochastic Gradient MCMC (2015, methodology): “We follow the experimental settings in [13] for Riemmanian samplers (SGRLD and SGRHMC), taking the hyper-parameters of Dirichlet priors to be γ = 0 .”
Towards Better Representations with Deep/Bayesian Learning (2018, methodology): “Later, SGLD was extended by (Ahn et al., 2012), (Patterson and Teh, 2013) and (Korattikara et al., 2015).”
Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks (2015, methodology): “(3)The update form in (Patterson and Teh 2013) is more complicated and seemingly different from (3); however, they can be shown to be equivalent.”
A Unified and General Framework for Continual Learning (2024, methodology): “To model this intuition, we set the matrix D ( θ ) = F − 1 , where F is the FIM on previous tasks and set Q ( θ ) = 0 (Patterson & Teh, 2013).”
Structured Stochastic Variational Inference (2014, unknown): “…defined as in equation 7 (main text) we can compute ∂R∂λ as ∂R ∂λ |u,λ = ∂T ∂R̂ |R̂(u,λ) ∂R̂ ∂λ |u,λ ∂R̂k ∂λ = −q̂k(R̂(uk, λ)) −1 ∂Q̂k ∂λ , (24) where q̂k is the PDF of the kth random variable obtained via the kth univariate quantile function R̂k and Q̂k is the CDF that is the inverse of R̂k.”
Riemannian Adaptive Optimization Methods (2018, methodology): “Stochastic gradient Langevin dynamics was generalized as well, to improve optimization on the probability simplex (Patterson & Teh, 2013).”
A survey of uncertainty in deep neural networks (2021, methodology): “and samplers in the Riemannian manifold [174] using the first order Langevin dynamics and Levy diffusion noise and momentum [175].”
WarpLDA: a Cache Efficient O(1) Algorithm for Latent Dirichlet Allocation (2015, background): “Examples include stochastic variational inference (SVI) [17], streaming variational Bayes [9], and stochastic gradient Riemann Langevin dynamics (SGRLD) [25].”
Gradient-based Constrained Sampling from Language Models (2022, background): “, 2017) as soft-representations is not compatible with Gaussian noise and can lead to undesirable behavior (Patterson and Teh, 2013).”
Stochastic Gradient Markov Chain Monte Carlo (2019, methodology): “Patterson and Teh (2013) proposed the first SGLD algorithm for sampling from the probability simplex.”
Bayesian Sampling Using Stochastic Gradient Thermostats (2014, methodology): “This idea was first used in optimization [9, 19] but was recently adapted for sampling methods based on stochastic differential equations (SDEs) such as Brownian dynamics [1, 18, 24] and Langevin dynamics [5].”
WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling (2018, methodology): “Different from AVITM (Srivastava & Sutton, 2017) that models a probabilistic simplex with the expanded-natural representation (Patterson & Teh, 2013), DLDA-TLASGR uses a more elegant simplex constraint and increases the sampling efﬁciency via the use of the Fisher information matrix (FIM) (Cong et…”
Stochastic natural gradient descent draws posterior samples in function space (2018, unknown): “Patterson & Teh (2013) combined SGLD with Riemannian Langevin dynamics while Li et al. (2016) combined SGLD with RMSprop.”
Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization (2015, methodology): “In the MCMC literature, preconditioning is alternatively referred to as Riemannian information geometry (Patterson and Teh, 2013).”