Bayesian Posterior Sampling via Stochastic Gradient Fisher Scoring

By S. Ahn, Sungjin Ahn, Anoop Korattikara Balan, Max Welling. 2012

318 citations recorded by the source corpus.

In this paper we address the following question: "Can we approximately sample from a Bayesian posterior distribution if we are only allowed to touch a small mini-batch of data-items for every sample we generate?". An algorithm based on the Langevin equation with stochastic gradients (SGLD) was previously proposed to solve this, but its mixing rate was slow. By leveraging the Bayesian Central Limit Theorem, we extend the SGLD algorithm so that at high mixing rates it will sample from a normal approximation of the posterior, while for slow mixing rates it will mimic the behavior of SGLD with a pre-conditioner matrix. As a bonus, the proposed algorithm is reminiscent of Fisher scoring (with stochastic gradients) and as such an efficient optimizer during burn-in.

Subfields: Artificial Intelligence, Statistics and Probability, Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

Variational Inference with Normalizing Flows (2015, methodology): “This approach has been explored for sampling from complex densities by Welling & Teh (2011); Ahn et al. (2012); Suykens et al. (1998).”
Bayesian Model-Agnostic Meta-Learning (2018, positive): “…et al. (2018), the task-train posterior of a Bayesian neural network (BNN) trained with a few-shot dataset would have a signiﬁcant amount of uncertainty which, according to the Bayesian central limit theorem (Le Cam, 1986; Ahn et al., 2012), cannot be well approximated by a Gaussian distribution.”
Variational Dropout and the Local Reparameterization Trick (2015, neutral): “Later useful refinements include [23] and [1].”
Stochastic Gradient Hamiltonian Monte Carlo (2014, methodology): “It is also possible to set B̂ = 12 V̂ , where V̂ is estimated using empirical Fisher information as in (Ahn et al., 2012) for SGLD.”
Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling (2018, methodology): “SGFS Burning = 500 ,learningrate γ = 0 .”
Bayesian dark knowledge (2015, neutral): “…[CFG14], stochastic gradient Nos ´ e-Hoover Ther-mostat (SG-NHT) [DFB + 14] (which improves upon SGHMC), stochastic gradient Fisher scoring (SGFS) [AKW12] (which uses second order information), stochastic gradient Riemannian Langevin Dynamics (for inferring parameters in the probability simplex)…”
Stochastic Gradient Descent as Approximate Bayesian Inference (2017, methodology): “We compared constant SGD without preconditioning and with diagonal (-d) and full rank (-f) preconditioning against Stochastic Gradient Langevin Dynamics and Stochastic Gradient Fisher Scoring (SGFS) with diagonal (-d) and full rank (-f) preconditioning, and BBVI. the smallest principal components of”
Stochastic Back-propagation and Variational Inference in Deep Latent Gaussian Models (2014, methodology): “In all cases, we train using mini-batches, which requires the introduction of scaling terms in the free energy objective function (14) in order to maintain the correct scale between the prior over the parameters and the remaining terms (Ahn et al., 2012; Welling & Teh, 2011).”
Privacy for Free: Posterior Sampling and Stochastic Gradient Monte Carlo (2015, methodology): “These include Stochastic Gradient Langevin dynamics (SGLD) (Welling & Teh, 2011), Stochstic Gradient Fisher scoring (SGFS) (Ahn et al., 2012), Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) (Chen et al., 2014) as well as more recent Stochastic Gradient Nos´e-Hoover Thermostat (SGNHT) (Ding et a”
Automated Scalable Bayesian Inference via Hilbert Coresets (2017, methodology): “…for variational Bayes (Hoﬀman et al., 2013; Broderick et al., 2013; Campbell et al., 2015), subsampling methods for MCMC (Welling and Teh, 2011; Ahn et al., 2012; Bardenet et al., 2014; Korattikara et al., 2014; Maclaurin and Adams, 2014; Bardenet et al., 2015), and distributed “consensus”…”
A Variational Analysis of Stochastic Gradient Algorithms (2016, methodology): “Speciﬁ-cally, we use the stochastic process perspective to compute the stationary sampling distribution of stochastic gradient Fisher scoring (Ahn et al., 2012).”
Expectation Propagation as a Way of Life: A Framework for Bayesian Inference on Partitioned Data (2014, background): “The top-down view deals with fitting statistical models to large data sets, for which many distributed (divide-and-conquer) algorithms have been proposed over the past few years (Ahn et al., 2012; Balan et al., 2014; Hoffman et al., 2013; Scott et al., 2016; Wang and Dunson, 2013; Neiswanger et al.,”
Measuring Sample Quality with Diffusions (2016, background): “and quantify bias-variance tradeos in approximate Markov chain Monte Carlo. A discussion of related and future work follows in Section7, and all proofs are deferred to the appendices. Notation For r2[1;1], let kk r denote the ‘r norm on Rd. We will use kkas a generic norm on Rd satisfying kkkk 2 and”
Advances in Variational Inference (2017, background): “In recent years, there has been considerable progress in both ﬁelds [7], [15], and in particular on bridging the gap between these methods [1], [90], [113], [154], [169].”