Explaining Variational Approximations

By John Thomas Ormerod, Matt P. Wand. 2010

428 citations recorded by the source corpus.

Variational approximations facilitate approximate inference for the parameters in complex statistical models and provide fast, deterministic alternatives to Monte Carlo methods. However, much of the contemporary literature on variational approximations is in Computer Science rather than Statistics, and uses terminology, notation, and examples from the former field. In this article we explain variational approximation in statistical terms. In particular, we illustrate the ideas of variational approximation using examples that are familiar to statisticians.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression (2012, methodology): “We compare our algorithm to the VBEM algorithm of Ormerod and Wand (2010) which makes use of the fact that the expectations required for this model can be calculated analytically.”
Penalized Wavelets: Embedding Wavelets into Semiparametric Regression (2011, methodology): “In Ormerod and Wand [49] we explained MFVB using statistical examples similar to those presented here.”
A guide to Bayesian model selection for ecologists (2015, methodology): “This gives us the flexibility to use model reparameterizations and newer computational tricks such as variational Bayes (e.g., Omerod and Wand 2010) and statistical emulators (e.g., Hooten et al. 2011) to speed up the model fitting process, which in turn aids in out-of-sample model selection.”
Mean field variational bayes for elaborate distributions (2011, methodology): “The Kullback-Leibler-based lower bound on the marginal log-likelihood is Ormerod and Wand, 2010).”
A Stochastic Variational Framework for Fitting and Diagnosing Generalized Linear Mixed Models (2012, methodology): “stochastic nonconjugate variational message passing algorithm for tting GLMMs, which is scalable to large data sets. We start with a brief introduction to variational approximation methods (see, e.g. Ormerod and Wand, 2010) and review of nonconjugate variational message passing (Knowles and Minka, 2”
Regression Density Estimation With Variational Methods and Stochastic Approximation (2012, background): “Ormerod and Wand (2010) gives an introduction to VA methods that is particularly accessible to statisticians.”
Scalable Variational Inference for Bayesian Variable Selection in Regression, and Its Accuracy in Genetic Association Studies (2012, methodology): “We investigate a two-part solution to this problem using variational methods (Jordan et al. 1999; Ormerod and Wand 2010; Wainwright and Jordan 2008) and importance sampling (e.g. Andrieu et al. 2003).”
A variational Bayes approach to variable selection (2017, methodology): “Mean ﬁeld variational Bayes (VB) is an efﬁcient but approximate alternative to MCMC for Bayesian inference (Bishop, 2006; Ormerod and Wand, 2010).”
Variance reduction properties of the reparameterization trick (2018, methodology): “Background Variational inference (VI) (Jordan et al., 1999; Ormerod and Wand, 2010; Blei et al., 2017) provides a fast and approximate alternative to exact Monte Carlo methods when performing Bayesian inference on parameters in complex statistical models.”
ASYMPTOTIC NORMALITY AND VALID INFERENCE FOR GAUSSIAN VARIATIONAL APPROXIMATION (2011, background): “As explained in [3] and [13], there are many types of variational approximations.”
Variational Approximations for Generalized Linear Latent Variable Models (2017, positive): “A comprehensive summary of the VA approach can be found in Ormerod and Wand (2010), but briefly, VA belongs to a rich class of approximations for converting a di ffi cult optimization problem to a simpler one, whose roots begin in quantum mechanics (Sakurai, 1985) and were subsequently taken up in…”
Variational Inference for Generalized Linear Mixed Models Using Partially Noncentered Parametrizations (2012, background): “Q 6 See Ormerod and Wand (2010) for an explanation of variational approximation methods very accessible to statisticians.”
Variational Inference for Heteroscedastic Semiparametric Regression (2015, methodology): “We now consider the problem of mean field-type variational inference for (5) (e.g. Wainwright & Jordan 2008; Ormerod & Wand 2010).”
Blind source separation problem in GPS time series (2016, background): “The most common restriction on p′(W) is that it factorizes into ∏N i=1 p′(wi ) for some partition {w1, ...,wN} of W (e.g., Ormerod and Wand 2010).”