Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression

By Tim Salimans, David A. Knowles. 2012

248 citations recorded by the source corpus.

textabstractWe propose a general algorithm for approximating nonstandard Bayesian posterior distributions. The algorithm minimizes the Kullback-Leibler divergence of an approximating distribution to the intractable posterior distribu- tion. Our method can be used to approximate any posterior distribution, provided that it is given in closed form up to the proportionality constant. The approxi- mation can be any distribution in the exponential family or any mixture of such distributions, which means that it can be made arbitrarily precise. Several exam- ples illustrate the speed and accuracy of our approximation method in practice.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Auto-Encoding Variational Bayes (2013, neutral): “In [SK13], a similar reparameterization as in this paper was used in an efﬁcient version of a stochastic variational inference algorithm for learning the natural parameters of exponential-family approximating distributions.”
Multiplicative Normalizing Flows for Variational Bayesian Neural Networks (2017, methodology): “…increase the ﬂexibility of the approximate posteriors in a Bayesian neural network we will rely on auxiliary random variables (Agakov & Bar-ber, 2004; Salimans et al., 2013; 2015; Ranganath et al., 2015; Maaløe et al., 2016); more speciﬁcally we will exploit the well known “multiplicative noise”…”
Variational Dropout and the Local Reparameterization Trick (2015, background): “Variable reparameterizations have a long history in the statistics literature, but have only recently found use for efficient gradient-based machine learning and inference [4, 13, 19].”
Stochastic Gradient Descent as Approximate Bayesian Inference (2017, methodology): “A modiﬁed form of black-box variational inference relies on re-parameterization gradients (Salimans and Knowles, 2013; Kingma and Welling, 2014; Rezende et al., 2014; Kucukelbir et al., 2015; Ruiz et al., 2016).”
Hierarchical Variational Models (2015, background): “Mixtures can approximate arbitrary distributions (given enough components), and have been considered as variational families (Jaakkola and Jordan, 1998; Lawrence, 2000; Gershman and Blei, 2012; Salimans et al., 2013).”
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam (2018, methodology): “In contrast, variational inference (VI) methods can scale to large models by using stochastic-gradient (SG) methods, as recent work has shown (Graves, 2011; Blundell et al., 2015; Ranganath et al., 2014; Salimans et al., 2013).”
The Generalized Reparameterization Gradient (2016, methodology): “The second approach to Monte Carlo gradients of the variational objective is through reparame-terization (Price, 1958; Bonnet, 1964; Salimans and Knowles, 2013; Kingma and Welling, 2014; Rezende et al., 2014).”
Stick-Breaking Variational Autoencoders (2016, background): “…Autoencoder (VAE) is model comprised of two multilayer perceptrons: one acts as a density network (MacKay & Gibbs, 1999) mapping a latent variable z i to an observed datapoint x i , and the other acts as an inference model (Salimans & Knowles, 2013) performing the reverse mapping from x i to z i .”
Deep Exponential Families (2014, methodology): “Thus we use more recent “black box” variational inference techniques that step around computing this expectation [40, 34, 27].”
Black-box $\alpha$-divergence Minimization (2015, methodology): “But unlike EP, which computes different cavity distributions for different data points, BB-α uses the same cavity distribution for each data point.”
An Introduction to Variational Autoencoders (2019, unknown): “In (Salimans and Knowles, 2013), a similar reparameterization as in this work was used in an efficient version of a stochastic variational inference algorithm for learning the natural parameters of exponential-family approximating distributions.”
Markov Chain Monte Carlo and Variational Inference: Bridging the Gap (2014, unknown): “Figure 2: R-squared accuracy measure [1] for approximate posteriors using a varying number of leapfrog steps.”
Semiparametric Mean Field Variational Bayes: General Principles and Numerical Issues (2016, methodology): “Salimans and Knowles (2013) devised a stochastic approximation algorithm for solving (6) when Q is a parametric family of exponential family form.”
Kullback-Leibler Proximal Variational Inference (2015, background): “Several generalizations of SVI have been proposed for general latent-variable models where the lower bound might be intractable [2, 3, 4].”
Advances in Variational Inference (2017, methodology): “Due to their flexibility as well as computational difficulties, advancing VI for mixture models has been of continuous interest [8], [51], [58], [124], [170].”
Monte Carlo Gradient Estimation in Machine Learning (2019, methodology): “o look at the non-dierentiable case by splitting regions into dierentiable and non-dierentiable components. The implicit reparameterisation gradients for univariate distributions were developed by Salimans and Knowles (2013). Homan and Blei (2015) used the implicit gradients to perform backpropagati”
Markov Chain Monte Carlo and Variational Inference: Bridging the Gap (2014, methodology): “Figure 2: R-squared accuracy measure [1] for approximate posteriors using a varying number of leapfrog steps.”
Local Expectation Gradients for Black Box Variational Inference (2015, background): “The second direction (expectation stochasticity) aims at dealing with the intractable expectations under the variational distribution that are encountered in non-conjugate probabilistic models [12, 14, 10, 18, 8, 15, 20].”