Nonparametric variational inference

By Samuel J. Gershman, Matthew D. Hoffman, David M. Blei. 2012

160 citations recorded by the source corpus.

Variational methods are widely used for approximate posterior inference. However, their use is typically limited to families of distributions that enjoy particular conjugacy properties. To circumvent this limitation, we propose a family of variational approximations inspired by nonparametric kernel density estimation. The locations of these kernels and their bandwidth are treated as variational parameters and optimized to improve an approximate lower bound on the marginal likelihood of the data. Unlike most other variational approximations, using multiple kernels allows the approximation to capture multiple modes of the posterior. We demonstrate the efficacy of the nonparametric approximation with a hierarchical logistic regression model and a nonlinear matrix factorization model. We obtain predictive performance as good as or better than more specialized variational methods and MCMC approximations. The method is easy to apply to graphical models for which standard variational methods are difficult to derive.

Subfields: Artificial Intelligence, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Variational Inference with Normalizing Flows (2015, background): “Another potentially powerful alternative would be to specify the approximate posterior as a mixture model, such as those developed by Jaakkola & Jordan (1998); Jordan et al. (1999); Gershman et al. (2012).”
BooVAE: Boosting Approach for Continual Learning of VAE (2019, background): “Several works (Miller et al., 2017; Gershman et al., 2012) perform direct optimization of ELBO with respect to the new component’s parameters.”
Hierarchical Variational Models (2015, methodology): “Inspired by kernel density estimation, this was classically studied by Jaakkola and Jordan (1998), and later revisited by Gershman et al. (2012), in which one speciar X iv :1 51 1.”
Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm (2016, methodology): “[5], and find they tend to give very similar results on these (relatively simple) datasets; see Appendix for more details.”
DeLiGAN: Generative Adversarial Networks for Diverse and Limited Data (2017, background): “For example, Ger-shman et al. [7], Jordan et al. [11] and Jaakkola et al. [10] model the approximate posterior of the inferred latent distribution as a mixture model to represent more complicated distributions.”
Semiparametric Mean Field Variational Bayes: General Principles and Numerical Issues (2016, methodology): “Recent summaries are given in, for example, Givens and Hoeting (2005) and Ackleh et al.”
Semi-Implicit Variational Inference (2018, methodology): “…between latent variables (Saul & Jordan, 1996; Hoffman & Blei, 2015), using a mixture of variational distributions (Bishop et al., 1998; Gershman et al., 2012; Salimans & Knowles, 2013; Guo et al., 2016; Miller et al., 2017), introducing a copula to capture the dependencies between…”
Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors (2020, methodology): “Earlier works have used a mixture of mean-ﬁeld approximations to match the posterior (Bishop et al., 1998; Jaakkola and Jordan, 1998; Gershman et al., 2012; Ranganath et al., 2016; Gal and Ghahramani, 2016; Miller et al., 2017; Chang et al., 2019).”
Decomposing Spatiotemporal Brain Patterns into Topographic Latent Sources (2014, unknown): “However, linearization often works quite well in practice (Braun and McAuliffe, 2010; Chappell et al., 2009; Friston et al., 2007; Gershman et al., 2012), and we confirm this empirically below.”
Variational Boosting: Iteratively Refining Posterior Approximations (2016, background): “Nonparametric variational inference [9] is a black-box variational inference that approximates a target distribution with a mixture of equally-weighted isotropic normals.”
Gaussian variational approximation with sparse precision matrices (2016, background): “Gershman et al., 2012; Salimans and Knowles, 2013). Rezende and Mohamed (2015) propose the specification of the approximate posterior using normalizing flows. Starting with say simple factorized distributions, highly flexible and complex approximate posteriors are constructed by transforming the ini”
Advances in Variational Inference (2017, methodology): “Due to their flexibility as well as computational difficulties, advancing VI for mixture models has been of continuous interest [8], [51], [58], [124], [170].”
Expectation Propagation as a Way of Life (2020, background): “Various approaches have been recently proposed to combine inferences from partitioned data (see Ahn, Korattikara, and Welling, 2012, Gershman, Hoffman, and Blei, 2012, Korattikara, Chen, and Welling, 2013, Hoffman et al., 2013, Wang and Blei, 2013, Scott et al., 2013, Wang and Dunson, 2013,…”
Importance Weighted Hierarchical Variational Inference (2019, background): “One possible approach is to augment some standard q(z|x) with help of copulas (Tran et al., 2015), mixtures (Guo et al., 2016; Gershman et al., 2012), or invertible transformations with tractable Jacobians also known as normalizing flows (Rezende and Mohamed, 2015; Kingma et al., 2016; Dinh et al.,…”
Provable Bayesian Inference via Particle Mirror Descent (2015, background): “We summarize the comparison between PMD and static SMC, SGLD, SVI and NPV in Table 1.”
Boosting Variational Inference (2016, methodology): “, Gaussians with isotropic covariance and equal weights [Gershman et al., 2012]).”
Affine Independent Variational Inference (2012, background): “In contrast, other deterministic approximate inference routines are not universally applicable to non-smooth target densities – for instance the Laplace approximation and [10] both require the target to be smooth.”
Two Methods for Wild Variational Inference (2016, methodology): “m all the hand-designed learning rates. Results averaged on 100 random trails. Bayesian Logistic Regression We consider Bayesian logistic regression for binary classiﬁcation using the same setting as Gershman et al. (2012), which assigns the regression weights wwith a Gaussian prior p 0(wj) = N(w; 1”
Variational inference in nonconjugate models (2012, unknown): “Gersh-man et al. (2012) proposed a nonparametric variational inference algorithm, which can be applied to nonconjugate models.”