Variational inference in nonconjugate models

By Chong Wang, David M. Blei. 2012

246 citations recorded by the source corpus.

Mean-field variational methods are widely used for approximate posterior inference in many probabilistic models. In a typical application, mean-field methods approximately compute the posterior with a coordinate-ascent optimization algorithm. When the model is conditionally conjugate, the coordinate updates are easily derived and in closed form. However, many models of interest--like the correlated topic model and Bayesian logistic regression--are nonconjugate. In these models, mean-field methods cannot be directly applied and practitioners have had to develop variational algorithms on a case-by-case basis. In this paper, we develop two generic methods for nonconjugate models, Laplace variational inference and delta method variational inference. Our methods have several advantages: they allow for easily derived variational algorithms with a wide class of nonconjugate models; they extend and unify some of the existing algorithms that have been derived for specific models; and they work well on real-world data sets. We studied our methods on the correlated topic model, Bayesian logistic regression, and hierarchical Bayesian logistic regression.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Variational Inference: A Review for Statisticians (2016, methodology): “Wang and Blei (2013) adapted Laplace approximations and the delta method to this end, improving inference in nonconjugate generalized linear models and topic models; this approach is also used by Bugbee, Breidt, and van der Woerd (2016) for semiparametric regression.”
Edward: A library for probabilistic modeling, inference, and criticism (2016, methodology): “This extends to many algorithms: for example, exact EM for exponential families; contrastive divergence (Hinton, 2002); pseudo-marginal and ABC methods (Andrieu and Roberts, 2009); Gibbs sampling within variational inference (Wang and Blei, 2012); Laplace variational inference (Wang and Blei, 2013);”
A General Method for Robust Bayesian Modeling (2015, methodology): “We address the problem with Laplace variational inference (Wang and Blei, 2013).”
Firefly Monte Carlo: Exact MCMC with Subsets of Data (2014, methodology): “We applied FlyMC to the logistic regression task described in Welling and Teh [2011] using the Jaakkola-Jordan bounds described earlier.”
Kullback-Leibler Proximal Variational Inference (2015, methodology): “Most of these methods first approximate the log p̃d(z|λ) term by using a linear or quadratic approximation and then compute the expectation.”
Advances in Variational Inference (2017, methodology): “r optimization routines [15], [182], [195] often become prohibitively slow for practical inference tasks. In contrast, approaches based on additional lower bounds with closed-form updates [74], [81], [183] can be computationally more efﬁcient. Examples include extensions of the variational message p”
Laplace Variational Approximation for Semiparametric Regression in the Presence of Heteroskedastic Errors (2016, methodology): “We use an embedded Laplace approximation to handle the nonconjugacy found in our variational approximation (Nott, Tran, and Leng 2012; Wang and Blei 2013).”
Modeling, Inference and Optimization With Composable Differentiable Procedures (2016, background): “As variational inference procedures are most often framed in terms of lower bounds on the marginal likelihood, we expect that Firefly Monte Carlo will benefit from developments in so-called “black box” variational methods [111, 83].”
Stochastic variational inference (2012, methodology): “or a speciﬁc nonconjugate Bayesian nonparametric model.) Recent research has developed general methods for non-conjugate models (Knowles and Minka, 2011; Gershman et al., 2012; Paisley et al., 2012a; Wang and Blei, 2013). Can these be scaled up with stochastic optimization? We developed our algorith”
A Model of Text for Experimentation in the Social Sciences (2016, background): “Quantitative approaches to text data analysis have a long history in the social sciences (Mendenhall 1887; Zipf 1932; Yule 1944; Miller, Newman, and Friedman 1958).”
stm: An R Package for Structural Topic Models (2019, methodology): “By contrast in STM we combine the Laplace approximate variational inference scheme of Wang and Blei (2013) with a partially collapsed strategy inspired by Khan and Bouchard (2009) in which we analytically integrate out the token-level parameters.”
Discovering Discrete Latent Topics with Neural Variational Inference (2017, methodology): “While alternatives have been explored, such as log-normal topic distributions (Blei & Laf-ferty, 2006; 2007), extra approximation (e.g. the Laplace approximation (Wang & Blei, 2013)) is required for closed form derivations.”
General Latent Feature Models for Heterogeneous Datasets (2017, background): “, 2012), and latent variable models with non-conjugate likelihoods (Wang and Blei, 2013; Khan et al., 2013).”
Celeste: Variational inference for a generative model of astronomical images (2015, methodology): “This technique is known as delta-method variational inference (Braun & McAuliffe, 2010; Wang & Blei, 2013).”
Expectation Propagation as a Way of Life (2020, background): “…proposed to combine inferences from partitioned data (see Ahn, Korattikara, and Welling, 2012, Gershman, Hoffman, and Blei, 2012, Korattikara, Chen, and Welling, 2013, Hoffman et al., 2013, Wang and Blei, 2013, Scott et al., 2013, Wang and Dunson, 2013, Neiswanger et al., 2013, and Wang, 2014).”