Boosting Variational Inference

By Fangjian Guo, Xiangyu Wang, Kai Fan, Tamara Broderick, David B. Dunson. 2016

81 citations recorded by the source corpus.

Variational inference (VI) provides fast approximations of a Bayesian posterior in part because it formulates posterior approximation as an optimization problem: to find the closest distribution to the exact posterior over some family of distributions. For practical reasons, the family of distributions in VI is usually constrained so that it does not include the exact posterior, even as a limit point. Thus, no matter how long VI is run, the resulting approximation will not approach the exact posterior. We propose to instead consider a more flexible approximating family consisting of all possible finite mixtures of a parametric base distribution (e.g., Gaussian). For efficient inference, we borrow ideas from gradient boosting to develop an algorithm we call boosting variational inference (BVI). BVI iteratively improves the current approximation by mixing it with a new component from the base distribution family and thereby yields progressively more accurate posterior approximations as more computing time is spent. Unlike a number of common VI variants including mean-field VI, BVI is able to capture multimodality, general posterior covariance, and nonstandard posterior shapes.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

BooVAE: Boosting Approach for Continual Learning of VAE (2019, methodology): “Guo et al. (2016) suggested using concave log-det regularization for gaussian base learners.”
MaxEntropy Pursuit Variational Inference (2019, background): “Our goal is to improve accuracy of the approximation in terms of the KL-divergence DKL[qt(θ)||p(θ”
Variational Boosting: Iteratively Refining Posterior Approximations (2016, methodology): “In concurrent work, boosting has been used to construct ﬂexible approximations to posterior distributions [11].”
Advances in Variational Inference (2017, methodology): “Due to their flexibility as well as computational difficulties, advancing VI for mixture models has been of continuous interest [8], [51], [58], [124], [170].”
Semi-Implicit Variational Inference (2018, methodology): “Note we have also tried modifying the code of variational boosting (Guo et al., 2016; Miller et al., 2017) for Bayesian logistic regression, but failed to obtain satisfactory results.”
Boosting Variational Inference: an Optimization Perspective (2017, methodology): “These optimization subtleties were not addressed in [4, 20] but are essential for the convergence of Algorithm 1.”
Importance Weighted Hierarchical Variational Inference (2019, background): “One possible approach is to augment some standard q(z|x) with help of copulas (Tran et al., 2015), mixtures (Guo et al., 2016; Gershman et al., 2012), or invertible transformations with tractable Jacobians also known as normalizing flows (Rezende and Mohamed, 2015; Kingma et al., 2016; Dinh et al.,…”