Variational Boosting: Iteratively Refining Posterior Approximations

By Andrew C. Miller, N. Foti, Ryan P. Adams. 2016

134 citations recorded by the source corpus.

We propose a black-box variational inference method to approximate intractable distributions with an increasingly rich approximating class. Our method, termed variational boosting, iteratively refines an existing variational approximation by solving a sequence of optimization problems, allowing the practitioner to trade computation time for accuracy. We show how to expand the variational approximating class by incorporating additional covariance structure and by introducing new components to form a mixture. We apply variational boosting to synthetic and real statistical models, and show that resulting posterior inferences compare favorably to existing posterior approximation algorithms in both accuracy and efficiency.

Subfields: Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

Citations in the corpus

BooVAE: Boosting Approach for Continual Learning of VAE (2019, background): “Several works (Miller et al., 2017; Gershman et al., 2012) perform direct optimization of ELBO with respect to the new component’s parameters.”
Boosting Variational Inference: an Optimization Perspective (2017, methodology): “This is substantially different to what is presented in [4, 20] as the explicit assumptions we make allows us to understand how the choices in the distribution family influences the rate.”
SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient (2018, methodology): “A straightforward and popular approach to optimize L is to use stochastic gradient methods [24, 26, 28, 35].”
Advances in Variational Inference (2017, methodology): “Here, Boosting VI and variational boosting [58], [124] reﬁne the approximate posterior iteratively by adding one component at a time while keeping previously ﬁtted components ﬁxed.”
Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors (2020, methodology): “Proofs and related lemmas are in Appendix A.”
Semi-Implicit Variational Inference (2018, methodology): “Note we have also tried modifying the code of variational boosting (Guo et al., 2016; Miller et al., 2017) for Bayesian logistic regression, but failed to obtain satisfactory results.”
MaxEntropy Pursuit Variational Inference (2019, background): “The other research direction explores the idea of incrementally expanding variational family by the additive mixture of tractable base learners [11,18].”
Boosting Variational Inference (2016, methodology): “We highlight that a similar idea and treatment is developed in a concurrent and independent paper Miller et al. [2016].”
Reinterpreting Importance-Weighted Autoencoders (2017, methodology): “To summarize our observations, the following is the ordering of lower bounds given speciﬁc proposal distributions, In light of this, IWAE can be seen as increasing the complexity of the approximate distribution q , similar to other methods that increase the complexity of q , such as Normalizing Flow”
Variance reduction properties of the reparameterization trick (2018, methodology): “Complex variational families have been proposed, e.g. Gaussian mixtures to account for multi-modality (Zobay, 2014; Miller et al., 2016) and Gaussian copulas (Han et al., 2016) for ﬂexible multivariate modeling.”