Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors

By Yuling Yao, Aki Vehtari, Andrew Gelman. 2020

79 citations recorded by the source corpus.

When working with multimodal Bayesian posterior distributions, Markov chain Monte Carlo (MCMC) algorithms can have difficulty moving between modes, and default variational or mode-based approximate inferences will understate posterior uncertainty. And, even if the most important modes can be found, it is difficult to evaluate their relative weights in the posterior. Here we propose an alternative approach, using parallel runs of MCMC, variational, or mode-based inference to hit as many modes or separated regions as possible, and then combining these using importance sampling based Bayesian stacking, a scalable method for constructing a weighted average of distributions so as to maximize cross-validated prediction utility. The result from stacking is not necessarily equivalent, even asymptotically, to fully Bayesian inference, but it serves many of the same goals. Under misspecified models, stacking can give better predictive performance than full Bayesian inference, hence the multimodality can be considered a blessing rather than a curse. We explore with an example where the stacked inference approximates the true data generating process from the misspecified model, an example of inconsistent inference, and non-mixing samplers. We elaborate the practical implantation in the context of latent Dirichlet allocation, Gaussian process regression, hierarchical model, variational inference in horseshoe regression, and neural networks.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Foundation Posteriors for Approximate Probabilistic Inference (2022, neutral): “To demonstrate the foundation posterior, we meta-amortize inference over a set of standard Stan [12] programs from PosteriorDB [43], a benchmark dataset for evaluating inference algorithms [4, 5, 69, 70, 20].”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, unknown): “[302] (SI: Stacked inference), Prijatelj et al.”
Bayesian Workflow (2020, methodology): “When you have computational problems, often there’s a problem with your model (Yao, Vehtari, and Gelman, 2020).”
Bernstein flows for flexible posteriors in variational Bayes (2022, unknown): “b) Cauchy experiment: Here, we follow an example from [31] and fit an unconditional Cauchy model Y ∼ Cauchy(ξ, γ) to six samples which we have drawn from a mixture of two Cauchy distributions Y ∼ Cauchy(ξ1 = −2.”