Validating Bayesian Inference Algorithms with Simulation-Based Calibration

By Sean Talts, Michael Betancourt, Daniel P. Simpson, Aki Vehtari, Andrew Gelman. 2018

386 citations recorded by the source corpus.

Verifying the correctness of Bayesian computation is challenging. This is especially true for complex models that are common in practice, as these require sophisticated model implementations and algorithms. In this paper we introduce \emph{simulation-based calibration} (SBC), a general procedure for validating inferences from Bayesian algorithms capable of generating posterior samples. This procedure not only identifies inaccurate computation and inconsistencies in model implementations but also provides graphical summaries that can indicate the nature of the problems that arise. We argue that SBC is a critical part of a robust Bayesian workflow, as well as being a useful tool for those developing computational algorithms and statistical software.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/e11817ce34636abb2aedf31442c040aee12208a4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flows (2018, methodology): “To assess whether SNL is well-calibrated, we performed a simulation-based calibration test [75]: we generated 200 pairs ( θ n , x n ) from the joint p ( θ , x ), used SNL to approximate the posterior for each x n , obtained 9 close-to-independent samples from each posterior, and calculated the rank…”
Towards constraining warm dark matter with stellar streams through neural simulation-based inference (2020, methodology): “For more recent methodologies we refer the reader to Talts et al. (2018) and Dalmasso et al. (2020).”
Likelihood-free MCMC with Approximate Likelihood Ratios (2019, methodology): “The diagnostic is based on the observation that if θ (cid:48) ∼ p ( θ ) and x ∼ p ( x | θ ) , then exact posterior quantiles for each parameter will be uniformly distributed, provided that the pos-teriors are absolutely continuous [28].”
BayesFlow: Learning Complex Stochastic Models With Invertible Neural Networks (2020, background): “, whether credibility intervals are indeed credible); • Simulation-based calibration (SBC, [50]) - to visually detect systematic biases in the approximate posteriors;”
Bayesian Workflow (2020, methodology): “Furthermore, depending on the specific approximation being used, generic diagnostic tools described by Yao et al. (2018a) and Talts et al. (2020) can be used to verify that a particular approximate algorithm reproduces the features of the posterior that you care about for a specific model.”
GILBO: One Metric to Measure Them All (2018, methodology): “Figure 7: Simulation-based calibration (Talts et al., 2018) of the variational encoder for the same three models as in Figure 6.”