Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
By Aki Vehtari, Andrew Gelman, Jonah Gabry. 2015
5,067 citations recorded by the source corpus.
Leave-one-out cross-validation (LOO) and the widely applicable information criterion (WAIC) are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model using the log-likelihood evaluated at the posterior simulations of the parameter values. LOO and WAIC have various advantages over simpler estimates of predictive error such as AIC and DIC but are less used in practice because they involve additional computational steps. Here we lay out fast and stable computations for LOO and WAIC that can be performed using existing simulation draws. We introduce an efficient computation of LOO using Pareto-smoothed importance sampling (PSIS), a new procedure for regularizing importance weights. Although WAIC is asymptotically equal to LOO, we demonstrate that PSIS-LOO is more robust in the finite case with weak priors or influential observations. As a byproduct of our calculations, we also obtain approximate standard errors for estimated predictive errors and for comparison of predictive errors between two models. We implement the computations in an R package called loo and demonstrate using models fit with the Bayesian inference package Stan.
Subfields: Artificial Intelligence, Control and Systems Engineering, Statistics and Probability, Probabilistic & Bayesian Methods
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Aki Vehtari
- Andrew Gelman
- Paul-Christian Burkner
- Yuling Yao
- Jonah Gabry
- Daniel P. Simpson
- Yong Luo
- Khaleel Al-Harbi
In Control and Systems Engineering
- Paul-Christian Burkner
- Aki Vehtari
- Juho Piironen
- M. Korevaar
- Yong Luo
- Khaleel Al-Harbi
- Marc K. Francke
- Yuling Yao
In Statistics and Probability
- Aki Vehtari
- Paul-Christian Burkner
- Andrew Gelman
- Yuling Yao
- Daniel P. Simpson
- Jonah Gabry
- Yong Luo
- Khaleel Al-Harbi
In Probabilistic & Bayesian Methods
- Paul-Christian Burkner
- Aki Vehtari
- Yong Luo
- Khaleel Al-Harbi
- Charles C. Margossian
- Yuling Yao
- Daniel P. Simpson
- Juho Piironen
Citations in the corpus
- Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis. (2017, methodology): “14,15 We fitted our blindness and moderate and severe vision impairment models using Bayesian inference, sampling from the posterior distribution over the parameters using Hamiltonian Monte Carlo, a Markov chain Monte Carlo method, as implemented in the RStanArm package (version 2.11.1), which…”
- A global synthesis reveals biodiversity-mediated benefits for crop production (2019, methodology): “year-nested model) using leave-one-out cross-validation (LOO), a fully Bayesian 525 model selection procedure for estimating pointwise out-of-sample prediction accuracy (56).”
- Using Stacking to Average Bayesian Predictive Distributions (with Discussion) (2017, methodology): “Our fast LOO approximation (PSIS-LOO) generally applies to factorizable models p(y|θ, x) = ∏N i=1 p(yi|θ, xi) such that the pointwise log-likelihood can be obtained easily by computing log p(yi|θ, xi).”
- Performances of LOO and WAIC as IRT Model Selection Methods (2017, methodology): “Interested readers are referred to Vehtari, Gelman, and Gabry (2015, 2016a).”
- Robustness (2020, neutral): “A well-known example is Tukey’s biweight ψ Tukey ( x ) = x (1 − x 2 ) 2 for | x | ≤ 1 and ψ Tukey ( x ) = 0 otherwise.”
- Housing markets in a pandemic: Evidence from historical outbreaks (2020, methodology): “PSIS-LOO stands for leave-one-out cross-validation using Pareto-smoothed importance sampling ( Vehtari et al., 2017 ).”
- Advanced Bayesian Multilevel Modeling with the R Package brms (2017, methodology): “Comparing model fit via leave-one-out cross validation as implemented in the loo package (Vehtari et al., 2016a,b). reveals that the second model using the number of children to predict both model parts has better fit.”
- Comparison of Bayesian predictive methods for model selection (2015, methodology): “Analytical approximations for LOO are discussed by Vehtari et al. (2014) and computations using posterior samples by Vehtari et al. (2016).”