Understanding predictive information criteria for Bayesian models

By Andrew Gelman, Jessica Hwang, Aki Vehtari. Statistics and computing, 2013

2,120 citations recorded by the source corpus.

We review the Akaike, deviance, and Watanabe-Akaike information criteria from a Bayesian perspective, where the goal is to estimate expected out-of-sample-prediction error using a bias-corrected adjustment of within-sample error. We focus on the choices involved in setting up these measures, and we compare them in three simple examples, one theoretical and two applied. The contribution of this paper is to put all these information criteria into a Bayesian predictive context and to better understand, through small examples, how these methods can apply in practice.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/39f7e8219e75256b34be2994068eebdbf58c9093

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

A quantile parametric mixed regression model for bounded response variables (2017, methodology): “Wanatabe’s proposal can be viewed as an approximation to cross-validation [15] and the only difference with respect to EAIC and EBIC is the computation of the complexity penalty pWAIC .”
A guide to Bayesian model selection for ecologists (2015, methodology): “Overall, WAIC seems very appealing, however, the main disadvantage is substantial depending on the area of application: its calculation relies on an independence assumption of the data given the parameters.”
The Electromagnetic Counterpart of the Binary Neutron Star Merger LIGO/Virgo GW170817. II. UV, Optical, and Near-infrared Light Curves and Comparison to Kilonova Models (2017, methodology): “We compare models using the Watanabe–Akaike Information Criteria (WAIC; Watanabe 2010; Gelman et al. 2014), which accounts for both the likelihood score and the number of fitted parameters.”
Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors (2020, methodology): “This ﬁrst requires estimation of the pointwise leave-one-out (loo) log predictive density (lpd, Gelman et al., 2014) from the k -th cluster (chain): where S ( K ) is the space of K -dimensional simplex and p prior ( w ) is prior regularization.”
A Unified Framework for Fitting Bayesian Semiparametric Models to Arbitrarily Censored Survival Data, Including Spatially Referenced Data (2017, background): “ach CPO i is penalized. In addition to DIC and LPML, the Watanabe-Akaike information criterion (WAIC) (Watanabe, 2010) has also gained popularity in recent years due to its stability compared to DIC (Gelman et al., 2014; Vehtari and Gelman, 2014). The WAIC is dened as WAIC = 2 Xn i=1 log 1 L XL l=1 ”
Programming With Models: Writing Statistical Algorithms for General Model Structures With NIMBLE (2015, methodology): “…et al., 2007; Lele et al., 2010; de Valpine, 2012), new approximations (e.g., Rue et al., 2009), methods for model selection and assessment (e.g., Hjort et al., 2006; Gelman et al., 2014), combinations of ideas such as sequential Monte Carlo and MCMC (Andrieu et al., 2010), and many others.”
Modelling the influence of biotic factors on species distribution patterns (2016, neutral): “But, all these methods over fit because of their equivalence to cross-validation (Plummer, 2008; Gelman et al ., 2014).”
Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC (2015, methodology): “…shows that WAIC gives an asymptotically unbiased estimate of the out-of-sample prediction error—this does not hold for hierarchical models with weak prior information as shown by Gelman, Hwang, and Vehtari (2014)—but exact LOO is slightly biased as the LOO posteriors use only n − 1 observations.”
A Semantic Approach for Estimating Consumer Content Preferences from Online Search Queries (2018, methodology): “We also tried other potential evaluation metrics for Bayesian models, such as DIC and Watanabe–Akaike information criterion (Gelman et al. 2014).”
R-squared for Bayesian Regression Models (2019, positive): “As with other measures of predictive model fit, overfitting should be less of an issue with Bayesian inference because averaging over the posterior distribution is more conservative than taking a leastsquares or maximum likelihood fit, but predictive accuracy for new data will still on average be lo”
The Magnetar Model for Type I Superluminous Supernovae. I. Bayesian Analysis of the Full Multicolor Light-curve Sample with MOSFiT (2017, methodology): “In the Appendix, we provide another measure of fit quality for Bayesian models, the Watanabe-Akaike Information Criterion (WAIC; Watanabe 2010; Gelman et al. 2014); however, this is primarily intended for comparison between different models.”
Performances of LOO and WAIC as IRT Model Selection Methods (2017, methodology): “Before we discuss the computation of LOO and WAIC, it is necessary to introduce log pointwise predictive density (LPPD), which can be computed with the following equation (Gelman et al., 2014) 1 log ( | ) ( )θ θ θ = = n i post i LPPD p y p d .”
Bayesian inference and model comparison for metallic fatigue data (2015, background): “• Watanabe-Akaike information criterion (WAIC) WAIC or widely applicable information criterion is a stable Bayesian predictive measure that approximates the leave-one-out cross-validation (see [27, 28, 29]) and is defined by”