Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory
By Sumio Watanabe. 2010
2,808 citations recorded by the source corpus.
In regular statistical models, the leave-one-out cross-validation is asymptotically equivalent to the Akaike information criterion. However, since many learning machines are singular statistical models, the asymptotic behavior of the cross-validation remains unknown. In previous studies, we established the singular learning theory and proposed a widely applicable information criterion, the expectation value of which is asymptotically equal to the average Bayes generalization loss. In the present paper, we theoretically compare the Bayes cross-validation loss and the widely applicable information criterion and prove two theorems. First, the Bayes cross-validation loss is asymptotically equivalent to the widely applicable information criterion as a random variable. Therefore, model selection and hyperparameter optimization using these two values are asymptotically equivalent. Second, the sum of the Bayes generalization error and the Bayes cross-validation error is asymptotically equal to 2λ/n, where λ is the real log canonical threshold and n is the number of training samples. Therefore the relation between the cross-validation error and the generalization error is determined by the algebraic geometrical structure of a learning machine. We also clarify that the deviance information criteria are different from the Bayes cross-validation and the widely applicable information criterion.
Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Aki Vehtari
- Andrew Gelman
- Quoc V. Le
- Kyunghyun Cho
- Mohit Bansal
- Yoshua Bengio
- Sergey Levine
- Percy Liang
In Probabilistic & Bayesian Methods
- Aki Vehtari
- Andrew Gelman
- Yong Luo
- Khaleel Al-Harbi
- Yuling Yao
- J. Guillochon
- Daniel P. Simpson
- José Miguel Hernández-Lobato
Citations in the corpus
- True Few-Shot Learning with Language Models (2021, methodology): “Prior work has developed other model selection criteria such as the Akaike Information Criterion [AIC; 42], Watanabe-Akaike Information Criterion [WAIC; 41], and Mallows’ C p [43].”
- Tradeoffs in Data Augmentation: An Empirical Study (2021, methodology): “We consider the mean log likelihood of augmented test images (Grathwohl et al., 2019), and the Watanabe–Akaike information criterion (WAIC) (Watanabe, 2010).”
- A guide to Bayesian model selection for ecologists (2015, background): “February 2015 19BAYESIAN MODEL SELECTION FOR ECOLOGISTS C O N C E P TS & S YN TH E S IS score, Richardson (2002), Celeux et al. (2006), and Watanabe (2010) propose the log point-wise predictive score log Yn i¼1 ½yi j y ¼ Xn i¼1 log Z ½yi j h ½h j y dh ð42Þ where Monte Carlo integration can be used…”
- Modelling the influence of biotic factors on species distribution patterns (2016, neutral): “WAIC has recently been made available in R-INLA for model comparison and is recommended because it is a more fully Bayesian approach for out-of-sample prediction (Watanabe, 2010; Gelman et al ., 2014), however, WAIC is not recommended for spatial analyses (Gelman et al ., 2014).”
- Model averaging in ecology: a review of Bayesian, information-theoretic, and tactical approaches for predictive inference (2018, background): “1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 RMSE rjMCMC median 0.00 0.01 0.00 0.11 0.00 0.00 0.08 0.11 0.00 0.14 0.00 0.09 0.14 0.13 0.10 0.09 1.069 BIC 0.00 0.01 0.00 0.18 0.00 0.03 0.17 0.04 0.00 0.19 0.00 0.04 0.24 0.05 0.05 0.01 1.074 Median† – – – – – – – – – – – – – – – – 1.075 m10‡ 0 0 0 0 0 0 0 0”
- Weighing Black Holes Using Tidal Disruption Events (2018, methodology): “To quantify how well the various combinations of parameters in the model fit each light curve, MOSFiT uses the Watanabe– Akaike information criteria (WAIC; Watanabe 2010) also known as the widely applicable Bayesian criteria.”
- WAIC, but Why? Generative Ensembles for Robust Anomaly Detection (2018, methodology): “l approximations), the ensemble can be used to implement the Watanabe-Akaike Information Criterion (WAIC), which consists of a density estimation score with a Bayesian correction term for model bias (Watanabe, 2010): WAIC(x) = E [logp (x)] Var [logp (x)] (1) 2 2.1 OOD DETECTION WITH GAN DISCRIMINATO”
- Pareto Smoothed Importance Sampling (2015, methodology): “Vehtari et al. (2017) provide additional comparisons between IS-LOO, TIS-LOO, PSIS-LOO, and widely applicable information criterion (Watanabe, 2010).”
- The Combined Ultraviolet, Optical, and Near-infrared Light Curves of the Kilonova Associated with the Binary Neutron Star Merger GW170817: Unified Data Set, Analytic Models, and Physical Implications (2017, methodology): “We compare the resulting fits utilizing the Watanabe–Akaike Information Criteria (WAIC; Watanabe 2010; Gelman et al. 2014), which accounts for both the likelihood score and number of fitted parameters for each model.”
- Spatiotemporal wildfire modeling through point processes with moderate and extreme marks (2021, methodology): “In a preliminary analysis of the regression models described in §4, we used the Widely Applicable Information Criterion (WAIC, Watanabe 2010) in a step-wise manner to compare nested models with different components in the regression equations (e.g., linear vs nonlinear effects of explanatory…”
- The deviance information criterion: 12 years on (2014, methodology): “…= ∑ i Eθ|y[−2 log{p.yi|θ/}]+2 ∑ i log{p.yi|y/}; pWAIC2 = 14 ∑ i Vθ|y[−2 log{p.yi|θ/}]: Model comparison by WAIC has been shown to be asymptotically equal to using Bayesian leaveone-out cross-validation (Watanabe, 2010), is invariant to reparameterization and will deal with mixture models.”
- Programming With Models: Writing Statistical Algorithms for General Model Structures With NIMBLE (2015, methodology): “…(Gelfand et al., 1992; Stern and Cressie, 2000), calibrated posterior predictive p-values (Hjort et al., 2006) or alternatives to DIC such as WAIC (Watanabe, 2010; Spiegelhalter et al., 2014); (vi) “likelihood free” or “plug-and-play” methods such as synthetic likelihood (Wood, 2010),…”
- Affinity and Diversity: Quantifying Mechanisms of Data Augmentation (2020, methodology): “We consider the mean log likelihood of augmented test images(Grathwohl et al., 2019), and the WatanabeAkaike information criterion (WAIC) (Watanabe, 2010).”
- Using Stacking to Average Bayesian Predictive Distributions (with Discussion) (2017, methodology): “More recently we have seen also Watanabe–Akaike information criterion (WAIC, Watanabe, 2010) and leave-one-out cross-validation estimates used to compute model weights following the idea of AIC weights.”
- The Magnetar Model for Type I Superluminous Supernovae. I. Bayesian Analysis of the Full Multicolor Light-curve Sample with MOSFiT (2017, methodology): “In the Appendix, we provide another measure of fit quality for Bayesian models, the Watanabe-Akaike Information Criterion (WAIC; Watanabe 2010; Gelman et al. 2014); however, this is primarily intended for comparison between different models.”