A widely applicable Bayesian information criterion

By Sumio Watanabe. Journal of machine learning research, 2012

927 citations recorded by the source corpus.

A statistical model or a learning machine is called regular if the map taking a parameter to a probability distribution is one-to-one and if its Fisher information matrix is always positive definite. If otherwise, it is called singular. In regular statistical models, the Bayes free energy, which is defined by the minus logarithm of Bayes marginal likelihood, can be asymptotically approximated by the Schwarz Bayes information criterion (BIC), whereas in singular models such approximation does not hold. Recently, it was proved that the Bayes free energy of a singular model is asymptotically given by a generalized formula using a birational invariant, the real log canonical threshold (RLCT), instead of half the number of parameters in BIC. Theoretical values of RLCTs in several statistical models are now being discovered based on algebraic geometrical methodology. However, it has been difficult to estimate the Bayes free energy using only training samples, because an RLCT depends on an unknown true distribution. In the present paper, we define a widely applicable Bayesian information criterion (WBIC) by the average log likelihood function over the posterior distribution with the inverse temperature 1/log n, where n is the number of training samples. We mathematically prove that WBIC has the same asymptotic expansion as the Bayes free energy, even if a statistical model is singular for or unrealizable by a statistical model. Since WBIC can be numerically calculated without any information about a true distribution, it is a generalized version of BIC onto singular statistical models.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/d9c9b74c8cbfa5475c8e91312efac956bf0518ab

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Understanding predictive information criteria for Bayesian models (2013, background): “Watanabe (2013) has also proposed a widely applicable Bayesian information criterion (WBIC) which works also in singular and unrealizable cases.”
ArviZ a unified library for exploratory analysis of Bayesian models in Python (2019, background): “Additionally, it supports a number of statistical checks, such as calculating the effective sample size, the r-hat statistic, Pareto-smoothed importance sampling leave-one-out cross validation (PSIS-LOO-CV) (Vehtari, Gelman, and Gabry 2015), and widely applicable information criterion (WAIC) (Watana”
Estimating animal density without individual recognition using information derivable exclusively from camera traps (2018, methodology): “The distribution for staying time in the focal area with the highest predictive power was selected from exponential, gamma, log- normal and Weibull distributions based on their Watanabe–Akaike information criterion (WAIC) values (Watanabe, 2013).”
Model averaging in ecology: a review of Bayesian, information-theoretic, and tactical approaches for predictive inference (2018, background): “The “widely applicable information criterion” WAIC (Watanabe 2010 and an equivalent WBIC Watanabe 2013) are motivated and actually analytically derived in a Bayesian framework (Gelman et al. 2014).”
A Bayesian information criterion for singular models (2013, methodology): “In our simulations, we also consider the ‘widely applicable Bayesian information criterion’ WBIC of Watanabe (2013).”
A guide to Bayesian model selection for ecologists (2015, methodology): “Furthermore, unlike DIC, WAIC is valid in both hierarchical and mixture models (Watanabe 2013).”
Subspace Inference for Bayesian Deep Learning (2019, methodology): “Large Scale UCI Regression Datasets Here, on all models except skillcraft 9, like in Wilson et al. (2016), we use a feed forward network with five hidden layers: [1000, 1000, 500, 50, 2, 2], ReLU activations and heteroscedastic noise variance.”