A survey of Bayesian predictive methods for model assessment, selection and comparison

By Aki Vehtari, Janne Ojanen. 2012

373 citations recorded by the source corpus.

To date, several methods exist in the statistical literature for model assessment, which purport themselves specifically as Bayesian predic- tive methods. The decision theoretic assumptions on which these methods are based are not always clearly stated in the original articles, however. The aim of this survey is to provide a unified review of Bayesian predictive model assessment and selection methods, and of methods closely related to them. We review the various assumptions that are made in this context and discuss the connections between different approaches, with an emphasis on how each method approximates the expected utility of using a Bayesian model for the purpose of predicting future data.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/62f38ace91bcd89f32cd64b5e74cf85937f1a57c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

A guide to Bayesian model selection for ecologists (2015, methodology): “In essence, these scoring rule characteristics guarantee that the predictive score adheres to the chosen model and data (Vehtari and Ojanen 2012, Gelman et al. 2014b).”
Sparsity information and regularization in the horseshoe and other shrinkage priors (2017, neutral): “Still, it is worthwhile to compare the results to a model with a relatively loose prior on c using standard model assessment techniques (Vehtari and Ojanen, 2012; Vehtari, Gelman and Gabry, 2017) to get an indication if the slab width was chosen to be too restrictive.”
Statistical Model Criticism using Kernel Two Sample Tests (2015, methodology): “For a recent survey of Bayesian methods for model assessment, selection and comparison see [35] which phrases many techniques as estimates of the utility of a model.”
Between the Devil and the Deep Blue Sea: Tensions Between Scientific Judgement and Statistical Model Selection (2018, methodology): “In the statistical context, predictive generalisation performance is typically assessed with respect to test data sampled from the same process as the training data (e.g. Vehtari and Ojanen 2012).”
Comparison of Bayesian predictive methods for model selection (2015, methodology): “The review of Vehtari and Ojanen (2012) being qualitative, our contribution is to compare many of the diﬀerent methods quantitatively in practical model selection problems, discuss the diﬀerences, and give recommendations about the preferred approaches.”
Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC (2015, methodology): “See Vehtari and Lampinen (2002), Celisse and Arlot (2010), and Vehtari and Ojanen (2012) for further discussion of these points.”
A NICER View of PSR J0030+0451: Millisecond Pulsar Parameter Estimation (2019, methodology): “We consider other measures such as the expected posterior utility (i.e., power for future statistical falsification) beyond the scope of this work (Vehtari & Ojanen 2012).”
Understanding predictive information criteria for Bayesian models (2013, methodology): “There are extensions of AIC which have an adjustment related to the eﬀective number of parameters (see Vehtari and Ojanen, 2012, section 5.5, and references therein) but these are seldom used due to stability problems and computational diﬃculties, issues that have motivated the construction of the…”
Using Stacking to Average Bayesian Predictive Distributions (with Discussion) (2017, methodology): “In Bayesian context, the natural target for prediction is to find a predictive distribution that is close to the true data generating distribution (Gneiting and Raftery, 2007; Vehtari and Ojanen, 2012).”