Comparison of Bayesian predictive methods for model selection

By Juho Piironen, Aki Vehtari. Statistics and computing, 2015

319 citations recorded by the source corpus.

The goal of this paper is to compare several widely used Bayesian model selection methods in practical model selection problems, highlight their differences and give recommendations about the preferred approaches. We focus on the variable subset selection for regression and classification and perform several numerical experiments using both simulated and real world data. The results show that the optimization of a utility estimate such as the cross-validation (CV) score is liable to finding overfitted models due to relatively high variance in the utility estimates when the data is scarce. This can also lead to substantial selection induced bias and optimism in the performance evaluation for the selected model. From a predictive viewpoint, best results are obtained by accounting for model uncertainty by forming the full encompassing model, such as the Bayesian model averaging solution over the candidate models. If the encompassing model is too complex, it can be robustly simplified by the projection method, in which the information of the full model is projected onto the submodels. This approach is substantially less prone to overfitting than selection based on CV-score. Overall, the projection method appears to outperform also the maximum a posteriori model and the selection of the most probable variables. The study also demonstrates that the model selection can greatly benefit from using cross-validation outside the searching process both for guiding the model size selection and assessing the predictive performance of the finally selected model.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/59320e4b76f3ac03b07ae6e06781a485f8bbfb77

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Using Stacking to Average Bayesian Predictive Distributions (with Discussion) (2017, methodology): “For high-dimensional variable selection we recommend a projection predictive approach (Piironen and Vehtari, 2016, 2017a), which has a smaller variance in selection process due to the use of the encompassing model as a reference model and has better predictive performance due to making the…”
A Unifying Framework for Gaussian Process Pseudo-Point Approximations using Power Expectation Propagation (2016, background): “It would be interesting to explore distributional hyperparameter estimates (see e.g. Piironen and Vehtari, 2017) that employ these pseudo-point approximations.”
Model Averaging and Its Use in Economics (2017, methodology): “y better than single models (such as the model with highest posterior probability) in growth data. Ley and Steel (2009b) corroborate these ﬁndings, especially with a hyperprior on w, as used in (12). Piironen and Vehtari (2017) focus on model selection methods, but state that “From the predictive po”
Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC (2015, methodology): “Cross-validation and WAIC should not be used to select a single model among a large number of models due to a selection induced bias as demonstrated, for example, by Piironen and Vehtari (2016).”
Uncertainty Quantification for the Horseshoe (with Discussion) (2016, methodology): “In this case even for the most relevant variables most of the posterior mass can concentrate around zero, see for example Figure 9 in Piironen and Vehtari (2017c), which makes a reliable variable selection based on the posterior intervals challenging.”
The Median Probability Model and Correlated Variables (2018, methodology): “…used for distilling posterior evidence towards variable selection; Clyde et al. (2011); Feldkircher (2012); Garcia-Donato and Martinez-Beneito (2013); Ghosh (2015); Piironen and Vehtari (2017) and Drachal (2018) are some of the articles that have used and dis- cussed the performance of the MPM.”