A survey of cross-validation procedures for model selection

By Sylvain Arlot, Alain Celisse. 2009

3,932 citations recorded by the source corpus.

Used to estimate the risk of an estimator or to perform model selection, cross-validation is a widespread strategy because of its simplicity and its apparent universality. Many results exist on the model selection performances of cross-validation procedures. This survey intends to relate these results to the most recent advances of model selection theory, with a particular emphasis on distinguishing empirical statements from rigorous theoretical results. As a conclusion, guidelines are provided for choosing the best cross-validation procedure according to the particular features of the problem in hand.

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/17d00ec9a7537004ee83cce5baa57c4d6f1b808c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

cutpointr: Improved Estimation and Validation of Optimal Cutpoints in R (2020, methodology): “Popular validation methods include training / test splits, k-fold cross-validation, and diﬀerent variations of the bootstrap (Altman and Royston 2000; Arlot and Celisse 2010).”
Big Data Creates New Opportunities for Materials Research: A Review on Methods and Applications of Machine Learning for Materials Design (2019, neutral): “There are several cross-validation strategies, among which the k -fold cross-validation method [52] is very popular.”
Improving the Prediction Accuracy of Soil Mapping through Geostatistics (2012, neutral): “To validate a predictive model and choose appropriate algorithm, Cross-Validation (CV) which is a statistical method, has been used [35].”
Stock Market Prediction through Technical and Public Sentiment Analysis (2015, methodology): “A more principled approach for time series cross validation is forward chaining [7].”
A review of change point detection methods (2018, methodology): “In [92, 112] Opt is applied several times on one training signal to find a suitable β that can be used to segment new signals, An adapted cross-validation procedure is described in [3] and [25] proposes another method based on the slope heuristics.”
On Optimal Generalizability in Parametric Learning (2017, methodology): “Related work: A main application of cross validation (see [1] for a recent survey) is in model selection [2–4].”
A K-fold Averaging Cross-validation Procedure (2015, background): “Substantial practical and theoretical results of traditional and newer CV approaches were summarised in a review paper by Arlot and Celisse (2010).”
Time-aware recommender systems: a comprehensive survey and analysis of existing evaluation protocols (2013, methodology): “Although this method has showed the lowest variability in results in generic prediction problems (Arlot and Celisse 2010), its high computational cost (the algorithms must be trained and evaluated |M | times) makes it unfeasible in many situations. Cremonesi and Turrin (2009) applied this method.”
Forecasting: theory and practice (2020, methodology): “ndomly into ksubsets, and then use each of the ksubsets to evaluate a model that has been estimated on the remaining subsets. An excellent overview of diﬀerent cross-validation techniques is given by Arlot and Celisse (2010). Despite its popularity in many ﬁelds, the use of CV in time series forecas”
Approximate Bayesian Computation (2013, methodology): “Beyond that, cross-validation techniques [36] and predictive checks [37,38] represent promising future strategies to evaluate the stability and out-of-sample predictive validity of ABC inferences.”
Assessing transferability of ecological models: an underappreciated aspect of statistical validation (2012, background): “Model validation and model selection are two sides of the same coin; it is reasonable to rank models based on their validated predictive performance and select the best performing model or a confidence set of good performing models (Arlot & Celisse 2010).”
Estimation of Total Nitrogen Content in Forage Maize (Zea mays L.) Using Spectral Indices: Analysis by Random Forest (2020, methodology): “This method has been widely used to validate machine learning models such as RF [12,28,29,52–55] and consists of dividing the entire dataset into two subsets; the first one is to train or adjust the model and while the second one is to validate it [56].”
Cross-validation pitfalls when selecting and assessing regression and classification models (2014, background): “In the literature this issue is known as the cross-validation bias [26].”
Reliable estimation of prediction errors for QSAR models under model uncertainty using double cross-validation (2014, neutral): “There are various kinds of cross-validation which split the original data differently into construction and validation data [28,47,48].”