Bias in error estimation when using cross-validation for model selection

By Sudhir Varma, Richard M. Simon. BMC Bioinformatics, 2006

2,007 citations recorded by the source corpus.

BackgroundCross-validation (CV) is an effective method for estimating the prediction error of a classifier. Some recent articles have proposed methods for optimizing classifiers by choosing classifier parameter values that minimize the CV error estimate. We have evaluated the validity of using the CV error estimate of the optimized classifier as an estimate of the true error expected on independent data.ResultsWe used CV to optimize the classification parameters for two kinds of classifiers; Shrunken Centroids and Support Vector Machines (SVM). Random training datasets were created, with no difference in the distribution of the features between the two classes. Using these "null" datasets, we selected classifier parameter values that minimized the CV error estimate. 10-fold CV was used for Shrunken Centroids while Leave-One-Out-CV (LOOCV) was used for the SVM. Independent test data was created to estimate the true error. With "null" and "non null" (with differential expression between the classes) data, we also tested a nested CV procedure, where an inner CV loop is used to perform the tuning of the parameters while an outer CV is used to compute an estimate of the error.The CV error estimate for the classifier with the optimal parameters was found to be a substantially biased estimate of the true error that the classifier would incur on independent data. Even though there is no real difference between the two classes for the "null" datasets, the CV error estimate for the Shrunken Centroid with the optimal parameters was less than 30% on 18.5% of simulated training data-sets. For SVM with optimal parameters the estimated error rate was less than 30% on 38% of "null" data-sets. Performance of the optimized classifiers on the independent test set was no better than chance.The nested CV procedure reduces the bias considerably and gives an estimate of the error that is very close to that obtained on the independent testing set for both Shrunken Centroids and SVM classifiers for "null" and "non-null" data distributions.ConclusionWe show that using CV to compute an error estimate for a classifier that has itself been tuned using CV gives a significantly biased estimate of the true error. Proper use of CV for estimating true error of a classifier developed using a well defined algorithm requires that all steps of the algorithm, including classifier parameter tuning, be repeated in each CV loop. A nested CV procedure provides an almost unbiased estimate of the true error.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/bf75f2555e8485d536ac217068b7963de3d6fa02

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Molecular Biology

Citations in the corpus

A survey of Bayesian predictive methods for model assessment, selection and comparison (2012, methodology): “Even if the procedure is based on an approach giving unbiased expected utility estimates for any particular model, the data-fitted model selection procedure causes the expected utility estimate of the selected model to be biased (see, for example, Stone, 1974; Rencher and Pun, 1980; Reunanen, 2003; ”
TSception: Capturing Temporal Dynamics and Spatial Asymmetry From EEG for Emotion Recognition (2021, methodology): “For a more robust evaluation of the networks, nested cross-validation [42] is further adopted with the inner loop being 5-fold cross-validation and the outer loop being leave-one-trial-out cross-validation.”
Cross-Validation, Shrinkage and Variable Selection in Linear Regression Revisited (2013, background): “As pointed out by [15], this is a bad thing to do.”
Radiomics with artificial intelligence: a practical guide for beginners. (2019, methodology): “For creating such unseen data sets, although the hold-out technique seems to be the most appropriate internal validation method, there is also nested cross-validation technique that is primarily used for this purpose and might give similar estimates to an independent validation (70).”
Detection of COVID-19 Infection from Routine Blood Exams with Machine Learning: A Feasibility Study (2020, methodology): “This was tackled by performing nested cross-validation in order to control for bias [38], and by”
A Fair Comparison of Graph Neural Networks for Graph Classification (2019, methodology): “When a test set is not explicitly given, a common way to proceed is to use k-fold Cross Validation (CV) (Stone, 1974; Varma & Simon, 2006; Cawley & Talbot, 2010). k -fold CV uses k different training/test splits to estimate the generalization performance of a model; for each partition, an internal…”
Batch Effect Confounding Leads to Strong Bias in Performance Estimates Obtained by Cross-Validation (2014, methodology): “The performance of each classifier is evaluated on the training data by the means of nested cross-validation [12,18–20] as well as on external data.”
Machine learning algorithm validation with a limited sample size (2019, methodology): “Varma and Simon [7] suggest that Nested CV provides almost unbiased performance estimates.”
Cross-validation: what does it estimate and how well does it do it? (2021, methodology): “To this end, Varma and Simon (2006) and Tibshirani and Tibshirani (2009) suggest a bias-correction for the model selected by cross-validation, Lei (2019) shows how to return a conﬁdence set for the best model parameters, and Yang (2007); Wager (2020) show that for CV, comparing two models is a…”
Multi-Objective Parameter Selection for Classifiers (2012, background): “Varma and Simon (2006) suggest a nested cross-validation for the parameter selection.”
Cross-validation pitfalls when selecting and assessing regression and classification models (2014, methodology): “It is important to note that both Stone [2] and Varma and Simon [9] use leave-one-out cross-validation, while we use V-fold cross-validation.”
Tree species classification using structural features derived from terrestrial laser scanning (2020, methodology): “5) was applied to determine the accuracy of each classifier using fully independent data (Varma and Simon, 2006).”
Toward more realistic drug–target interaction predictions (2014, positive): “In addition to the aforementioned CV issues, selection of hyper-parameters, such as the regularization [20–23] that if CV estimate is used as a parameter selection tool, the same estimate is no longer reliable for estimating the prediction performance of the model trained with the optimal…”