On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation

By Gavin C. Cawley, Nicola L. C. Talbot. 2010

2,364 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Statistical Learning Theory

Citations in the corpus

Learning kernel logistic regression in the presence of class label noise (2014, methodology): “For experiments where cross-validation (CV) is needed, we adopt the ‘best practice’ described in [8], where model selection is seen as part of the learning and 5-folds CV is performed independently on each split of the data.”
Predictive Multiplicity in Classification (2019, methodology): “Given these hyperparameters, however, we still deploy the model that optimizes performance on the training data for these hyperparameters [9].”
Variational Dropout Sparsifies Deep Neural Networks (2017, methodology): “The Relevance Vector Machine (RVM, (Tipping, 2001) (Cawley, 2010).”
On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice (2020, background): “This is similar to over-fitting issues with ML models that occur when a model is closely fit to a finite number of known data points but is unfit to unseen data [128].”
Prediction and behavioral analysis of travel mode choice: A comparison of machine learning and logit models (2020, background): “However, SVM can be very sensitive to overﬁtting especially for nonlinear kernels (Cawley and Talbot, 2010).”
3FabRec: Fast Few-Shot Face Alignment by Reconstruction (2019, background): “are specifically optimized on the particular datasets that are tested, increasing the potential of overfitting to that dataset [7].”
The need for uncertainty quantification in machine-assisted medical decision making (2019, background): “Some approaches, such as attempts to empirically learn generalizable patterns with insertion of random noise 19,20 , or the use of cross-validation to determine the progression of generalizable learning, move us forward in this problem space, while still carrying the problem of overfitting 21 .”
A Minimax Theory for Adaptive Data Analysis (2016, methodology): “In Dwork et al. (2015b), T is the class of statistical queries that are uniformly bounded.”
Machine learning–XGBoost analysis of language networks to classify patients with epilepsy (2017, methodology): “The validation strategy is commonly used to prevent overfitting and to have a good assessment of model validity [44].”
Detection of COVID-19 Infection from Routine Blood Exams with Machine Learning: A Feasibility Study (2020, methodology): “The models mentioned above have been trained, and evaluated, through a nested cross validation [19, 9] procedure.”
A Support Vector Machine for Landslide Susceptibility Mapping in Gangwon Province, Korea (2017, methodology): “The drawback of using SVMs is that they only cover the determination of the parameters for a given value of the regularization and kernel parameters, and choice of kernel [24].”