Cross-validation pitfalls when selecting and assessing regression and classification models
By Damjan Krstajic, Ljubomir J. Buturovic, David E. Leahy, Simon D. A. Thomas. Journal of Cheminformatics, 2014
961 citations recorded by the source corpus.
BackgroundWe address the problem of selecting and assessing classification and regression models using cross-validation. Current state-of-the-art methods can yield models with high variance, rendering them unsuitable for a number of practical applications including QSAR. In this paper we describe and evaluate best practices which improve reliability and increase confidence in selected models. A key operational component of the proposed methods is cloud computing which enables routine use of previously infeasible approaches.MethodsWe describe in detail an algorithm for repeated grid-search V-fold cross-validation for parameter tuning in classification and regression, and we define a repeated nested cross-validation algorithm for model assessment. As regards variable selection and parameter tuning we define two algorithms (repeated grid-search cross-validation and double cross-validation), and provide arguments for using the repeated grid-search in the general case.ResultsWe show results of our algorithms on seven QSAR datasets. The variation of the prediction performance, which is the result of choosing different splits of the dataset in V-fold cross-validation, needs to be taken into account when selecting and assessing classification and regression models.ConclusionsWe demonstrate the importance of repeating cross-validation when selecting an optimal model, as well as the importance of repeating nested cross-validation when assessing a prediction error.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/acfb0cdb75695064148b427005d88b6aa51994f6
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- J. Westfall
- A. Casson
- Tal Yarkoni
- Emma Gowen
- Knut Baumann
- Ellen Poliakoff
- Désirée Baumann
- Andrius Vabalas
Citations in the corpus
- Choosing prediction over explanation in psychology: Lessons from machine learning (2017, background): “…should ideally embed any model selection steps into the cross-validation procedure itself, rather than simply picking the model that gives the best cross-validated result (for further explanation, see Cawley & Talbot, 2010; Krstajic, Buturovic, Leahy, & Thomas, 2014; Varma & Simon, 2006).”
- Machine learning algorithm validation with a limited sample size (2019, background): “One possible solution which avoids pooling training and validation data, but at the same time is economical (allows all the data to be used for training and reuse for validation) is Nested CV [7, 11], Fig 2C.”
- Reliable estimation of prediction errors for QSAR models under model uncertainty using double cross-validation (2014, positive): “Although many successful applications of double cross-validation have been published in recent years [7,9,10,12,14,16,18,19,51-56], there is still some reluctance to use double cross-validation.”
- Development of Artificial Intelligence for Determining Major Depressive Disorder Based on Resting-State EEG and Single-Pulse Transcranial Magnetic Stimulation-Evoked EEG Indices (2024, methodology): “120 cases in the present study, we used five-fold outer loop double cross-validation [29,30] to evaluate the estimation performance of We applied machine learning to these feature sets to create a model that can discriminate whether the participant has depression or not based on EEG data. the model.”