Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap

By Ji-Hyun Kim. Computational Statistics & Data Analysis, 2009

748 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/07f04c23147fd83e2274fe01374380a02798d60a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Applied Predictive Modeling (2013, background): “Research (Molinaro 2005; Kim 2009) indicates that repeating k-fold crossvalidation can be used to effectively increase the precision of the estimates while still maintaining a small bias.”
On The Stability of Interpretable Models (2018, neutral): “Variability of the estimator is accounted for by adopting repetitions [35].”
Resampling Methods for Meta-Model Validation with Recommendations for Evolutionary Computation (2012, background): “Its main drawback is the difficult combination with tuning (see comment 10) and that it might result in an optimistic bias when more complex models are considered (Kim, 2009; Molinaro et al., 2005).”
Reliable Accuracy Estimates from k-Fold Cross Validation (2020, background): “Vanwinckelen and Blockeel [5] and Kim [6] also pointed out that repeating k -fold cross validation can stabilize the variability of accuracy estimates.”
Complete Search for Feature Selection in Decision Trees (2019, positive): “Variability of the estimator is accounted for by adopting repetitions (Kim, 2009).”
Predicting Diabetes Mellitus With Machine Learning Techniques (2018, methodology): “Model Validation In many studies, authors often used two validation methods, namely hold-out method and k-fold cross validation method, to evaluate the capability of the model (Kohavi, 1995; Bengio and Grandvalet, 2005; Kim, 2009; Chen et al., 2016; Refaeilzadeh et al., 2016; Yang et al., 2016, 2018”
Land cover and land use classification performance of machine learning algorithms in a boreal landscape using Sentinel-2 data (2019, methodology): “The entire procedure was then repeated using a prede ﬁ ned set of repeats in order to reduce model variance (Kim 2009; Kuhn and Johnson 2013a).”
An Empirical Comparison of Model Validation Techniques for Defect Prediction Models (2017, methodology): “For example, some studies conduct simulation experiments [11], [24], [28], [32], [45], [55], while others use empiri-”
Subtree Replacement in Decision Tree Simplification (2012, methodology): “We ameliorate possible variability of the estimator by adopting repetition of cross validation, as discussed in [16].”
Fast nonlinear gravity inversion in spherical coordinates with application to the South American Moho (2017, methodology): “Here, we will adopt the hold-out method of cross-validation (Kim 2009).”
Batch Effect Confounding Leads to Strong Bias in Performance Estimates Obtained by Cross-Validation (2014, background): “It has been shown that in general, the bias of these methods can be quite different [42].”
Mean Birds: Detecting Aggression and Bullying on Twitter (2017, methodology): “data mining toolkit and repeated (10 times) 10-fold cross validation [27], providing the relevant standard deviation (STD).”