Reliable Accuracy Estimates from k-Fold Cross Validation

By Tzu-Tsung Wong, P. Yeh. IEEE Transactions on Knowledge and Data Engineering, 2020

883 citations recorded by the source corpus.

It is popular to evaluate the performance of classification algorithms by <italic>k</italic>-fold cross validation. A reliable accuracy estimate will have a relatively small variance, and several studies therefore suggested to repeatedly perform <italic>k</italic>-fold cross validation. Most of them did not consider the correlation among the replications of <italic>k</italic>-fold cross validation, and hence the variance could be underestimated. The purpose of this study is to explore whether <italic>k</italic>-fold cross validation should be repeatedly performed for obtaining reliable accuracy estimates. The dependency relationships between the predictions of the same instance in two replications of <italic>k</italic>-fold cross validation are first analyzed for <italic>k</italic>-nearest neighbors with <inline-formula><tex-math notation="LaTeX">$k= 1$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>k</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="wong-ieq1-2912815.gif"/></alternatives></inline-formula>. Then, statistical methods are proposed to test the strength of the dependency level between the accuracy estimates resulting from two replications of <italic>k</italic>-fold cross validation. The experimental results on 20 data sets show that the accuracy estimates obtained from various replications of <italic>k</italic>-fold cross validation are generally highly correlated, and the correlation will be higher as the number of folds increases. The <italic>k</italic>-fold cross validation with a large number of folds and a small number of replications should be adopted for performance evaluation of classification algorithms.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/bd9a698efc195e82ed2a2bce408ae6fd0c8ebf56