CONSISTENCY OF CROSS VALIDATION FOR COMPARING REGRESSION PROCEDURES

By Yuhong Yang. 2007

187 citations recorded by the source corpus.

Theoretical developments on cross validation (CV) have mainly focused on selecting one among a list of finite-dimensional models (e.g., subset or order selection in linear regression) or selecting a smoothing parameter (e.g., bandwidth for kernel smoothing). However, little is known about consistency of cross validation when applied to compare between parametric and nonparametric methods or within nonparametric methods. We show that under some conditions, with an appropriate choice of data splitting ratio, cross validation is consistent in the sense of selecting the better procedure with probability approaching 1. Our results reveal interesting behavior of cross validation. When comparing two models (procedures) converging at the same nonparametric rate, in contrast to the parametric case, it turns out that the proportion of data used for evaluation in CV does not need to be dominating in size. Furthermore, it can even be of a smaller order than the proportion for estimation while not affecting the consistency property.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/25f2611ed47b22ba002c7ea882a99192e8adff69

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Catching up faster by switching sooner: a predictive approach to adaptive estimation with an application to the AIC–BIC dilemma (2012, background): “The only other procedure that we know of, for which somewhat related results have been shown, is a version of cross-validation that was proposed by Yang (2007a) to select between AIC and BIC in regression problems.”
Estimation Stability With Cross-Validation (ESCV) (2013, methodology): “A related instability measure is defined in Yuan and Yang (2005). It is a function of the size of the data perturbation, and is not normalized by the solution size as in ESCV , but instead by an estimate of the noise in the model.”
A survey of cross-validation procedures for model selection (2009, unknown): “Note that according to a simulation experiments, CV with averaging (that is, CV as usual) and CV with voting are equivalent at ﬁrst but not at second order, so that they can diﬀer when n is small (Yang, 2007).”
Cross-validation: what does it estimate and how well does it do it? (2021, methodology): “See supplementary Appendix H for a complementary analysis in the traditional p fixed, n → ∞ asymptotic regime and Yang (2007) and Wager (2020) for a related discussion.”
Multiple predicting K-fold cross-validation for model selection (2018, background): “Yang (2007) revealed that the sample size in the evaluation data does not have to dominate under some nonparametric models.”
Differentially private model selection with penalized and constrained likelihood (2016, background): “It is well known that, in general, cross-validation cannot produce consistent model selection even in the classical situation of low dimensional linear regression, unless the training sample ratio is chosen in a very unconventional way (Shao, 1993; Yang, 2007).”
A survey of Bayesian predictive methods for model assessment, selection and comparison (2012, result): “…there is not yet an asymptotic consistency result for the Bayesian cross-validation in model selection, but results from non-Bayesian literature (Yang, 2005, 2007; Arlot and Celisse, 2010) imply that the usual Bayesian LOO-CV would not be consistent, although an alternative Bayesian CV could be…”
Cross-Validation With Confidence (2017, background): “Early theoretical studies of cross-validation [26, 36, 35] indicate that, under a low-dimensional linear model, cross-validation cannot consistently select the correct model unless the training-testing split ratio tends to zero, which precludes both the leave-one-out and V-fold cross-validation.”
Sensitivity and Specificity of Information Criteria (2018, unknown): “For example, some researchers (Andrews and Currim, 2003; Fonseca and Cardoso, 2007; Yang and Yang, 2007) have suggested using An = 3 in expression (1) instead of 2.”
Parametric or nonparametric? A parametricness index for model selection (2011, methodology): “To assess performance of model selection criteria, pointwise asymptotic results (e.g., [17, 27, 39, 43, 46, 48–51, 54, 57, 61, 63, 67, 71, 74, 75]) have been established mostly in terms of either selection consistency or an asymptotic optimality.”
Quasi-oracle estimation of heterogeneous treatment effects (2017, background): “This phenomenon plays a key role in the literature on model selection via cross-validation (Yang, 2007). llllllllllllllllllllllllll l l l lllllllllllllllllllllllllllllllllllll l ll ll ll l ll l l l ll l lll ll lllll l l lll l l l l llll ll l l lll l l l lllll l l l llllllll l ll l ll l ll ll ll l l…”
Estimation of prediction error by using K-fold cross-validation (2011, methodology): “Cross-validation has also been used for model selection (for example, Li 1987; Shao 1993; Yang 2007), but it should be noted that a better estimate of prediction error does not necessarily lead to a better model selection criterion.”