Cross-Validation With Confidence
By Jing Lei. 2017
115 citations recorded by the source corpus.
Abstract Cross-validation is one of the most popular model and tuning parameter selection methods in statistics and machine learning. Despite its wide applicability, traditional cross-validation methods tend to overfit, due to the ignorance of the uncertainty in the testing sample. We develop a novel statistically principled inference tool based on cross-validation that takes into account the uncertainty in the testing sample. This method outputs a set of highly competitive candidate models containing the optimal one with guaranteed probability. As a consequence, our method can achieve consistent variable selection in a classical linear regression setting, for which existing cross-validation methods require unconventional split ratios. When used for tuning parameter selection, the method can provide an alternative trade-off between prediction accuracy and model interpretability than existing variants of cross-validation. We demonstrate the performance of the proposed method in several simulated and real data examples. Supplemental materials for this article can be found online.
Subfields: Statistics and Probability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Statistics and Probability
- Robert Tibshirani
- Trevor J. Hastie
- Jonathan E. Taylor
- Stephen Bates
- Stefan Wager
- Cynthia Rudin
- Jianqing Fan
- Peter Buhlmann
Citations in the corpus
- Forward stability and model path selection (2021, neutral): “These procedures also rely heavily on exhaustive search or backward elimination-type procedures, making them computationally daunting and potentially infeasible in high-dimensional settings without a valid variable screening tool (Lei, 2020; Jiang et al., 2008).”
- Cross-validation: what does it estimate and how well does it do it? (2021, methodology): “To this end, Varma and Simon (2006) and Tibshirani and Tibshirani (2009) suggest a bias-correction for the model selected by cross-validation, Lei (2019) shows how to return a confidence set for the best model parameters, and Yang (2007); Wager (2020) show that for CV, comparing two models is a…”
- The Role of AI in Hospitals and Clinics: Transforming Healthcare in the 21st Century (2024, methodology): “To address this, techniques like cross-validation are employed [151].”