Estimation Stability With Cross-Validation (ESCV)

By Chinghway Lim, Bin Yu. 2013

114 citations recorded by the source corpus.

Cross-validation (CV) is often used to select the regularization parameter in high-dimensional problems. However, when applied to the sparse modeling method Lasso, CV leads to models that are unstable in high-dimensions, and consequently not suited for reliable interpretation. In this article, we propose a model-free criterion ESCV based on a new estimation stability (ES) metric and CV. Our proposed ESCV finds a smaller and locally ES-optimal model smaller than the CV choice so that it fits the data and also enjoys estimation stability property. We demonstrate that ESCV is an effective alternative to CV at a similar easily parallelizable computational cost. In particular, we compare the two approaches with respect to several performance measures when applied to the Lasso on both simulated and real datasets. For dependent predictors common in practice, our main finding is that ESCV cuts down false positive rates often by a large margin, while sacrificing little of true positive rates. ESCV usually outperforms CV in terms of parameter estimation while giving similar performance as CV in terms of prediction. For the two real datasets from neuroscience and cell biology, the models found by ESCV are less than half of the model sizes by CV, but preserves CV's predictive performance and corroborates with subject knowledge and independent work. We also discuss some regularization parameter alignment issues that come up in both approaches. Supplementary materials are available online.

Subfields: Molecular Biology, Pediatrics, Perinatology and Child Health, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

In Pediatrics, Perinatology and Child Health

In Statistics and Probability

Citations in the corpus

Iterative random forests to discover predictive and stable high-order interactions (2017, background): “We note that a principled approach for selecting the total number of iterations K can be formulated in terms of estimation stability with cross validation (ESCV) (Lim and Yu, 2015), which would balance trade-offs between interpretability and predictive accuracy.”
Group bound: confidence intervals for groups of variables in sparse high dimensional regression without assumptions on the design (2013, background): “…stability of estimated sparse solutions has sometimes been analysed and exploited for better structure discovery (Meinshausen and Bühlmann, 2010; Shah and Samworth, 2013; Lim and Yu, 2013), formal significance tests were provided in Wasserman and Roeder (2009) and Meinshausen et al. (2009).”
Cross-Validation With Confidence (2017, methodology): “[22] use a stability-based criterion, which is the ratio between the empirical variance of the fitted vectors and the squared `2-norm of the average fitted vector.”
A random forest based approach for predicting spreads in the primary catastrophe bond market (2020, methodology): “Various ways to measure the stability of algorithmic results have been presented in Turney (1995), Lange et al. (2004), Ntoutsi et al. (2008), Lim & Yu (2016) and Philipp et al. (2018).”
Definitions, methods, and applications in interpretable machine learning (2019, background): “For example, if the set of sparse parameters changes due to small perturbations in the data set, the coefficients should not be interpreted (30).”
Model Selection Techniques: An Overview (2018, background): “Considerations along these lines also lead to stabilized variable selection methods [81], [83], [84].”
Communication-Avoiding Optimization Methods for Distributed Massive-Scale Sparse Inverse Covariance Estimation (2017, methodology): “Furthermore, the running time required to compute the CONCORD estimates across a grid of tuning parameters, as in resampling methods such as cross-validation, the bootstrap, and stability selection [37, 33], would be prohibitive.”
Selective Sequential Model Selection (2015, background): “Crossvalidation, another general-purpose algorithm for tuning parameter selection, targets out-of-sample error and tends to select many noise variables when the signal is sparse (e.g. Lim and Yu, 2015).”