Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso

By Trevor J. Hastie, Robert Tibshirani. 2017

216 citations recorded by the source corpus.

In exciting new work, Bertsimas et al. (2016) showed that the classical best subset selection problem in regression modeling can be formulated as a mixed integer optimization (MIO) problem. Using recent advances in MIO algorithms, they demonstrated that best subset selection can now be solved at much larger problem sizes that what was thought possible in the statistics community. They presented empirical comparisons of best subset selection with other popular variable selection procedures, in particular, the lasso and forward stepwise selection. Surprisingly (to us), their simulations suggested that best subset selection consistently outperformed both methods in terms of prediction accuracy. Here we present an expanded set of simulations to shed more light on these comparisons. The summary is roughly as follows: (a) neither best subset selection nor the lasso uniformly dominate the other, with best subset selection generally performing better in high signal-to-noise (SNR) ratio regimes, and the lasso better in low SNR regimes; (b) best subset selection and forward stepwise perform quite similarly throughout; (c) the relaxed lasso (actually, a simplified version of the original relaxed estimator defined in Meinshausen, 2007) is the overall winner, performing just about as well as the lasso in low SNR scenarios, and as well as best subset selection in high SNR scenarios.

Subfields: Statistics and Probability, Mathematical Optimization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

In Mathematical Optimization

Citations in the corpus

Forward stability and model path selection (2021, positive): “Note that the provided RTE is slightly different from that presented in Hastie et al. (2017), which opts for a ratio of the expected test errors.”
Getting Better from Worse: Augmented Bagging and a Cautionary Tale of Variable Importance (2020, methodology): “The form of this covariance corresponds to that utilized frequently in the recent work by Mentch and Zhou (2019) and to the ‘beta-type 2’ setup utilized in Hastie et al. (2017).”
A Critical Review of LASSO and Its Derivatives for Variable Selection Under Dependence Among Covariates (2020, methodology): “See Hastie et al. (2017) for a comparison of this procedure with more current methods.”
Outlier detection using distributionally robust optimization under the Wasserstein metric (2017, methodology): “idation set. Using MAD as a selection criterion serves to hedge against the potentially large noise in the validation samples. As to the range of values for the tuned parameters, we borrow ideas from Hastie et al. (2017), where the LASSO was tuned over 50 values ranging from λm = kX′yk∞ to a small f”