Least squares after model selection in high-dimensional sparse models

By Alexandre Belloni, Victor Chernozhukov. 2009

336 citations recorded by the source corpus.

In this paper we study post-model selection estimators which apply ordinary least squares (ols) to the model selected by first-step penalized estimators, typically lasso. It is well known that lasso can estimate the non-parametric regression function at nearly the oracle rate, and is thus hard to improve upon. We show that ols post lasso estimator performs at least as well as lasso in terms of the rate of convergence, and has the advantage of a smaller bias. Remarkably, this performance occurs even if the lasso-based model selection “fails” in the sense of missing some components of the “true” regression model. By the “true” model we mean here the best s-dimensional approximation to the nonparametric regression function chosen by the oracle. Furthermore, ols post lasso estimator can perform strictly better than lasso, in the sense of a strictly faster rate of convergence, if the lasso-based model selection correctly includes all components of the “true” model as a subset and also achieves sufficient sparsity. In the extreme case, when lasso perfectly selects the “true” model, the ols post lasso estimator becomes the oracle estimator. An important ingredient in our analysis is a new sparsity bound on the dimension of the model selected by lasso which guarantees that this dimension is at most of the same order as the dimension of the “true” model. Our rate results are non-asymptotic and hold in both parametric and nonparametric models. Moreover, our analysis is not limited to the lasso estimator acting as selector in the first step, but also applies to any other estimator, for example various forms of thresholded lasso, with good rates and good sparsity properties. Our analysis covers both traditional thresholding and a new practical, data-driven thresholding scheme that induces maximal sparsity subject to maintaining a certain goodness-of-fit. The latter scheme has theoretical guarantees similar to those of lasso or ols post lasso, but it dominates these procedures as well as traditional thresholding in a wide variety of experiments. First arXiv version: December 2009.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/dd6c80c9931277700238714e088a6b75effae7cb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Double/debiased machine learning for treatment and structural parameters (2017, neutral): “See, among many others, Bickel et al. (2009), Bühlmann and van de Geer (2011), Belloni et al.”
Program evaluation with high-dimensional data (2013, methodology): “…et al. (2008); Bickel et al. (2009); Meinshausen and Yu (2009); Bach (2010); Huang et al. (2010); Belloni and Chernozhukov (2011); Kato (2011); Belloni et al. (2012); Belloni and Chernozhukov (2013); Belloni et al. (2013b); Belloni et al. (2013c); Caner and Zhang (2014); and the references therein.”
Identifying Latent Structures in Panel Data (2016, methodology): “In a totally different framework, Belloni and Chernozhukov (2013) study post-Lasso estimators which apply OLS to the model selected by first-step penalized estimators and show that the post-Lasso estimators perform at least as well as Lasso in terms of rate of convergence and have the advantage of…”
Lasso adjustments of treatment effect estimates in randomized experiments (2015, methodology): “[24] Belloni A, Chernozhukov V (2009) Least Squares After Model Selection in High-dimensional Sparse Models.”
Changes in Social Network Structure in Response to Exposure to Formal Credit Markets (2021, positive): “Because we only have 150 observations but many controls (up to 18 controls and their interactions before adding the ﬁxed effects), we use the double post-LASSO (DPL) procedure (Belloni and Chernozhukov, 2009; Belloni et al ., 2014a, 2014b) to select the controls.”
Uniform post-selection inference for least absolute deviation regression and other Z-estimation problems (2013, background): “where λ and Γ̂ are the penalty level and data-driven penalty loadings described in [2] (restated in Appendix D), or the associated post-model selection estimator (Post-Lasso) [4, 2] defined as θ̃ ∈ argmin θ { En[(d − x′θ)2] : θj = 0 if θ̂j = 0 } and set v̂i = di − xiθ̃.”
Post-Selection and Post-Regularization Inference in Linear Models with Many Controls and Instruments (2015, background): “The theoretical results do not depend on whether the Lasso estimator or the Post-Lasso estimator of Belloni and Chernozhukov (2013) is used.”
Taming the Factor Zoo: A Test of New Factors (2019, neutral): “(2009), Meinshausen and Yu (2009), Tibshirani (2011), Wainwright (2009), Zhang and Huang (2008), Belloni and Chernozhukov (2013). For the post-selection-inference method, see, for example, Belloni et al.”
Two-Step Lasso Estimation of the Spatial Weights Matrix (2015, methodology): “This is consistent with [30,31], who show that the post-Lasso often performs as least as good as the Lasso.”
Statistical inference for model parameters in stochastic gradient descent (2016, methodology): “(2013) and under certain assumptions, Belloni and Chernozhukov (2013) proved that S ⊆ Ŝ and |Ŝ\S| .”
The Dynamics of Non-Performing Loans during Banking Crises (2019, positive): “We use the “post rigorous least absolute shrinkage and selection operator” (“post-r-lasso”; Belloni et al., 2012; Belloni and Chernozhukov, 2013) model selection approach to determine the most informative combination of NPL predictors.”