Pivotal estimation via square-root Lasso in nonparametric regression

By Alexandre Belloni, Victor Chernozhukov, Lie Wang. 2011

152 citations recorded by the source corpus.

We propose a self-tuning $\sqrt{\mathrm {Lasso}} $ method that simultaneously resolves three important practical problems in high-dimensional regression analysis, namely it handles the unknown scale, heteroscedasticity and (drastic) non-Gaussianity of the noise. In addition, our analysis allows for badly behaved designs, for example, perfectly collinear regressors, and generates sharp bounds even in extreme cases, such as the infinite variance case and the noiseless case, in contrast to Lasso. We establish various nonasymptotic bounds for $\sqrt{\mathrm {Lasso}} $ including prediction norm rate and sparsity. Our analysis is based on new impact factors that are tailored for bounding prediction norm. In order to cover heteroscedastic non-Gaussian noise, we rely on moderate deviation theory for self-normalized sums to achieve Gaussian-like results under weak conditions. Moreover, we derive bounds on the performance of ordinary least square (ols) applied to the model selected by $\sqrt{\mathrm {Lasso}} $ accounting for possible misspecification of the selected model. Under mild conditions, the rate of convergence of ols post $\sqrt{\mathrm {Lasso}} $ is as good as $\sqrt{\mathrm {Lasso}} $’s rate. As an application, we consider the use of $\sqrt{\mathrm {Lasso}} $ and ols post $\sqrt{\mathrm {Lasso}} $ as estimators of nuisance parameters in a generic semiparametric problem (nonlinear moment condition or $Z$-problem), resulting in a construction of $\sqrt{n}$-consistent and asymptotically normal estimators of the main parameters.

Subfields: Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Double/De-Biased Machine Learning of Global and Local Parameters Using Regularized Riesz Representers (2018, methodology): “For instance, Belloni et al. (2014) employ these tools to show that, for the bounded design case ‖b‖∞ ≤ C, λ can be chosen as in the Gaussian error case, provided that errors follow t(2 + δ) distribution (having above 2 bounded moments), and get the error bounds similar to the Gaussian case.”
Double/Debiased Machine Learning for Treatment and Causal Parameters (2016, methodology): “plement this important classic work as well as the rapidly expanding body of work on estimation under unconfoundedness using ML methods; see, among others, Athey et al. (2016), Belloni et al. (2017), Belloni et al. (2014), Farrell (2015), and Imai and Ratkovic (2013). We specically consider estimati”
On the Prediction Performance of the Lasso (2014, background): “≤ ¯ckβ∗k1 is included in the deﬁnition of C0(T,¯c). On the one hand, more generally, one can deﬁne the set C0(T,¯c, β) = {u ∈ Rp : kβ+uk1−kβ Tk1 < (¯c−1)(¯c+1)−1kuk1}, contained in the one used in [BCW13], and let T,¯c, β be the 1−δ quantile of η¯ = sup u∈C0(T,c,¯ β) |ξ⊤Xu| σ∗kXuk2 . Then the ris”
Program evaluation and causal inference with high-dimensional data (2013, methodology): “…For comparison, we also report estimates of the eligibility effect from the linear model without selection and with selection using the approach of Belloni et al. (2014a) and estimates of the participation effect from linear instrumental variables estimation without selection and with selection…”
Learning L2 Continuous Regression Functionals via Regularized Riesz Representers (2018, methodology): “The influence function of an L2 continuous functional of a regression limit is the product of the regression residual with the Riesz representer (RR) of the functional derivative, as shown in Newey (1994). Therefore, DML of regression functionals requires a machine learner of the RR. We provide here”
Linear and conic programming estimators in high dimensional errors‐in‐variables models (2014, background): “A discussion of the case of repeated regressors is given in Belloni et al. (2014). The prediction sensitivity is closely related to the identifiability factor defined in Chernozhukov et al. (2013). Lemma 8 in section 3 of the on-line supplementary material shows that κpr.”
Asymptotically Honest Confidence Regions for High Dimensional Parameters by the Desparsified Conservative Lasso (2014, methodology): “Statistical inference in the sense of constructing tests and confidence bands was to the best of our knowledge considered for the first time in a seminal series of papers by Belloni et al. (2010, 2012, 2011b, 2014, 2011a).”
Program evaluation with high-dimensional data (2013, methodology): “To the best of our knowledge, Belloni et al. (2010) and Belloni et al. (2012) were the first to use the orthogonality (1.2) to expressly address the question of the uniform post-selection inference without imposing the “beta-min” conditions, either in high-dimensional settings with p n or in…”
Double/debiased machine learning for treatment and structural parameters (2017, unknown): “See, among many others, Bickel et al. (2009), Bühlmann and van de Geer (2011), Belloni et al. (2011), Belloni and Chernozhukov (2011), Belloni et al. (2012), and Belloni and Chernozhukov (2013) for `1-penalized and related methods in a variety of sparse models; Kozbur (2016) for forward selection…”
Two-Step Lasso Estimation of the Spatial Weights Matrix (2015, background): “The main advantage of the square-root Lasso is that the optimal penalty level does not depend on the unknown error variance [56,57].”
Valid Post-Selection Inference in High-Dimensional Approximately Sparse Quantile Regression Models (2013, methodology): “The uniform validity of 1-penalized methods for a continuum of problems (indexed by T in our case) has been established for quantile regression in Belloni andChernozhukov (2011) and for least squares in Belloni et al. (2013b).”
Double/de-biased machine learning using regularized Riesz representers (2018, methodology): “For instance, Tsybakov (2012) and Belloni et al. (2014b) give examples of Sobolev and rearranged Sobolev balls, respectively, as the function classes and elements of the Fourier basis as the dictionary b , in which sparse approximations have small errors.”
Uniform post-selection inference for least absolute deviation regression and other Z-estimation problems (2013, methodology): “Examples of functions hj that permit such estimation by sparse methods include the standard Sobolev spaces as well as more g eneral rearranged Sobolev spaces [7, 6].”