Debiasing the lasso: Optimal sample size for Gaussian designs

By Adel Javanmard, Andrea Montanari. 2015

207 citations recorded by the source corpus.

Performing statistical inference in high-dimension is an outstanding challenge. A major source of difficulty is the absence of precise information on the distribution of high-dimensional estimators. Here, we consider linear regression in the high-dimensional regime $p\gg n$. In this context, we would like to perform inference on a high-dimensional parameters vector $\theta^*\in{\mathbb R}^p$. Important progress has been achieved in computing confidence intervals for single coordinates $\theta^*_i$. A key role in these new methods is played by a certain debiased estimator $\hat{\theta}^{\rm d}$ that is constructed from the Lasso. Earlier work establishes that, under suitable assumptions on the design matrix, the coordinates of $\hat{\theta}^{\rm d}$ are asymptotically Gaussian provided $\theta^*$ is $s_0$-sparse with $s_0 = o(\sqrt{n}/\log p )$. The condition $s_0 = o(\sqrt{n}/ \log p )$ is stronger than the one for consistent estimation, namely $s_0 = o(n/ \log p)$. We study Gaussian designs with known or unknown population covariance. When the covariance is known, we prove that the debiased estimator is asymptotically Gaussian under the nearly optimal condition $s_0 = o(n/ (\log p)^2)$. Note that earlier work was limited to $s_0 = o(\sqrt{n}/\log p)$ even for perfectly known covariance. The same conclusion holds if the population covariance is unknown but can be estimated sufficiently well, e.g. under the same sparsity conditions on the inverse covariance as assumed by earlier work. For intermediate regimes, we describe the trade-off between sparsity in the coefficients and in the inverse covariance of the design. We further discuss several applications of our results to high-dimensional inference. In particular, we propose a new estimator that is minimax optimal up to a factor $1+o_n(1)$ for i.i.d. Gaussian designs.

Subfields: Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Double/De-Biased Machine Learning of Global and Local Parameters Using Regularized Riesz Representers (2018, methodology): “These types of conditions appear to be rather sharp, matching similar conditions used in Javanmard and Montanari (2018) in the case of inference on a single coefficient in Gaussian exactly sparse linear regression models.”
Approximate residual balancing: debiased inference of average treatment effects in high dimensions (2016, methodology): “Thus, the estimator (12) can be thought of as an adaptation of the method of Javanmard and Montanari (2014, 2015) that debiases β̂c specifically along the direction of interest ξ.”
Spectral Method and Regularized MLE Are Both Optimal for Top-$K$ Ranking (2017, unknown): “robust M-estimators (El Karoui, 2017), confidence intervals for Lasso (Javanmard and Montanari, 2015), likelihood ratio test (Sur et al.”
Learning L2 Continuous Regression Functionals via Regularized Riesz Representers (2018, methodology): “It has zero first order bias because α0(x)[y− γ0(x)] is the influence function for E[m(W, γ)], as shown in Newey (1994), and because a sample average plus an average of an estimate of that influence function has zero order bias; see Chernozhukov et al.”
Linear Hypothesis Testing in Dense High-Dimensional Linear Models (2016, background): “…0 = o ( √ n/ log p ) ; see Zhang and Zhang (2014); Belloni et al. (2014); Van de Geer et al. (2014); Javanmard and Montanari (2014a); Ning and Liu (2014); Javanmard and Montanari (2015); Mitra and Zhang (2014); Bühlmann and van de Geer (2015); Belloni et al. (2015); Chernozhukov et al. (2015).”
Breaking the curse of dimensionality in regression (2017, background): “Sparsity assumption on the model signals has had a significant role in achieving optimal inference – Cai and Guo (2015); Javanmard and Montanari (2015); Cai and Guo (2016) found minimax results quantifying the direct effect of the size of the sparsity.”
Double/de-biased machine learning using regularized Riesz representers (2018, methodology): “These types of conditions are rather sharp, matching similar conditions used in Javanmard and Montanari (2015) in the case of inference on a single coefﬁcient in Gaussian sparse linear regression models, and those in Zhu and Bradic (2017a) in the case of point testing general linear hypotheses on…”
High-dimensional regression adjustments in randomized experiments (2016, unknown): “Conversely, if we only have k n/ log( p ), accurate inference about the coeﬃcients of β is in general impossible without further conditions on the covariance of X [32, 33].”
Semiparametric efficiency bounds for high-dimensional models (2016, methodology): “In view of statistical inference, a de-sparsified or de-biased version of the Lasso was then considered [see Zhang and Zhang (2014), van de Geer et al. (2014), Javanmard and Montanari (2014a, 2014b, 2015)], which was shown to be asymptotically normal for estimation of β0j .”