Hypothesis Testing in High-Dimensional Regression Under the Gaussian Random Design Model: Asymptotic Theory

By Adel Javanmard, Andrea Montanari. 2013

167 citations recorded by the source corpus.

We consider linear regression in the high-dimensional regime where the number of observations n is smaller than the number of parameters p. A very successful approach in this setting uses 11-penalized least squares (also known as the Lasso) to search for a subset of s0 <; n parameters that best explain the data, while setting the other parameters to zero. Considerable amount of work has been devoted to characterizing the estimation and model selection problems within this approach. In this paper, we consider instead the fundamental, but far less understood, question of statistical significance. More precisely, we address the problem of computing p-values for single regression coefficients. On one hand, we develop a general upper bound on the minimax power of tests with a given significance level. We show that rigorous guarantees for earlier methods do not allow to achieve this bound, except in special cases. On the other, we prove that this upper bound is (nearly) achievable through a practical procedure in the case of random design matrices with independent entries. Our approach is based on a debiasing of the Lasso estimator. The analysis builds on a rigorous characterization of the asymptotic distribution of the Lasso estimator and its debiased version. Our result holds for optimal sample size, i.e., when n is at least on the order of s0 log(p/s0). We generalize our approach to random design matrices with independent identically distributed Gaussian rows xi ~ N(0, Σ). In this case, we prove that a similar distributional characterization (termed standard distributional limit) holds for n much larger than s0(log p)2.Our analysis assumes Σ is known. To cope with unknown Σ, we suggest a plug-in estimator for sparse covariances Σ and validate the method through numerical simulations. Finally, we show that for optimal sample size, n being at least of order s0 log(p/s0), the standard distributional limit for general Gaussian designs can be derived from the replica heuristics in statistical physics. This derivation suggests a stronger conjecture than the result we prove, and near-optimality of the statistical power for a large class of Gaussian designs.

Subfields: Artificial Intelligence, Computational Mechanics, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Statistics and Probability

Citations in the corpus

Double/De-Biased Machine Learning of Global and Local Parameters Using Regularized Riesz Representers (2018, background): “, to Newey [1994b]. In contrast, here we treat the case where the ambient covariate space is very high-dimensional, but we localize with respect to a value of a low-dimensional subvector.”
On asymptotically optimal confidence regions and tests for high-dimensional models (2013, neutral): “The Annals of Statistics 2014, Vol. 42, No. 3, 1166–1202 DOI: 10.1214/14-AOS1221 © Institute of Mathematical Statistics, 2014”
Double/Debiased Machine Learning for Treatment and Causal Parameters (2016, methodology): “n be seen as \debiasing" the score function (Y D g(Z))D, which does not possess the orthogonality property unless m 0(Z) = 0. This second approach represents a generalization of the approach of [33, 55, 61] considered for the case of Lasso-type methods without cross-tting. Like above, our gener”
Optimal Inference After Model Selection (2014, background): “There has also recently been work on inference in high-dimensional linear regression models, notably Javanmard and Montanari (2013) and Dezeure et al. (2014).”
A General Theory of Hypothesis Tests and Confidence Regions for Sparse High Dimensional Models (2014, methodology): “From a different perspective, [13, 14, 34, 42] proposed a debiased method (named as LDPE) or desparsifying method to construct confidence intervals for linear or generalized linear models with the Lasso penalty.”
Estimating structured high-dimensional covariance and precision matrices: Optimal rates and adaptive estimation (2016, methodology): “See Marčenko and Pastur (1967), Johnstone and Lu (2009), Paul (2007), and Johnstone (2001). In particular, when p > n, the sample covariance matrix is not invertible, and thus cannot be applied in many applications that require estimation of the precision matrix. To overcome the difficulty due to th”
Linear Hypothesis Testing in Dense High-Dimensional Linear Models (2016, methodology): “In high-dimensional literature Gaussian design is a very common assumption (e.g. Javanmard and Montanari (2014b); Cai and Guo (2015)).”
A modern maximum-likelihood theory for high-dimensional logistic regression (2018, unknown): “Later, for the linear model, [62] relaxed the ultra sparsity condition slightly to o(n/(log p)2); recently, [12] proposed a de-biased Lasso estimator with additional adjustments, which provides valid inference for low-dimensional projections as long as s0 = o(p) and s0 log(p/s0)/n = o(1).”
Double/de-biased machine learning using regularized Riesz representers (2018, methodology): “2 is a weak assumption that can be satisfied by taking l1n = l2n = √ log(p ∨ n) under weak assumptions on moments, as follows from self-normalized moderate deviation bounds (Jing et al. (2003); Belloni et al.”
Breaking the curse of dimensionality in regression (2017, result): “…guarantees asymptotic exactness of the proposed GRIP test in a class of high-correlation sparse models where the feature correlation and therefore sparsity of the precision matrix is unrestricted; Javanmard and Montanari (2014a) have a robust test in this setting when the testing set is univariate.”
Exact Post-Selection Inference for Sequential Regression Procedures (2014, background): “We have extended the asymptotic results for the covariance test of Lockhart et al. (2013), and more importantly proposed an asymptotically equivalent, finite sample test (the “spacings test”) that is exact under Gaussian errors.”
Double/debiased machine learning for treatment and structural parameters (2017, unknown): “See, for example, the work by Levit (1975), Ibragimov and Hasminskii (1981), Bickel (1982), Robinson (1988), Newey (1990), van der Vaart (1991), Andrews (1994a), Newey (1994), Newey et al.”
Debiasing the lasso: Optimal sample size for Gaussian designs (2015, methodology): “ [Buh13,ZZ14, JM14b,VdGBRD14,JM14a]. The present authors were motivated by the AMP analysis of the Lasso [DMM09,BM11,BM12,BLM15], and by the Gaussian limits that this analysis implies. In particular [JM14b] used those techniques to analyze standard Gaussian designs (i.e. the case = I) in the asympto”
Semiparametric efficiency bounds for high-dimensional models (2016, methodology): “In view of statistical inference, a de-sparsified or de-biased version of the Lasso was then considered [see Zhang and Zhang (2014), van de Geer et al. (2014), Javanmard and Montanari (2014a, 2014b, 2015)], which was shown to be asymptotically normal for estimation of β0j .”
Asymptotically Honest Confidence Regions for High Dimensional Parameters by the Desparsified Conservative Lasso (2014, methodology): “This paper in turn is related to Zhang and Zhang (2014), Javanmard and Montanari (2013) and Javanmard and Montanari (2014). The idea behind desparsification is to remove the bias introduced by shrinkage via desparsifying the estimator using a cleverly constructed approximate inverse of the non-inver”