Confidence intervals and hypothesis testing for high-dimensional regression

By Adel Javanmard, Andrea Montanari. 2013

832 citations recorded by the source corpus.

Fitting high-dimensional statistical models often requires the use of non-linear parameter estimation procedures. As a consequence, it is generally impossible to obtain an exact characterization of the probability distribution of the parameter estimates. This in turn implies that it is extremely challenging to quantify the uncertainty associated with a certain parameter estimate. Concretely, no commonly accepted procedure exists for computing classical measures of uncertainty and statistical significance as confidence intervals or p- values for these models. We consider here high-dimensional linear regression problem, and propose an efficient algorithm for constructing confidence intervals and p-values. The resulting confidence intervals have nearly optimal size. When testing for the null hypothesis that a certain parameter is vanishing, our method has nearly optimal power. Our approach is based on constructing a 'de-biased' version of regularized M-estimators. The new construction improves over recent work in the field in that it does not assume a special structure on the design matrix. We test our method on synthetic data and a high-throughput genomic data set about riboflavin production rate, made publicly available by Buhlmann et al. (2014).

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Deep Neural Networks for Estimation and Inference: Application to Causal Effects and Other Semiparametric Estimands (2018, background): “Initial theoretical results were concerned with obtaining valid inference on a coefficient in a high-dimensional regression, following model selection or regularization, with particular focus on the lasso (Belloni et al., 2012; Javanmard and Montanari, 2014; van de Geer et al., 2014).”
Double/De-Biased Machine Learning of Global and Local Parameters Using Regularized Riesz Representers (2018, methodology): “De-Biased Machine Learning of Global and Local Parameters Using Regularized Riesz Representers Victor Chernozhukov†, Whitney K. Newey†, and Rahul Singh† †MIT Economics, 50 Memorial Drive, Cambridge MA 02142, USA.”
Approximate residual balancing: debiased inference of average treatment effects in high dimensions (2016, methodology): “Thus, the estimator (12) can be thought of as an adaptation of the method of Javanmard and Montanari (2014, 2015) that debiases β̂c specifically along the direction of interest ξ.”
Distributed Multi-Task Learning (2015, methodology): “Here, we adopt the approach proposed in [Javanmard and Montanari, 2014], as it leads to weakest assumption on the model in (16): each machine uses a matrix M t = ( ˆ m tj ) =1 with rows: where e j is the vector with j -th component equal to 1 and 0 otherwise and ˆΣ t = n − 1 X Tt X t .”
On asymptotically optimal confidence regions and tests for high-dimensional models (2013, methodology): “At the time of writing of this paper, [22] developed another modification which directly computes an approximate inverse of the Gram matrix.”
A General Theory of Hypothesis Tests and Confidence Regions for Sparse High Dimensional Models (2014, methodology): “When the linear model assumption is invalid, the results in [34] and [13] are not directly applicable.”
Distribution-Free Predictive Inference for Regression (2016, methodology): “In the first class, population-based approaches, the linear model is assumed to be true and the focus is on providing confidence intervals for the coefficients in this model (see, e.g., Belloni et al. 2012; Buhlmann 2013; Javanmard and Montanari 2014; van de Geer et al. 2014; Zhang and Zhang 2014).”
Efficient Distributed Learning with Sparsity (2016, methodology): “In particular, the debiasing step requires solving Oppq generalized lasso problems, which is computationally prohibitive for high-dimensional problems (Zhang and Zhang, 2013; Javanmard and Montanari, 2014).”
Prediction error of cross-validated Lasso (2015, background): “In the high dimensional case, this problem has gained some prominence in recent times, partly because of the emergence of literature on significance tests for Lasso [19, 22].”
High-dimensional inference in misspecified linear models (2015, background): “Most of the recent contributions for this task discuss some methods and approaches for highdimensional linear models [5, 35, 31, 19, 22, 13], but generalized linear models [24, 25, 31], undirected graphical models [26, 18], instrumental variable models [1] or very general models [23] have been consi”
Familywise error rate control via knockoffs (2015, background): “There are also works that obtain asymptotic control of the FWER under some assumptions on the distribution of the design matrix (see, for example, [5, 17]).”
Regression Analysis for Microbiome Compositional Data (2016, methodology): “The proposed debiased procedure extends that of Javanmard and Montanari (2014) to take into account the linear constraints on regression coefficients.”
Confidence sets in sparse regression (2012, result): “…θ such as low-dimensional projections, the situation may be different: for instance, in the recent papers of Zhang and Zhang (2011), van de Geer, Bühlmann and Ritov (2013) and Javanmard and Montanari (2013) one-dimensional confidence intervals for a fixed element θj in the vector θ are constructed.”
Local Inference in Additive Models with Decorrelated Local Linear Estimator (2019, background): “Debiased estimators were developed in [44], [22] and [39] to study the inference problem for a single regression coefficient βi.”
Minimax Estimation of Functionals of Discrete Distributions (2014, background): “The idea of decreasing the bias at the expense of increasing variance to obtain good confidence intervals also appears in [96].”
Honest confidence regions and optimality in high-dimensional precision matrix estimation (2015, methodology): “Our methodology is a simple modification of the nodewise Lasso estimator proposed in Meinshausen and Bühlmann (2006). The idea is to remove the bias term which arises in the nodewise Lasso estimator due to `1-penalty regularization.”