A SIGNIFICANCE TEST FOR THE LASSO

By Richard A. Lockhart, Jonathan E. Taylor, Robert Tibshirani. Annals of Statistics, 2013

683 citations recorded by the source corpus.

In the sparse linear regression setting, we consider testing the significance of the predictor variable that enters the current lasso model, in the sequence of models visited along the lasso solution path. We propose a simple test statistic based on lasso fitted values, called the covariance test statistic, and show that when the true model is linear, this statistic has an Exp(1) asymptotic distribution under the null hypothesis (the null being that all truly active variables are contained in the current lasso model). Our proof of this result for the special case of the first predictor to enter the model (i.e., testing for a single significant predictor variable against the global null) requires only weak assumptions on the predictor matrix X. On the other hand, our proof for a general step in the lasso path places further technical assumptions on X and the generative model, but still allows for the important high-dimensional case p > n, and does not necessarily require that the current lasso model achieves perfect recovery of the truly active variables. Of course, for testing the significance of an additional variable between two nested linear models, one typically uses the chi-squared test, comparing the drop in residual sum of squares (RSS) to a χ12 distribution. But when this additional variable is not fixed, and has been chosen adaptively or greedily, this test is no longer appropriate: adaptivity makes the drop in RSS stochastically much larger than χ12 under the null hypothesis. Our analysis explicitly accounts for adaptivity, as it must, since the lasso builds an adaptive sequence of linear models as the tuning parameter λ decreases. In this analysis, shrinkage plays a key role: though additional variables are chosen adaptively, the coefficients of lasso active variables are shrunken due to the l1 penalty. Therefore, the test statistic (which is based on lasso fitted values) is in a sense balanced by these two opposing properties—adaptivity and shrinkage—and its null distribution is tractable and asymptotically Exp(1).

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/8125d755ad1fa64b869657d0f102836908af7a09

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Selective inference with a randomized response (2015, background): “Selective inference seeks to address this problem, see Fithian, Sun and Taylor (2014), Lee et al. (2016), Lockhart et al. (2014), Tibshirani et al. (2016).”
Exact Post-Selection Inference for Sequential Regression Procedures (2014, methodology): “Our final main result resolves the conjecture in (6), taken from Lockhart et al. (2013). We describe the asymptotic behavior of the spacings under more stringent assumptions than Theorems 1 and 2. While Lockhart et al. (2013) consider the case that there may be some strong signals in the model, we o”
Panning for gold: ‘model‐X’ knockoffs for high dimensional controlled variable selection (2016, methodology): “Relationship with work on inference after selection There is a line of work on inference after selection, or post-selection inference, for high dimensional regression, the goal of which is first to perform selection to make the problem low dimensional, and then to produce p-values that are valid con”
Two-Step Lasso Estimation of the Spatial Weights Matrix (2015, methodology): “[51] and the sample splitting approaches proposed by Wasserman and Roeder [52] and Meinshausen et al.”
Prediction error of cross-validated Lasso (2015, background): “In the high dimensional case, this problem has gained some prominence in recent times, partly because of the emergence of literature on significance tests for Lasso [19, 22].”
A General Framework for Robust Testing and Confidence Regions in High-Dimensional Quantile Regression (2014, background): “A separate line of research (Lockhart et al., 2014; G’Sell et al., 2013; Taylor et al., 2013, 2014) studied significance tests conditional on the selected model.”
Machine Learning Advances for Time Series Forecasting (2020, background): “Therefore, uniformly valid asymptotic confidence intervals for 𝛽 01 can be constructed in the usual fashion.”
Uniform asymptotic inference and the bootstrap after model selection (2015, background): “We prove that the test statistic of Tibshirani et al. (2016) is asymptotically valid, as the number of samples n grows and the dimension d of the regression problem stays fixed.”
Using Lasso for Predictor Selection and to Assuage Overfitting: A Method Long Overlooked in Behavioral Sciences (2015, methodology): “P values for Lasso estimates using the method by Lockhart et al. (2014) will be compared to p values obtained from OLS when variables enter the model one at a time, in the same order.”
Some Natural Solutions to the p-Value Communication Problem—and Why They Won’t Work (2017, methodology): “Our own preferred replacement for hypothesis testing and p-values is model expansion and Bayesian inference, addressing concerns of multiple comparisons using hierarchical modeling (Gelman, Hill, and Yajima, 2013) or through non-Bayesian regularization techniques such as lasso (Lockhart et al., 2013”
Valid Post-Selection and Post-Regularization Inference: An Elementary, General Approach (2015, neutral): “Related developments within this approach appear in G’Sell et al. (2013), Lee and Taylor (2014), Lee et al. (2013), Lockhart et al. (2014), Loftus and Taylor (2014), Taylor et al. (2014), and Fithian et al. (2014).”