A Study of Error Variance Estimation in Lasso Regression

By S. Reid, Robert Tibshirani, Jerome H. Friedman. 2013

237 citations recorded by the source corpus.

Variance estimation in the linear model when $p > n$ is a difficult problem. Standard least squares estimation techniques do not apply. Several variance estimators have been proposed in the literature, all with accompanying asymptotic results proving consistency and asymptotic normality under a variety of assumptions. It is found, however, that most of these estimators suffer large biases in finite samples when true underlying signals become less sparse with larger per element signal strength. One estimator seems to be largely neglected in the literature: a residual sum of squares based estimator using Lasso coefficients with regularisation parameter selected adaptively (via cross-validation). In this paper, we review several variance estimators and perform a reasonably extensive simulation study in an attempt to compare their finite sample performance. It would seem from the results that variance estimators with adaptively chosen regularisation parameters perform admirably over a broad range of sparsity and signal strength settings. Finally, some intial theoretical analyses pertaining to these types of estimators are proposed and developed.

Subfields: Artificial Intelligence, Computer Networks and Communications, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Statistics and Probability

Citations in the corpus

Selecting the number of principal components: estimation of the true rank of a noisy matrix (2014, methodology): “The estimator σ̂(2) λ and σ̂ 2 λ,df corresponds to the ordinary mean of sum of squared residuals and to mean of sum of squared residuals weighted accounting for degrees of freedom (Reid et al. 2013) respectively.”
Exact post-selection inference, with application to the lasso (2013, methodology): “and Tibshirani, R. (2004). Least angle regression. Annals of Statistics, 32 407–499. Fan, J., Guo, S. and Hao, N. (2012). Variance estimation using refitted cross-validation in ultrahigh dimensional regression.”
Splitting strategies for post-selection inference (2021, background): “The good performance of this estimator in sparse models was demonstrated in Reid et al. (2016).”
High-dimensional simultaneous inference with the bootstrap (2016, methodology): “= n 1=2 ˙^ "kZ jk 2= p n jZT j X j=nj ; ˙^2 " = 1 n s^ kY X^k2 2; (4) with ^sthe number of nonzero coecients in the estimate ^. This choice of ^˙2 " is based on the recommendation of Reid et al. (2016) and supported by our own empirical experience with dierent variance estimators. Thi”
A Critical Review of LASSO and Its Derivatives for Variable Selection Under Dependence Among Covariates (2020, background): “To give some guidance in this ﬁeld we refer to Fan et al. (2012) or Reid et al. (2016), although this still is a growing study ﬁeld.”
Prediction error of cross-validated Lasso (2015, methodology): “The recent paper of Reid, Tibshirani and Friedman [26] gives a comprehensive survey of these techniques and compares their strengths and weaknesses through extensive simulation studies.”
Efficient Inference of Average Treatment Effects in High Dimensions via Approximate Residual Balancing (2016, methodology): “rly, our conﬁdence intervals for τ are built using a cross-validated choice of λinstead of the ﬁxed choice assumed by Corollary 4, as motivated by several recent papers [Chatterjee and Jafarov, 2015, Reid et al., 2016]. Our implementation of approximate residual balancing, as well as all the discuss”
Simultaneous Inference for High-Dimensional Linear Models (2016, methodology): “To overcome this problem, we suggest the following modiﬁed variance estimator (see a similar estimator in Reid et al. 2014), where β sc denotes the scaled Lasso estimator with the tuning parameter λ 0 . b in the supplement provides boxplots of σ/σ for the variance estimator delivered by the scaled…”
More powerful post-selection inference, with application to the Lasso (2018, unknown): “However, Reid et al. (2016) shows that σ̂(2) = 1 n− pλ̂ n ∑”
Exact post-selection inference with the lasso (2013, methodology): “and Tibshirani, R. (2004). Least angle regression. Annals of Statistics, 32 407–499. Fan, J., Guo, S. and Hao, N. (2012). Variance estimation using refitted cross-validation in ultrahigh dimensional regression.”