Scaled sparse linear regression

By Tingni Sun, Cun-Hui Zhang. 2011

532 citations recorded by the source corpus.

Scaled sparse linear regression jointly estimates the regression coefficients and noise level in a linear model. It chooses an equilibrium with a sparse regression method by iteratively estimating the noise level via the mean residual square and scaling the penalty in proportion to the estimated noise level. The iterative algorithm costs little beyond the computation of a path or grid of the sparse regression estimator for penalty levels above a proper threshold. For the scaled lasso, the algorithm is a gradient descent in a convex minimization of a penalized joint loss function for the regression coefficients and noise level. Under mild regularity conditions, we prove that the scaled lasso simultaneously yields an estimator for the noise level and an estimated coefficient vector satisfying certain oracle inequalities for prediction, the estimation of the noise level and the regression coefficients. These inequalities provide sufficient conditions for the consistency and asymptotic normality of the noise-level estimator, including certain cases where the number of variables is of greater order than the sample size. Parallel results are provided for least-squares estimation after model selection by the scaled lasso. Numerical results demonstrate the superior performance of the proposed methods over an earlier proposal of joint convex minimization. Copyright 2012, Oxford University Press.

Subfields: Computational Mechanics, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/7cd1619370c1ab9b68acbb05d91b6562c78ed8a8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Statistics and Probability

Citations in the corpus

Confidence intervals for high-dimensional linear regression: Minimax rates and adaptivity (2015, methodology): “The second term ûᵀ 1 n Xᵀ(y − Xβ̂) in (3.6) is incorporated to reduce the bias of the scaled Lasso estimator β̂ .”
Ridge regression and asymptotic minimax estimation over spheres of growing dimension (2016, methodology): “The author would like to thank Alan Edelman, Bill Strawderman, Cun-Hui Zhang and Sihai Zhao for their helpful comments and inspiration.”
Hypothesis Testing in High-Dimensional Regression Under the Gaussian Random Design Model: Asymptotic Theory (2013, methodology): “This line of work is largely aimed at deriving asymptotically exact expressions for the risk of speciﬁc estimators, e.g., the Lasso or the Bayes optimal (minimum MSE) estimator.”
Confidence intervals for causal effects with invalid instruments by using two‐stage hard thresholding with voting (2016, unknown): “The proof has been established in Sun and Zhang (2012) for fixed designs under certain assumptions for the design. In the following, we will check that the assumptions in Corollary 1 in Sun and Zhang (2012) are satisfied with high probability for the subgaussian random designs considered in this pap”
Regression Analysis for Microbiome Compositional Data (2016, methodology): “;] i;i (1 (P C) i;i k 0 )2 b i;i : 4.3 Selection of the tuning parameters In real applications, the estimator bn, tuning parameter and estimation of noise level b˙are obtained through scaled lasso (Sun and Zhang, 2012). Specically, the following two steps are iterated until convergence: nb argmin C&”
Prediction error of cross-validated Lasso (2015, methodology): “There are a number of proposed methods for error variance estimation in Lasso [8, 11, 28, 31, 32, 33].”
A Critical Review of LASSO and Its Derivatives for Variable Selection Under Dependence Among Covariates (2020, unknown): “…(Bach (2008)), stability selection with randomized LASSO (Meinshausen and Bühlmann (2010)), the random LASSO (Wang et al. (2011)), the scaled LASSO (Sun and Zhang (2012)) or the combination of classic estimators with variable selection diagnostics measures (Nan and Yang (2014)), among others, are…”
Simultaneous Inference for High-Dimensional Linear Models (2016, methodology): “N (0 , 1) and deﬁne the multiplier bootstrap statistic, where σ 2 ǫ is a consistent estimator of the error variance σ 2 ǫ , e.g., the variance estimator from the scaled Lasso (Sun and Zhang 2012).”
Linear regression with sparsely permuted data (2017, background): “In this case, one can work with the 2-norm in (13) rather than with the squared 2-norm [7], or if σ is of independent interest, it can be estimated simultaneously [50, 51].”
Perspective Functions: Proximal Calculus and Applications in High-Dimensional Statistics (2016, methodology): “A prominent instance of this family of estimators is the scaled Lasso [26] formulation”
Non-negative least squares for high-dimensional linear models: consistency and sparse recovery without regularization (2012, background): “In [4] and [46] two related variants of the lasso are proposed that have similar theoretical guarantees, while the tuning parameter can be set without knowledge of the noise level.”
Linear Hypothesis Testing in Dense High-Dimensional Linear Models (2016, methodology): “The canonical examples of successful estimators represent Lasso and Dantzig selector, deﬁned as ˆ β l and ˆ β d below, (3.5) Although Lasso and Dantzig selector are deﬁned in diﬀerent times, Bickel et al. (2009) established equivalence between the two estimators under the conditions of moderate desi”
Local Inference in Additive Models with Decorrelated Local Linear Estimator (2019, methodology): “Additionally, we implement scaled Lasso estimator [36] to decouple the relation between Xi1 and Xi,−1,”