The adaptive and the thresholded Lasso for potentially misspecified models (and a lower bound for the Lasso)

By Sara A. van de Geer, Peter Buhlmann, Shuheng Zhou. 2011

142 citations recorded by the source corpus.

Abstract: We revisit the adaptive Lasso as well as the thresholded Lasso with refitting, in a high-dimensional linear model, and study prediction error, lq-error (q ∈ {1, 2}), and number of false positive selections. Our theoretical results for the two methods are, at a rather fine scale, comparable. The differences only show up in terms of the (minimal) restricted and sparse eigenvalues, favoring thresholding over the adaptive Lasso. As regards prediction and estimation, the difference is virtually negligible, but our bound for the number of false positives is larger for the adaptive Lasso than for thresholding. We also study the adaptive Lasso under beta-min conditions, which are conditions on the size of the coefficients. We show that for exact variable selection, the adaptive Lasso generally needs more severe beta-min conditions than thresholding. Both the two-stage methods add value to the one-stage Lasso in the sense that, under appropriate restricted and sparse eigenvalue conditions, they have similar prediction and estimation error as the one-stage Lasso but substantially less false positives. Regarding the latter, we provide a lower bound for the Lasso with respect to false positive selections.

Subfields: Artificial Intelligence, Finance, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Finance

In Statistics and Probability

Citations in the corpus

CAM: Causal Additive Models, high-dimensional order search and penalized regression (2013, background): “Assumption (B2,ii) can be ensured by assuming max j | A j | ≤ M 1 < ∞ and e.g. group restricted eigenvalue assumptions for the design matrix (with the given basis); see Zhang and Huang (2008) and van de Geer et al. (2011) for the case without groups.”
Uncertainty Quantification for the Horseshoe (with Discussion) (2016, methodology): “Lasso simply uses an L1-norm,∑K i=1 |θi|, as opposed to the horseshoe which (essentially) uses the penalty πHS(θi|τ) = − log pHS(θi|τ) = − log log ( 1 + 2τ2 θ2i ) .”
High-dimensional simultaneous inference with the bootstrap (2016, methodology): “The latter holds under a sparse eigenvalue condition on the design (Zhang and Huang 2008) or when using, e.g., the adaptive or thresholded Lasso in the construction of the bootstrap samples (van de Geer et al. 2011) and (Bühlmann and van de Geer 2011, Ch.7.8-7.9).”
Efficient Estimation of Approximate Factor Models (2012, background): “Stock and Watson (1998, 2002) and Bai (2003) considered the principal components analysis (PCA), and they developed large-sample inferential theory.”
$\ell_0$-penalized maximum likelihood for sparse directed acyclic graphs (2012, background): “, In addition the price to pay, in a standard regression problem with an ℓ 1 -norm regularization, is a bias which should be further controlled with, for example, adaptive ℓ 1 -norm regularization [Zou (2006)] or thresh-olding [van de Geer, B¨uhlmann and Zhou (2011)].”
High dimensional thresholded regression and shrinkage effect (2014, background): “In the problem of Gaussian mean estimation, there is a well-known phenomenon of Stein’s shrinkage effect (Stein, 1956; James and Stein, 1961) stating that the maximum likelihood estimator or least-squares estimator may no longer be optimal in risk under the quadratic loss in multiple dimensions.”
Preconditioning the Lasso for sign consistency (2015, methodology): “…(Zou, 2006); this applies a diﬀerent ℓ 1 penalty to each element of the coeﬃcient vector; these penalty weights come from an initial run of OLS. van de Geer et al. (2011) illustrated how this can be extended to the high dimensional setting by using an initial run of the Lasso instead of OLS.”
A Critical Review of LASSO and Its Derivatives for Variable Selection Under Dependence Among Covariates (2020, methodology): “Some of them opt to add a second selection step after solving the LASSO problem, such as the relaxed LASSO (Meinshausen (2007)) or thresholded LASSO (Zhou (2010) to the covariates proportional to their importance, as the adaptive LASSO (Zou (2006), Huang et al. (2008), Van de Geer et al. (2011)).”