Degrees of freedom in lasso problems

By Robert Tibshirani, Jonathan E. Taylor. 2011

383 citations recorded by the source corpus.

We derive the degrees of freedom of the lasso fit, placing no assumptions on the predictor matrix $X$. Like the well-known result of Zou, Hastie and Tibshirani [Ann. Statist. 35 (2007) 2173-2192], which gives the degrees of freedom of the lasso fit when $X$ has full column rank, we express our result in terms of the active set of a lasso solution. We extend this result to cover the degrees of freedom of the generalized lasso fit for an arbitrary predictor matrix $X$ (and an arbitrary penalty matrix $D$). Though our focus is degrees of freedom, we establish some intermediate results on the lasso and generalized lasso that may be interesting on their own.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Additive models with trend filtering (2017, background): “Of course, additive trend filtering is a not a linear smoother; however, it is a particular type of generalized lasso estimator, and degrees of freedom for such a class of estimators is well-understood [Tibshirani and Taylor, 2011, 2012].”
Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso (2017, neutral): “…on their respective active sets; • thanks to such shrinkage, the fitted values from the lasso (for any fixed λ ≥ 0) are continuous functions of y (Zou et al., 2007; Tibshirani and Taylor, 2012), whereas the fitted values from forward stepwise and best subset selection (for fixed k ≥ 1) jump…”
Very short term irradiance forecasting using the lasso (2015, methodology): “Alternatively, information criteria can be used (see Zou et al., 2007; Tibshirani and Taylor, 2012).”
Likelihood-Free Inference by Ratio Estimation (2016, background): “Different schemes to choose λ have been proposed that aim at minimising the prediction risk (Zou et al., 2007; Wang and Leng, 2007; Tibshirani and Taylor, 2012; Dutta et al., 2012).”
Wind Turbine Gearbox Failure Identification With Deep Neural Networks (2017, background): “where L and v are two BIC parameters, which can be estimated according to [38].”
Prediction error of cross-validated Lasso (2015, methodology): “Systematic ways of deterministically choosing the tuning parameter have been proposed by Wang and Leng [40], Zou et al. [44] and Tibshirani and Taylor [36].”
Effective degrees of freedom: a flawed metaphor (2013, neutral): “For the Lasso, we have the following results relating DF to the number of nonzero coefficients in the fitted model (NNZ): • DF = E[NNZ] for the penalized form of the Lasso (Zou et al., 2007; Tibshirani and Taylor, 2012).”
On cross-validated Lasso in high dimensions (2016, background): “In turn, this formula, being a consequence of the Stein identity characterizing the standard Gaussian distribution, has a simple form whenever ε ∼ N(0, σ2); see [49] and [42].”
Degrees of freedom and model search (2014, background): “Theorem 1 (Zou et al. 2007, Tibshirani & Taylor 2012).”
The Lasso Problem and Uniqueness (2012, methodology): “…the explicit form of a lasso solution given by Lemma 8 may be helpful for the purposes of mathematical analysis; for example, this form is used by Tibshirani & Taylor (2012) to give a simpler proof of the degrees of freedom of the lasso ﬁt, for a general X , in terms of the equicorrelation set.”
A LASSO FOR HIERARCHICAL INTERACTIONS (2012, methodology): “(cid:0) (cid:1) we solve for ˆ φ in the same manner as is done in (Ryan) Tibshirani and Taylor (2012). e e Thus, P X T XP ˆ φ = P X T ( y − ( P X T ) + P w ) from which we get ˆ φ = ( XP ) + ( y − ( P X T ) + P w ) + b for b ∈ null( XP ) and such that Db = 0 and D i ˆ φ > 0 for i ∈ A (ˆ φ ).”