STANDARDIZATION AND THE GROUP LASSO PENALTY

By Noah R. Simon, Robert Tibshirani. 2012

159 citations recorded by the source corpus.

We re-examine the original Group Lasso paper of Yuan and Lin (2007). The form of penalty in that paper seems to be designed for problems with uncorrelated features, but the statistical community has adopted it for general problems with correlated features. We show that for this general situation, a Group Lasso with a different choice of penalty matrix is generally more effective. We give insight into this formulation and show that it is intimately related to the uniformly most powerful invariant test for inclusion of a group. We demonstrate the efficacy of this method– the “standardized Group Lasso”– over the usual group lasso on real and simulated data sets. We also extend this to the Ridged Group Lasso to provide within group regularization as needed. We discuss a simple algorithm based on group-wise coordinate descent to fit both this standardized Group Lasso and Ridged Group Lasso.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Peter Richtárik
2. Martin J. Wainwright
3. Robert Tibshirani
4. Trevor J. Hastie
5. Jian Huang
6. Daniela M. Witten
7. Martin Takác
8. Patrick Breheny

In Statistics and Probability

1. Robert Tibshirani
2. Jian Huang
3. Trevor J. Hastie
4. Patrick Breheny
5. Cun-Hui Zhang
6. Martin J. Wainwright
7. Daniela M. Witten
8. Noah R. Simon

Citations in the corpus

• Fused Lasso Additive Model (2014, methodology): “(2009) proposed a high-dimensional extension of GAM called sparse additive models (SpAM), which induces sparsity in the function estimates using a standardized group lasso penalty (Simon and Tibshirani 2012).”
• Group descent algorithms for nonconvex penalized linear and logistic regression models with grouped predictors (2012, methodology): “As discussed in Simon and Tibshirani (2011), this results in variable selection which is roughly equivalent to the uniformly most powerful invariant test for inclusion of the j th group.”
• Statistical Learning with Sparsity: The Lasso and Generalizations (2015, background): “Although the original authors (Yuan and Lin 2006) and many others since have made the orthonormality assumption, it has implications that are not always reasonable (Simon and Tibshirani 2012).”
• Network Reconstruction From High-Dimensional Ordinary Differential Equations (2016, background): “The optimization problem (17a) in the main paper is a standardized group lasso problem (Simon and Tibshirani, 2012).”
• Inexact Coordinate Descent: Complexity and Preconditioning (2013, methodology): “Applications where such problems arise, and where first-order methods have proved successful, include machine learning [1,2], compressive sensing [3,4], group lasso [5,6], matrix completion [7,8], and truss topology design [9].”
• PruneTrain: Gradual Structured Pruning from Scratch for Faster Neural Network Training (2019, background): “(Simon & Tibshirani, 2012; Alvarez & Salzmann, 2016) propose to penalize each group proportionally to its number of weights in order to maintain similar regularization intensity across all groups.”