A note on the group lasso and a sparse group lasso

By Jerome H. Friedman, Trevor J. Hastie, Robert Tibshirani. 2010

867 citations recorded by the source corpus.

We consider the group lasso penalty for the linear model. We note that the standard algorithm for solving the problem assumes that the model matrices in each group are orthonormal. Here we consider a more general penalty that blends the lasso (L1) with the group lasso (\two-norm"). This penalty yields solutions that are sparse at both the group and individual feature levels. We derive an ecien t algorithm for the resulting convex problem based on coordinate descent. This algorithm can also be used to solve the general form of the group lasso, with non-orthonormal model matrices.

Subfields: Cancer Research, Rheumatology, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cancer Research

In Rheumatology

In Statistics and Probability

Citations in the corpus

Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2016, background): “…helps in selecting physically meaningful groups of attributes in the data science models, and various extensions of group Lasso have been explored for handling different types of domain characteristics, e.g., overlapping group Lasso [62], tree-guided group Lasso [63], and sparse group Lasso [64].”
Regularizing Text Categorization with Clusters of Words (2016, background): “Fortunately, the problem can be resolved by adding a lasso regularization (Friedman et al., 2010).”
Optimization with Sparsity-Inducing Penalties (2011, background): “induces sparsity within the groups is quite natural [45, 111].”
Distributed Multi-Task Learning (2015, background): “DSML can be easily extended to other types of structured sparsity, including sparse group lasso [Friedman et al., 2010], tree-guided group lasso [Kim and Xing, 2010] and the dirty model [Jalali et al., 2010].”
Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function (2011, background): “(iv) There are many more choices such as the elastic net [32], group lasso [10,14,30] and sparse group lasso [4].”
Tree-Guided Group Lasso for Multi-Task Regression with Structured Sparsity (2009, background): “Although the L 1 /L 2 penalty has been shown to be effective in a joint covariate selection in multi-task learning, it assumed that all of the tasks are equally related with each other and share the same relavant inputs.”
Energy Disaggregation via Discriminative Sparse Coding (2010, methodology): “We use a coordinate descent approach here, both for the standard and group Lasso version of the opt imization problems, as these have been recently shown to be efficient algorithms for l1-type optimization problems [8, 9], and have the added benefit that we can warm-start the optimization with th e ”
Feature Selection (2016, methodology): “For supervised multi-view feature selection, the most common approach is Sparse Group Lasso (Friedman et al. 2010; Peng et al. 2010).”
A Selective Review of Group Selection in High-Dimensional Models. (2012, methodology): “Algorithms proposed for computing the group LASSO solutions without using Rj =X ′ jXj/n include Friedman et al. (2007), Jacob, Obozinski and Vert (2009) and Liu and Ye (2010).”
The joint graphical lasso for inverse covariance estimation across multiple classes (2011, background): “And one can show that the oﬀ-diagonal elements take the form (Friedman et al. 2010) where S denotes the soft-thresholding operator.”
Making the Most of Bag of Words: Sentence Regularization with Alternating Direction Method of Multipliers (2014, methodology): “Often linear combinations of different regularizers are used; examples include the elastic net (which combines ridge and lasso; Zou & Hastie, 2005) and the sparse group lasso (which combines lasso and group lasso; Friedman et al., 2010).”
Group Sparse Additive Models (2012, methodology): “In this sense, GroupSpAM can be viewed as a nonparametric extension of the generalized group lasso (Friedman et al., 2010), which allows non-orthonormal covariates within a group, to additive models.”
Structured sparsity through convex optimization (2011, background): “In this section, we consider structured sparsity-inducing norms that induce estimated vectors that are not only sparse, as for the ℓ 1 -norm, but whose support also displays some structure known a priori that reﬂects potential relationships between the variables.”
An empirical investigation of sparse log-linear models for improved dialogue act classification (2013, methodology): “Finally, to induce different hierarchies of sparsity in the feature space, we introduce the sparse group lasso model [39] that combines the element-wise and the group-wise lasso:”
Optimization for machine learning (2010, background): “The possibility of combining an l1/lq-norm that takes advantage of sparsity at the group level with an l1-norm that induces sparsity within the groups is quite natural (Friedman et al., 2010; Sprechmann et al., 2010).”
Randomized Block Coordinate Descent for Online and Stochastic Optimization (2014, methodology): “To induce both group sparsity and sparsity, sparse group lasso [9] uses composite regularizers gj(xj) = λ1‖xj‖2 + λ2‖xj‖1 where λ1 and λ2 are the tuning parameters.”
Group sparse regularization for deep neural networks (2016, methodology): “To force this, we also consider the following composite ‘sparse group Lasso’ (SGL) penalty [25], [26]: The SGL penalty has the same properties as its constituting norms, namely, it is convex but non-differentiable.”