The composite absolute penalties family for grouped and hierarchical variable selection

By Peng Zhao, Guilherme V. Rocha, Bin Yu. 2009

688 citations recorded by the source corpus.

Extracting useful information from high-dimensional data is an important focus of today's statistical research and practice. Penalized loss function minimization has been shown to be effective for this task both theoretically and empirically. With the virtues of both regularization and sparsity, the $L_1$-penalized squared error minimization method Lasso has been popular in regression models and beyond. In this paper, we combine different norms including $L_1$ to form an intelligent penalty in order to add side information to the fitting of a regression or classification model to obtain reasonable estimates. Specifically, we introduce the Composite Absolute Penalties (CAP) family, which allows given grouping and hierarchical relationships between the predictors to be expressed. CAP penalties are built by defining groups and combining the properties of norm penalties at the across-group and within-group levels. Grouped selection occurs for nonoverlapping groups. Hierarchical variable selection is reached by defining groups with particular overlapping patterns. We propose using the BLASSO and cross-validation to compute CAP estimates in general. For a subfamily of CAP estimates involving only the $L_1$ and $L_{\infty}$ norms, we introduce the iCAP algorithm to trace the entire regularization path for the grouped selection problem. Within this subfamily, unbiased estimates of the degrees of freedom (df) are derived so that the regularization parameter is selected without cross-validation. CAP is shown to improve on the predictive performance of the LASSO in a series of simulated experiments, including cases with $p\gg n$ and possibly mis-specified groupings. When the complexity of a model is properly calculated, iCAP is seen to be parsimonious in the experiments.

Subfields: Computational Mechanics, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Statistics and Probability

Citations in the corpus

Sparse Partially Linear Additive Models (2014, methodology): “The penalty function SPLAM is convex and is an instance of the hierarchical group lasso (Zhao, Rocha, and Yu 2009; Jenatton et al. 2010).”
Optimization with Sparsity-Inducing Penalties (2011, methodology): “3 have notably proven to be useful in the context of hierarchical variable selection [9, 121, 157],”
Distributed Multi-Task Learning (2015, methodology): “For example, the group lasso penalty uses pen( t ∈ m β 2 tj [Yuan and Lin, 2006], while the iCAP uses pen( Zhao et al., 2009].”
Robust Lasso With Missing and Grossly Corrupted Observations (2011, neutral): “[1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15]).”
Tree-Guided Group Lasso for Multi-Task Regression with Structured Sparsity (2009, methodology): “We present the experimental results using simulated data and yeast data in Section 5, and conclude in Section 6.”
Bayesian group factor analysis with structured sparsity (2016, background): “…structure of Λ in GFA models belongs to a general class of structured sparsity methods that has drawn attention recently (Zou and Hastie, 2005; Yuan and Lin, 2006; Jenatton et al., 2011, 2010; Kowalski, 2009; Kowalski and Torrésani, 2009; Zhao et al., 2009; Huang et al., 2011; Jia et al., 2010).”
SLEP: Sparse Learning with Efficient Projections (2011, background): “The l1/lq-regularization belongs to the composite absolute penalti s (CAP) [73] family.”
A LASSO FOR HIERARCHICAL INTERACTIONS (2012, methodology): “For example, Zhao, Rocha and Yu (2009) consider the penalty Likewise, the framework of Bach et al. (2012) if specialized to this paper’s focus would lead to a penalty of the form for some q > 1 and d j > 0.”
Policy evaluation with temporal differences: a survey and comparison (2015, background): “There has been work on exploiting such structures with hierarchical regularization schemes in regression and classification problems (Zhao et al., 2009; Jenatton et al., 2010).”
Bayesian Sparse Group Selection (2016, methodology): “For example, partitioning around medoids algorithm was used by Zhao, Rocha, and Yu (2009), and Bühlmann et al. (2013) proposed bottom-up agglomerative clustering algorithm based on canonical correlations.”
Causal Interaction in Factorial Experiments: Application to Conjoint Analysis (2018, neutral): “While there exist other regularization methods for categorical variables (e.g., Yuan and Lin 2006; Meier, Van De Geer, and Bühlmann 2008; Zhao, Rocha, and Yu 2009; Huang et al. 2009; Huang, Breheny, and Ma 2012; Lim and Hastie 2015), these methods regularize coe ffi cients rather than their di ff…”
Forest Garrote (2009, background): “The beneficial effect of this bias reduction for important variables has been noted in, amongst others, Yuan and Lin (2007) and Zou (2006).”
On Multiplicative Multitask Feature Learning (2014, methodology): “A common strategy is to impose a blockwise joint regularization (Meier et al., 2008; Zhao et al., 2009) on all task model parameters to shrink the effects of features across the tasks and simultaneously minimize the regression or classification loss.”
Hyperspectral Image Classification Based on Structured Sparse Logistic Regression and Three-Dimensional Wavelet Texture Features (2013, background): “It makes the coefficients w sparse not only between groups but also in individual variables of each group [28] such that”