Simultaneous Variable Selection

By Berwin A. Turlach, William N. Venables, Stephen J. Wright. 2005

409 citations recorded by the source corpus.

We propose a new method for selecting a common subset of explanatory variables where the aim is to model several response variables. The idea is a natural extension of the LASSO technique proposed by Tibshirani (1996) and is based on the (joint) residual sum of squares while constraining the parameter estimates to lie within a suitable polyhedral region. The properties of the resulting convex programming problem are analyzed for the special case of an orthonormal design. For the general case, we develop an efficient interior point algorithm. The method is illustrated on a dataset with infrared spectrometry measurements on 14 qualitatively different but correlated responses using 770 wavelengths. The aim is to select a subset of the wavelengths suitable for use as predictors for as many of the responses as possible.

Subfields: Management Science and Operations Research, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Incremental Majorization-Minimization Optimization with Application to Large-Scale Machine Learning (2014, neutral): “For instance, the logarithm can be replaced by any smooth concave nondecreasing function, or group-sparsity penalties [52, 56] can be used, such as θ → ∑ g∈G log(‖θg‖2 + ε), where G is a partition of {1, .”
Optimization with Sparsity-Inducing Penalties (2011, methodology): “A regularization norm exploiting explicitly this group structure, or `1-group norm, can be shown to improve the prediction performance and/or interpretability of the learned models [61, 83, 106, 116, 141, 156].”
Distributed Multi-Task Learning (2015, background): “In this paper, we focus on the high-dimensional multi-task setting with joint support where a few variables are related to all tasks, while others are not predictive [6, 7, 8].”
Online Learning for Matrix Factorization and Sparse Coding (2009, neutral): “A Cholesky-based implementation of LARS-Lasso for decomposing a single signal has a complexity of O ( kms + ks 2 ), where s is the number of nonzero coeﬃcients.”
Multi-population GWA mapping via multi-task regularized regression (2010, background): “Various block-structured norms in the form of L1/Lq, q>0, to combine information from related inputs or outputs in a regression problem have been previously proposed (Turlach et al., 2005; Zhao et al., 2008).”
Multitask Sparsity via Maximum Entropy Discrimination (2011, methodology): “The multitask extension to such sparse estimation techniques is known as the Group Lasso and allows sparsity to be explored over predefin subsets of variables (Turlach et al., 2005; Yuan and Lin, 2006).”
Multi-level Lasso for Sparse Multi-task Regression (2012, methodology): “…and multi-task regression models, is based on extending the Lasso formulation (Tibshirani, 1994) to impose block-structured regularization via the l1-lq norm with q > 1 (Turlach et al., 2005; Obozinski et al., 2006; Negahban & Wainwright, 2009; Lounici et al., 2009; Tropp et al., 2006).”
Non-local sparse models for image restoration (2009, methodology): “We propose to extend and combine these two approaches by using simultaneous sparse coding [28] [29] [31] to impose that similar patches share the same dictionary elements in their sparse decomposition .”
A Complete Analysis of the l_1, p Group-Lasso (2012, methodology): “Algorithms for the 1 , ∞ -variant of the Group-Lasso were studied in (Turlach et al., 2005; Schmidt & Murphy, 2008; Quattoni et al., 2009) and (Vogt & Roth, 2010).”
Collaborative Sparse Regression for Hyperspectral Unmixing (2014, background): “Criterion (2) is similar to that of the collaborative (also called " multitask " or " simultaneous " ) sparse coding problem [21]–[24].”
Least angle and ℓ1 penalized regression: A review (2008, methodology): “Turlach et al. (2005) extend LASSO to select a common subset of predictors for predicting multiple response variables using the following criterion: N∑ r=1 ‖Y(tr) − Xβ(tr)‖22 + θ p∑ j=1 max r=1,...,N |βj(tr)|.”
Feature Selection Based on Structured Sparsity: A Comprehensive Study (2017, background): “Other applications of l∞,1 regularization are simultaneous sparse signal approximation [74], [75], structure learning [77], multiview learning [78], and multitask learning [79].”
Sparse Reconstruction by Separable Approximation (2008, methodology): “An interiorpoint method for the GScase was described in [74].”
Graph-Structured Multi-task Regression and an Efficient Optimization Method for General Fused Lasso (2010, methodology): “When the tasks are assumed to be equally related to inputs without any structure, a mixed-norm regularization such as ℓ 1 /ℓ 2 - and ℓ 1 /ℓ ∞ -norms has been used to ﬁnd inputs relevant to all of the outputs jointly [15, 19].”
Network Flow Algorithms for Structured Sparsity (2010, neutral): “However, the l1 norm encodes poor information, just cardinality! Another popular choice for Ω: The l1-lq norm [Turlach et al., 2005], with q = 2 or q = ∞ ∑”