Taking Advantage of Sparsity in Multi-Task Learning

By Karim Lounici, Massimiliano Pontil, Alexandre B. Tsybakov, Sara A. van de Geer. 2009

298 citations recorded by the source corpus.

We study the problem of estimating multiple linear regression equations for the purpose of both prediction and variable selection. Following recent work on multi-task learning Argyriou et al. [2008], we assume that the regression vectors share the same sparsity pattern. This means that the set of relevant predictor variables is the same across the different equations. This assumption leads us to consider the Group Lasso as a candidate estimation method. We show that this estimator enjoys nice sparsity oracle inequalities and variable selection properties. The results hold under a certain restricted eigenvalue condition and a coherence condition on the design matrix, which naturally extend recent work in Bickel et al. [2007], Lounici [2008]. In particular, in the multi-task learning scenario, in which the number of tasks can grow, we are able to remove completely the effect of the number of predictor variables in the bounds. Finally, we show how our results can be extended to more general noise distributions, of which we only require the variance to be finite.

Subfields: Computational Mechanics, Computer Networks and Communications, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Networks and Communications

In Statistics and Probability

Citations in the corpus

A Survey on Multi-Task Learning (2017, background): “Several studies [230], [231], [232] investigate conditions to well recover true features for multi-task feature selection methods.”
Optimization with Sparsity-Inducing Penalties (2011, background): “A regularization norm exploiting explicitly this group structure, or `1-group norm, can be shown to improve the prediction performance and/or interpretability of the learned models [61, 83, 106, 116, 141, 156].”
Robust Lasso With Missing and Grossly Corrupted Observations (2011, neutral): “Then if U is distributed according to the Haar measure, P(f(U) ≥ median(f) + τ) ≤ exp(− mτ 2 8 ‖f‖2L ).”
Scalable Transfer Learning with Expert Models (2020, methodology): “We can combine the learning problems via regularization and shared sparsity patterns (Argyriou et al., 2007; Lounici et al., 2009), or by imposing some prior knowledge regarding the task structure (Evgeniou et al.”
Multi-Task Learning and Algorithmic Stability (2015, background): “Besides the generalization bound, the oracle inequality is studied in (Lounici et al. 2009; Solnon, Arlot, and Bach 2012; Solnon 2013).”
Learning Multi-Level Task Groups in Multi-Task Learning (2015, background): “Assumption 1 refers to the restricted eigenvalue assumption as introduced in (Lounici et al. 2009).”
Cross-Domain Multitask Learning with Latent Probit Models (2012, methodology): “We follow (Bickel et al., 2009; Lounici et al., 2009) to similarly define κs = min δ 6=0 n −1/2 t ‖δJ‖ −1 2 ‖Ψ T δ‖2, then ‖δJ‖2 ≤ κ−1s n −1/2 t ‖Ψ T δ‖2, (14) Substitution of (14) into (13) yields ‖ΨT δ‖2 ≤ 2r √ nts/κs, which is substituted back into (14) to give ‖δJ‖2 ≤ 2rκ−2s √ s. (15) By the…”
Learning Sparse FRAME Models for Natural Image Patterns (2014, background): “2006) under the name of simultaneous sparse coding, and in statistics and machine learning (Obozinski et al. 2011; Lounici et al. 2009) under the names of multi-task learning and support union recovery.”
Estimation of high-dimensional low-rank matrices (2009, methodology): “The main diﬀerence is that the sparsity index s appearing in Lounici et al. (2009, 2010) is now replaced by r .”