Clustered Multi-Task Learning: A Convex Formulation

By Laurent Jacob, Francis R. Bach, Jean-Philippe Vert. 2008

521 citations recorded by the source corpus.

In multi-task learning several related tasks are considered simultaneously, with the hope that by an appropriate sharing of information across tasks, each task may benefit from the others. In the context of learning linear functions for supervised classification or regression, this can be achieved by including a priori information about the weight vectors associated with the tasks, and how they are expected to be related to each other. In this paper, we assume that tasks are clustered into groups, which are unknown beforehand, and that tasks within a group have similar weight vectors. We design a new spectral norm that encodes this a priori assumption, without the prior knowledge of the partition of tasks into groups, resulting in a new convex optimization formulation for multi-task learning. We show in simulations on synthetic examples and on the IEDB MHC-I binding dataset, that our approach outperforms well-known convex methods for multi-task learning, as well as related non-convex methods dedicated to the same problem.

Subfields: Artificial Intelligence, Computer Networks and Communications, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

An Overview of Multi-task Learning (2018, methodology): “Unlike[29,30],whichareBayesianmodels,there are several regularized methods [31–35] to do task clustering.”
A Survey on Multi-Task Learning (2017, methodology): “According to [65], it is well known in the vaccine design community that some molecules/tasks in the MHC-I dataset can be grouped into empirically defined supertypes known to have similar binding behaviors.”
Federated Multi-Task Learning (2017, methodology): “In (12), it can be shown [55, 20] that the optimal solution for Ω has the same column space as the rows of W .”
Subspace Regularizers for Few-Shot Class Incremental Learning (2021, background): “Past work on other multitask learning problems has demonstrated the effectiveness of constraining parameters for related tasks to be similar (Jacob et al., 2008), lie on the same manifold (Agarwal et al., 2010) or even on the same linear subspace (Argyriou et al., 2007a).”
Expert Gate: Lifelong Learning with a Network of Experts (2016, background): “As an alternative, [15, 38, 19] assume that the parameters of related task models lie close by in the original space or in a lower dimensional subspace and thus cluster the tasks’ parameters.”
Learning Task Grouping and Overlap in Multi-task Learning (2012, methodology): “Examples of such approaches are (Jacob et al., 2008; Xue et al., 2007) where tasks are assumed to be clustered and parameters of tasks within a cluster lie close to each other in ℓ 2 norm sense.”
A Convex Formulation for Learning Task Relationships in Multi-Task Learning (2010, methodology): “The task clustering methods in [25, 3, 26, 16] may be viewed as a way to learn the task relationships from data.”
Latent Multi-Task Architecture Learning (2017, methodology): “We cast learning what to share as a matrix regularization problem, following (Jacob et al. 2009; Yang and Hospedales 2017).”
GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks (2017, methodology): “Clustering methods have shown success beyond deep models (Jacob et al., 2009; Kang et al., 2011), while constructs such as deep relationship networks (Long & Wang, 2015) and cross-stich networks (Misra et al.”
Sluice networks: Learning what to share between loosely related tasks (2017, methodology): “We cast learning what to share as a matrix regularization problem, following [15, 29].”
Deep Multi-task Representation Learning: A Tensor Factorisation Approach (2016, background): “This diﬀers from the previously discussed studies [Evgeniou and Pontil, 2004, Argyriou et al., 2008, Bonilla et al., 2007, Jacob et al., 2009, Kumar and Daum´e III, 2012, Romera-paredes et al., 2013, Wimalawarne et al., 2014] which implicitly assume that each task has a single output.”
Scalable Transfer Learning with Expert Models (2020, background): “, 2009), or by imposing some prior knowledge regarding the task structure (Evgeniou et al., 2005; Jacob et al., 2009; Kim et al., 2012).”
SpotTune: Transfer Learning Through Adaptive Fine-Tuning (2018, methodology): “Early methods were designed for shallow classification models [52, 17, 36], while more recent approaches address the problem of “with whom” each task should share features using deep neural networks [29, 32].”
Fully-Adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification (2016, background): “A few methods have addressed the problem of “with whom” each task should share features [45, 16, 51, 18, 21, 26].”
Multi-Task Learning and Algorithmic Stability (2015, methodology): “…works, such as (Evgeniou and Pontil 2004; Evgeniou, Micchelli, and Pontil 2005; Kato et al. 2007) which assume that Ω is given as a priori information and (Jacob, Bach, and Vert 2008; Zhang and Yeung 2010a; Solnon, Arlot, and Bach 2012; Solnon 2013) which aim to learn Ω from data in different ways.”