Learning Multi-Level Task Groups in Multi-Task Learning

By Lei Han, Yu Zhang. 2015

78 citations recorded by the source corpus.

In multi-task learning (MTL), multiple related tasks are learned jointly by sharing information across them. Many MTL algorithms have been proposed to learn the underlying task groups. However, those methods are limited to learn the task groups at only a single level, which may be not sufficient to model the complex structure among tasks in many real-world applications. In this paper, we propose a Multi-Level Task Grouping (MeTaG) method to learn the multi-level grouping structure instead of only one level among tasks. Specifically, by assuming the number of levels to be H, we decompose the parameter matrix into a sum of H component matrices, each of which is regularized with a l2 norm on the pairwise difference among parameters of all the tasks to construct level-specific task groups. For optimization, we employ the smoothing proximal gradient method to efficiently solve the objective function of the MeTaG model. Moreover, we provide theoretical analysis to show that under certain conditions the MeTaG model can recover the true parameter matrix and the true task groups in each level with high probability. We experiment our approach on both synthetic and real-world datasets, showing competitive performance over state-of-the-art MTL methods.

Subfields: Artificial Intelligence, Computational Mechanics, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qiang Yang
2. Yu Zhang
3. Ying Wei
4. Lei Han
5. Yu Zhang
6. Jian Pei
7. Yutao Huang
8. Jiangchuan Liu

In Computational Mechanics

1. Qiang Yang
2. Yu Zhang
3. Lei Han
4. Yu Zhang
5. Ying Wei
6. Yutao Huang
7. Jian Pei
8. Jiangchuan Liu

In Evaluation & Benchmarks

1. Qiang Yang
2. Yu Zhang
3. Lei Han
4. Yu Zhang
5. Jian Pei
6. Ying Wei
7. Yong Zhang

Citations in the corpus

• An Overview of Multi-task Learning (2018, methodology): “:3) j > i w i − w j 2 , is proposed in [34] to enforce any pair of model parameters to be fused.”
• A Survey on Multi-Task Learning (2017, methodology): “(20) implies that WTW follows a Wishart distribution asW(·|0,Ω), Zhang and Yeung [91] generalize it as (WTW)t ∼ W(·|0,Ω), (22) where t is a positive integer to model high-order task relationships.”
• Learning Tree Structure in Multi-Task Learning (2015, methodology): “We compare the TAT method with several state-of-the-art MTL models, including the multi-task feature learning (MTFL) model [19],(3) the dirty model (Dirty) [15],(3) the Cascade model [28], the clustered multi-task learning (CMTL) model [14],(4) the grouping and overlap MTL (GOMTL) model [18], and th”
• Multi-Stage Multi-Task Learning with Reduced Rank (2016, methodology): “…feature representations (Caruana 1997; Argyriou, Evgeniou, and Pontil 2008; Liu, Ji, and Ye 2009; Zhang, Yeung, and Xu 2010), learning task groups (Han and Zhang 2015a), learning task relations (Zhang and Yeung 2010a; Zhang 2013; Zhang and Yeung 2014), utilizing or learning tree structure among…”
• Personalized Federated Learning: An Attentive Collaboration Approach (2020, methodology): “Inspired by several widely adopted formulations in multi-task learning [38, 39], we formulate the personalized federated learning problem as min w 1 ,...,w m where the first term is the training loss of the local models of all clients, λ > 0 is a regularization parameter, and D ( · , · ) is a…”
• Learning to Multitask (2018, background): “This approach can be further classified into four categories: low-rank approach [1, 28, 17], task clustering approach [18, 20, 15], task relation learning approach [39, 40, 36, 35, 41, 42, 21, 37], and decomposition approach [10, 19, 44, 16].”
• Graph‐guided multi‐task sparse learning model: a method for identifying antigenic variants of influenza A(H3N2) virus (2018, unknown): “(5) is convex and smooth with gradient rgl W ð Þ 1⁄4 A >C, where A is the optimal solution to (5) (Han and Zhang, 2015).”
• Learning Sparse Task Relations in Multi-Task Learning (2017, methodology): “For example, when Ω ∝ I where I denotes an identity matrix with appropriate size, the second term is the squared Frobenius norm regularization on W and if Ω is a diagonal matrix, it becomes the weighted Frobenius norm regularization for W. Similar to (Zhang and Yeung 2010a), Ω, which is assumed to b”