Encoding Tree Sparsity in Multi-Task Learning: A Probabilistic Framework

By Lei Han, Yu Zhang, Guojie Song, Kunqing Xie. 2014

43 citations recorded by the source corpus.

Multi-task learning seeks to improve the generalization performance by sharing common information among multiple related tasks. A key assumption in most MTL algorithms is that all tasks are related, which, however, may not hold in many real-world applications. Existing techniques, which attempt to address this issue, aim to identify groups of related tasks using group sparsity. In this paper, we propose a probabilistic tree sparsity (PTS) model to utilize the tree structure to obtain the sparse solution instead of the group structure. Specifically, each model coefficient in the learning model is decomposed into a product of multiple component coefficients each of which corresponds to a node in the tree. Based on the decomposition, Gaussian and Cauchy distributions are placed on the component coefficients as priors to restrict the model complexity. We devise an efficient expectation maximization algorithm to learn the model parameters. Experiments conducted on both synthetic and real-world problems show the effectiveness of our model compared with state-of-the-art baselines.

Subfields: Artificial Intelligence, Computational Mechanics, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qiang Yang
2. Yu Zhang
3. Xinchao Wang
4. Lei Han
5. Yu Zhang
6. Mingli Song
7. Qi Tian
8. Jingwen Ye

In Computational Mechanics

1. Qiang Yang
2. Yu Zhang
3. Lei Han
4. Yu Zhang
5. Xinchao Wang
6. Mingli Song
7. Jingwen Ye
8. Qi Tian

In Representation & Self-Supervised Learning

1. Qiang Yang
2. Yu Zhang
3. Lei Han
4. Yu Zhang
5. Wen Gao
6. Qi Tian
7. Yixin Ji
8. Mingli Song

Citations in the corpus

• An Overview of Multi-task Learning (2018, background): “This model is extended in [13,14] to more general settings.”
• A Survey on Multi-Task Learning (2017, methodology): “[36] propose a multi-component product based decomposition for wij where the number of components in the decomposition can be arbitrary instead of only 2 in [34], [35].”
• Learning Multi-Level Task Groups in Multi-Task Learning (2015, background): “For example, to find the cross-talks from gene expressions, the correlated genes are often interacted with multi-level clusters as studied in (Kim and Xing 2010; Han et al. 2014).”
• Multi-Task Learning with Low Rank Attribute Embedding for Multi-Camera Person Re-Identification (2018, methodology): “Dictionary learning [57] and tree sparsity constraint [58] are also integrated with the standard MTL framework.”
• Multi-Stage Multi-Task Learning with Reduced Rank (2016, methodology): “Many MTL methods have been proposed by learning common feature representations (Caruana 1997; Argyriou, Evgeniou, and Pontil 2008; Liu, Ji, and Ye 2009; Zhang, Yeung, and Xu 2010), learning task groups (Han and Zhang 2015a), learning task relations (Zhang and Yeung 2010a; Zhang 2013; Zhang and Yeung”
• Learning Sparse Task Relations in Multi-Task Learning (2017, methodology): “Moreover, Han et al. (Han et al. 2014) utilize the given hierarchical structure among different tasks as a priori information Copyright c© 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org).”
• Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN (2020, methodology): “[10] proposes a probabilistic tree sparsity model that utilizes the tree structure to obtain the sparse solution.”
• Amalgamating Filtered Knowledge: Learning Task-customized Student from Multi-task Teachers (2019, methodology): “For example, [Han et al., 2014] proposes a probabilistic tree sparsity model that utilizes the tree structure to obtain the sparse solution instead of the group structure.”
• Learning Tree Structure in Multi-Task Learning (2015, methodology): “Moreover, some domain knowledge which specifies the tree structure among tasks has been utilized in [17, 9, 13].”