A Convex Formulation for Learning Task Relationships in Multi-Task Learning

By Yu Zhang, Dit-Yan Yeung. 2010

476 citations recorded by the source corpus.

Multi-task learning is a learning paradigm which seeks to improve the generalization performance of a learning task with the help of some other related tasks. In this paper, we propose a regularization formulation for learning the relationships between tasks in multi-task learning. This formulation can be viewed as a novel generalization of the regularization framework for single-task learning. Besides modeling positive task correlation, our method, called multi-task relationship learning (MTRL), can also describe negative task correlation and identify outlier tasks based on the same underlying principle. Under this regularization framework, the objective function of MTRL is convex. For efficiency, we use an alternating method to learn the optimal model parameters for each task as well as the relationships between tasks. We study MTRL in the symmetric multi-task learning setting and then generalize it to the asymmetric setting as well. We also study the relationships between MTRL and some existing multi-task learning methods. Experiments conducted on a toy problem as well as several benchmark data sets demonstrate the effectiveness of MTRL.

Subfields: Artificial Intelligence, Civil and Structural Engineering, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Civil and Structural Engineering

In Representation & Self-Supervised Learning

Citations in the corpus

An Overview of Multi-Task Learning in Deep Neural Networks (2017, methodology): “[Daumé III, 2009] propose a hierarchical Bayesian model, which learns a latent task hierarchy, while [Zhang and Yeung, 2010] use a GP-based regularization for MTL and extend a previous GP-based approach to be more computationally feasible in larger settings.”
An Overview of Multi-task Learning (2018, methodology): “ThesetwomethodsextendtheMTFLand MTRLmethods[5,44],twomodelsintheMTSL setting,totheclusteringscenarioandtheformu-lationsintheproposedtwomulti-task-clustering methods are almost identical to those in the MTFL and MTRL methods, with the only difference being”
A Survey on Multi-Task Learning (2017, methodology): “(20) implies that WTW follows a Wishart distribution asW(·|0,Ω), Zhang and Yeung [91] generalize it as (WTW)t ∼ W(·|0,Ω), (22) where t is a positive integer to model high-order task relationships.”
Federated Multi-Task Learning (2017, methodology): “This prior generates a regularizer of the following form [54]: Unfortunately, such a regularizer is non-convex with respect to Ω due to the concavity of log | Ω | .”
Learning Task Grouping and Overlap in Multi-task Learning (2012, background): “There also exist probabilistic models which attempt to learn full task covariance matrix and use it in learning of predictor functions (Zhang & Yeung, 2010b;a; Zhang & Schneider, 2010; Archambeau et al., 2011).”
Multi-Task Learning and Algorithmic Stability (2015, methodology): “…(Evgeniou, Micchelli, and Pontil 2005; Kato et al. 2007), learning the task relation from data in different ways (Jacob, Bach, and Vert 2008; Zhang and Yeung 2010a; Solnon, Arlot, and Bach 2012; Zhang and Yeung 2013; 2014), decomposition methods (Chen, Liu, and Ye 2010; Jalali et al. 2010;…”
Multi-Task Learning as Multi-Objective Optimization (2018, background): “Zhang and Yeung (2010) consider the case where each learner is a linear model and use a task afﬁnity matrix.”
Attribute relation learning for zero-shot classification (2014, methodology): “As multiple attributes are usually used in the attributesbased classification, we resort to the task relationship learning approach in multi-task learning [41] to model the correlation relation between attributes.”
Multi-Task Boosting by Exploiting Task Relationships (2012, methodology): “In this section, we study MTBoost empirically on some applications and compare it with a single-task boosting method called AnyBoost [22], a multi-task boosting method called Multi-boost [20], a multi-task learning method called multi-task GP (MTGP) [10] and MTRL [12] which can also learn the task r”
Multi-task Recurrent Neural Network for Immediacy Prediction (2015, methodology): “Multi-task learning [40] have been used to directly model the relationships among correlated tasks.”
Multi-Task Learning in Heterogeneous Feature Spaces (2011, methodology): “More recently, some methods such as (Zhang and Yeung 2010) have been proposed to learn the task relationships under the regularization framework.”
Inductive multi-task learning with multiple view data (2012, methodology): “A remedy to those situation, as we investigate here, is to introduce a task relationship inference component [25].”
Flexible Clustered Multi-Task Learning by Learning Representative Tasks (2016, background): “Extensive experimental results are presented in Section 4, followed by conclusions and future works in Section 5.”
Learning to Multitask (2018, methodology): “…Schatten norm regularization with r = 2 (SNR 2 ) which is equivalent to the Schatten -norm regularization according to Theorem 1, the MTRL method [35, 36], squared Schatten norm regularization with r = 2 (SSNR 2 ) which is equivalent to squared Schatten -norm regularization according to Theorem…”
Federated Multi-Task Learning under a Mixture of Distributions (2021, background): “Problem (7) can also be justified by probabilistic priors [69] or graphical models [35] (see [59, App.”
Simultaneously Leveraging Output and Task Structures for Multiple-Output Regression (2012, methodology): “Experimental resultson both synthetic and real datasets demonstrate the effectivenessof our method.”