Convex Multitask Learning with Flexible Task Clusters

By Leon Wenliang Zhong, James T. Kwok. 2012

69 citations recorded by the source corpus.

Traditionally, multitask learning (MTL) assumes that all the tasks are related. This can lead to negative transfer when tasks are indeed incoherent. Recently, a number of approaches have been proposed that alleviate this problem by discovering the underlying task clusters or relationships. However, they are limited to modeling these relationships at the task level, which may be restrictive in some applications. In this paper, we propose a novel MTL formulation that captures task relationships at the feature-level. Depending on the interactions among tasks and features, the proposed method construct different task clusters for different features, without even the need of pre-specifying the number of clusters. Computationally, the proposed formulation is strongly convex, and can be efficiently solved by accelerated proximal methods. Experiments are performed on a number of synthetic and real-world data sets. Under various degrees of task relationships, the accuracy of the proposed method is consistently among the best. Moreover, the feature-specific task clusters obtained agree with the known/plausible task structures of the data.

Subfields: Artificial Intelligence, Computational Mechanics, Optimization & Training, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Optimization & Training

In Statistical Learning Theory

Citations in the corpus

An Overview of Multi-task Learning (2018, methodology): “Unlike these methods, g ( U ) in [57] penalizes its complexity via the squared Frobenius norm and clusters feature in different tasks based on the fused lasso regular-izer.”
Learning Multi-Level Task Groups in Multi-Task Learning (2015, methodology): “…assignments based on relaxed integer programming, Kumar and Daume III (2012) generalize the non-overlapping cluster assignments for tasks to overlapping ones, and Zhong and Kwok (2012) focus on learning feature-level task groups where different features in one task can have different task groups.”
No Matter Where You Are: Flexible Graph-Guided Multi-task Learning for Multi-view Head Pose Classification under Target Motion (2013, background): “This is confirmed by the fact that in all situations (varying training set sizes and number of camera views) FTC MTL [24], Clustered MTL [25] and FEGA-MTL achieve superior performance.”
Using Task Descriptions in Lifelong Machine Learning for Improved Performance and Zero-Shot Transfer (2017, methodology): “…of forms, making use of common priors (Wilson, Fern, Ray, & Tadepalli, 2007; Lazaric & Ghavamzadeh, 2010), using regularization terms that couple task parameters (Evgeniou & Pontil, 2004; Zhong & Kwok, 2012), and ﬁnding mixtures of experts that can be shared across tasks (Bakker & Heskes, 2003).”
Flexible Clustered Multi-Task Learning by Learning Representative Tasks (2016, background): “Extensive experimental results are presented in Section 4, followed by conclusions and future works in Section 5.”
Encoding Tree Sparsity in Multi-Task Learning: A Probabilistic Framework (2014, background): “Typical examples include the dirty model (Jalali et al. 2010), robust MTL models (Chen, Zhou, and Ye 2011; Gong, Ye, and Zhang 2012), the multi-task cascade model (Zweig and Weinshall 2013), and some others (Chen, Liu, and Ye 2012; Zhong and Kwok 2012).”
Learning multiple visual tasks while discovering their structure (2015, methodology): “For each dataset we generated respectively 50 samples for training and 100 for test.”
Convex Multi-Task Learning by Clustering (2015, background): “Zhong and Kwok [28] proposed to learn two sets of models to be combined in prediction, the first captures the shared structure and the second captures variations specific to each task.”
Learning Transferable Knowledge Through Embedding Spaces (2019, methodology): “Bayesian approaches take a variety of forms, making use of common priors [173; 174], using regularization terms that couple task parameters [175; 176], and finding mixtures of experts that can be shared across tasks [177].”
Online Multitask Learning for Machine Translation Quality Estimation (2015, methodology): “Other methods (Jacob et al., 2009; Zhong and Kwok, 2012; Yan et al., 2014) cluster tasks into different groups and share”