Clustered Multi-Task Learning Via Alternating Structure Optimization

By Jiayu Zhou, Jianhui Chen, Jieping Ye. 2011

320 citations recorded by the source corpus.

Multi-task learning (MTL) learns multiple related tasks simultaneously to improve generalization performance. Alternating structure optimization (ASO) is a popular MTL method that learns a shared low-dimensional predictive structure on hypothesis spaces from multiple related tasks. It has been applied successfully in many real world applications. As an alternative MTL approach, clustered multi-task learning (CMTL) assumes that multiple tasks follow a clustered structure, i.e., tasks are partitioned into a set of groups where tasks in the same group are similar to each other, and that such a clustered structure is unknown a priori. The objectives in ASO and CMTL differ in how multiple tasks are related. Interestingly, we show in this paper the equivalence relationship between ASO and CMTL, providing significant new insights into ASO and CMTL as well as their inherent relationship. The CMTL formulation is non-convex, and we adopt a convex relaxation to the CMTL formulation. We further establish the equivalence relationship between the proposed convex relaxation of CMTL and an existing convex relaxation of ASO, and show that the proposed convex CMTL formulation is significantly more efficient especially for high-dimensional data. In addition, we present three algorithms for solving the convex CMTL formulation. We report experimental results on benchmark datasets to demonstrate the efficiency of the proposed algorithms.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Yu Zhang
2. Qiang Yang
3. Dacheng Tao
4. Yi Yang
5. Luc Van Gool
6. Kristen Grauman
7. Liang Zheng
8. Yoshua Bengio

Citations in the corpus

• A Survey on Multi-Task Learning (2017, background): “Moreover, the equivalence between problems (13) and (16), two typical methods in the low-rank and task clustering approaches, has been proved in [72].”
• Federated Multi-Task Learning (2017, methodology): “However, it is natural to assume multiple clusters exist, and to learn this clustering structure directly from the data [55].”
• Learning Task Grouping and Overlap in Multi-task Learning (2012, methodology): “Some methods assume that tasks can be grouped in clusters and parameters of tasks within a cluster are either close to each other in some distance metric or share a common probabilistic prior (Bakker & Heskes, 2003; Jacob et al., 2008; Xue et al., 2007; Zhou et al., 2011).”
• Multi-Stage Multi-Task Feature Learning (2012, neutral): “We evaluate the four multi-task feature learning algorithms in terms of normalized mean squared error (nMSE) and averaged means squared error (aMSE), which are commonly used in multi-task learning problems (Zhang and Yeung, 2010; Zhou et al., 2011; Gong et al., 2012).”
• SpotTune: Transfer Learning Through Adaptive Fine-Tuning (2018, methodology): “Early methods were designed for shallow classification models [52, 17, 36], while more recent approaches address the problem of “with whom” each task should share features using deep neural networks [29, 32].”
• Fully-Adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification (2016, neutral): “A few methods have addressed the problem of “with whom” each task should share features [45, 16, 51, 18, 21, 26].”
• Multi-Task Learning as Multi-Objective Optimization (2018, methodology): “We summarize the work most closely related to ours and refer the interested reader to reviews by Ruder (2017) and Zhou et al. (2011b) for additional background.”
• Federated Kernelized Multi-Task Learning (2018, methodology): “This formulation learns the relationships between tasks [5, 7, 17, 18] while also performing L 2 regularization on each local model.”
• Convex Multitask Learning with Flexible Task Clusters (2012, background): “Interestingly, it is recently shown that this clustered MTL approach is equivalent to alternating structure optimization (Ando & Zhang, 2005) that assumes the tasks share a low-dimensional structure (Zhou et al., 2011).”
• Hierarchical Clustering Multi-Task Learning for Joint Human Action Grouping and Recognition (2017, methodology): “Differing from clustered multi-task learning (CMTL) [12] which performs task grouping and modeling in one shot, the proposed method decomposes this joint work into two sub-tasks: 1) multi-task learning: we formulate the objective function for”
• Privacy-Preserving Distributed Multi-Task Learning with Asynchronous Updates (2017, background): “Several types of shared representations can be encountered in MTL literature such as shared feature subspace [2, 11], shared feature subset [12, 31, 46, 50], clustering structure [47] and shared parameters in neural networks [8].”
• Multi-Task Learning with Low Rank Attribute Embedding for Multi-Camera Person Re-Identification (2018, background): “There are some representative works concerning Multi-Task Learning, including clustered MTL [49], Robust MTL [50], trace norm regularization [51], and [52].”
• Learning to Transfer: Transferring Latent Task Structures and Its Application to Person-Specific Facial Action Unit Detection (2015, background): “Recently, some works have aimed at automatically estimating the structure of relations among the different tasks [11, 13, 29].”
• On Multiplicative Multitask Feature Learning (2014, unknown): “Note that the feature sharing patterns may not be revealed by the recent methods on clustered multitask learning that cluster tasks into groups [10, 8, 23] because no cluster structure is present in Figure 1b, for instance.”
• Learning to Share Latent Tasks for Action Recognition (2013, background): “Multi-task learning has been shown to improve the generalization capability of each single task in the machine learning community [4, 6, 33].”
• Interactive Multi-task Relationship Learning (2016, background): “Examples of such task relatedness regularizations include shared sets of features via sparsity inducing norms [23], shared low-dimensional subspace via the nuclear norm [5], and clustering structures via spectral [34].”
• Smart Multitask Bregman Clustering and Multitask Kernel Clustering (2015, methodology): “One way is to group tasks into several clusters in which the tasks in the same group are regarded as related [Bakker and Heskes 2003; Jacob et al. 2008; Romera-Paredes et al. 2012; Zhou et al. 2011].”
• AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning (2019, methodology): “Early methods have studied feature sharing among tasks using shallow classification models [30, 24, 66, 69, 41].”