Integrating low-rank and group-sparse structures for robust multi-task learning

By Jianhui Chen, Jiayu Zhou, Jieping Ye. 2011

344 citations recorded by the source corpus.

Multi-task learning (MTL) aims at improving the generalization performance by utilizing the intrinsic relationships among multiple related tasks. A key assumption in most MTL algorithms is that all tasks are related, which, however, may not be the case in many real-world applications. In this paper, we propose a robust multi-task learning (RMTL) algorithm which learns multiple tasks simultaneously as well as identifies the irrelevant (outlier) tasks. Specifically, the proposed RMTL algorithm captures the task relationships using a low-rank structure, and simultaneously identifies the outlier tasks using a group-sparse structure. The proposed RMTL algorithm is formulated as a non-smooth convex (unconstrained) optimization problem. We propose to adopt the accelerated proximal method (APM) for solving such an optimization problem. The key component in APM is the computation of the proximal operator, which can be shown to admit an analytic solution. We also theoretically analyze the effectiveness of the RMTL algorithm. In particular, we derive a key property of the optimal solution to RMTL; moreover, based on this key property, we establish a theoretical bound for characterizing the learning performance of RMTL. Our experimental results on benchmark data sets demonstrate the effectiveness and efficiency of the proposed algorithm.

Subfields: Artificial Intelligence, Computational Mechanics, Electrical and Electronic Engineering, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Electrical and Electronic Engineering

In Optimization & Training

Citations in the corpus

An Overview of Multi-task Learning (2018, background): “The choices of g (U) in [54,55] enforce U to be low-rank via the trace norm as the regularizer and constraint, respectively.”
Cross-Stitch Networks for Multi-task Learning (2016, background): “However, most of such work, with publicly available code, formulates multi-task learning in an optimization framework that requires all data points in memory [6, 14, 23, 34, 49, 60, 61].”
A Survey on Multi-Task Learning (2017, methodology): “gi(·)’s in [102] with CW = ∅ are defined as”
Federated Multi-Task Learning (2017, background): “The ﬁrst (e.g., [13, 4, 10, 23]) assumes that a clustered, sparse, or low-rank structure between the tasks is known a priori .”
Learning Task Grouping and Overlap in Multi-task Learning (2012, methodology): “Other methods assume that there is one group of re-lated tasks and a small number of outlier tasks that are not related to any task in the pool (Yu et al., 2007; Chen et al., 2011).”
Lifelong Learning for Sentiment Classification (2015, methodology): “ knowledge (Chen and Liu, 2014b,Chen and Liu, 2014a,Ruvolo and Eaton, 2013) across multiple tasks. Multi-task learning optimizes the learning of multiple related tasks at the same time (Caruana, 1997,Chen et al., 2011,Saha et al., 2011,Zhang et al., 2008). However, these methods are not for sentimen”
Multi-Task Learning and Algorithmic Stability (2015, methodology): “For example, in (Chen, Liu, and Ye 2010; Chen, Zhou, and Ye 2011; Chen, Liu, and Ye 2012) one component is to capture the low-rank property of the model parameters and another one learns the (group) sparse parameters but in (Jalali et al. 2010; Gong, Ye, and Zhang 2012) both components learn the…”
Learning Multi-Level Task Groups in Multi-Task Learning (2015, methodology): “If the desired accuracy is ε, i.e., |f̃(W(t)h ) − f̃(W∗h)| ≤ ε, according to (Chen et al. 2011), Algorithm 2 needsO( √ 2D/ε) iterations to converge.”
On Multiplicative Multitask Feature Learning (2014, methodology): “A robust MTFL method in [3] uses the trace norm on one component for mining a low-rank structure shared by tasks and a column-wise `1,2-norm on the other component for identifying task outliers.”
Learning Tree Structure in Multi-Task Learning (2015, neutral): “For example, low-rank structure based methods in [1, 3] can learn model parameters sharing a low dimensional subspace for multiple tasks, probabilistic MTL models [27, 25] place probabilistic priors on multiple tasks to learn the model parameters as well as identifying the task relations, regularize”
Hierarchical Clustering Multi-Task Learning for Joint Human Action Grouping and Recognition (2017, methodology): “from each task to form the training set and used the rest of the samples as the test set as [12], [25].”
Which Looks Like Which: Exploring Inter-class Relationships in Fine-Grained Visual Categorization (2014, methodology): “A popular way of tackling this robust MTL problem is to use a decomposition framework, which forms a learning objective with a structure term and an outlier penalty term [24,16,22].”
Convex Multitask Learning with Flexible Task Clusters (2012, methodology): “…in a Bayesian model (Yu et al., 2005; Zhang & Schneider, 2010; Zhang & Yeung, 2010); that the data follows a dirty model (Jalali et al., 2010); that most of the tasks lie in a low-dimensional subspace (Ando & Zhang, 2005; Chen et al., 2010), or that outlier tasks are present (Chen et al., 2011a).”
Efficient Multitask Feature and Relationship Learning (2017, methodology): “Interestingly, they have close connection to the proximal method proposed in the literature to solve matrix completion (Cai et al., 2010), or Euclidean projection under trace norm constraint (Chen et al., 2011, 2012).”
Interactive Multi-task Relationship Learning (2016, background): “The same framework can accommodate more complicated assumptions such as dirty models [20] and robust models [10, 15].”