Robust multi-task feature learning

By Pinghua Gong, Jieping Ye, Changshui Zhang. 2012

317 citations recorded by the source corpus.

Multi-task learning (MTL) aims to improve the performance of multiple related tasks by exploiting the intrinsic relationships among them. Recently, multi-task feature learning algorithms have received increasing attention and they have been successfully applied to many applications involving high dimensional data. However, they assume that all tasks share a common set of features, which is too restrictive and may not hold in real-world applications, since outlier tasks often exist. In this paper, we propose a Robust Multi-Task Feature Learning algorithm (rMTFL) which simultaneously captures a common set of features among relevant tasks and identifies outlier tasks. Specifically, we decompose the weight (model) matrix for all tasks into two components. We impose the well-known group Lasso penalty on row groups of the first component for capturing the shared features among relevant tasks. To simultaneously identify the outlier tasks, we impose the same group Lasso penalty but on column groups of the second component. We propose to employ the accelerated gradient descent to efficiently solve the optimization problem in rMTFL, and show that the proposed algorithm is scalable to large-size problems. In addition, we provide a detailed theoretical analysis on the proposed rMTFL formulation. Specifically, we present a theoretical bound to measure how well our proposed rMTFL approximates the true evaluation, and provide bounds to measure the error between the estimated weights of rMTFL and the underlying true weights. Moreover, by assuming that the underlying true weights are above the noise level, we present a sound theoretical result to show how to obtain the underlying true shared features and outlier tasks (sparsity patterns). Empirical studies on both synthetic and real-world data demonstrate that our proposed rMTFL is capable of simultaneously capturing shared features among tasks and identifying outlier tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qiang Yang
2. Yu Zhang
3. Dacheng Tao
4. Lei Han
5. Yu Zhang
6. Yu Zhang
7. Qi Tian
8. Jiayu Zhou

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Yi Yang
3. Qi Tian
4. Luc Van Gool
5. Ming-Hsuan Yang
6. Xiaogang Wang
7. Qiang Yang
8. Liang Zheng

In Multimodal Learning

1. Qiang Yang
2. Yu Zhang
3. Dacheng Tao
4. Lei Han
5. Yu Zhang
6. Qiang Zhou
7. Yu Zhang
8. Qi Zhao

In Representation & Self-Supervised Learning

1. Qiang Yang
2. Yu Zhang
3. Yu Zhang
4. Lei Han
5. Yu Zhang
6. Dacheng Tao
7. Qiang Zhou
8. Qi Zhao

Citations in the corpus

• An Overview of Multi-task Learning (2018, methodology): “Based on Table 1, we can see that the choices of g ( U ) in [53,56] make U row-sparse via the (cid:2) ∞ ,1 and (cid:2) 2,1 norms, respectively.”
• A Survey on Multi-Task Learning (2017, methodology): “In [172], a multi-task decomposition dictionary learning method that follows the idea of [103] is applied to multi-view tracking.”
• Multi-Stage Multi-Task Feature Learning (2012, neutral): “We evaluate the four multi-task feature learning algorithms in terms of normalized mean squared error (nMSE) and averaged means squared error (aMSE), which are commonly used in multi-task learning problems (Zhang and Yeung, 2010; Zhou et al., 2011; Gong et al., 2012).”
• Multi-Task CNN Model for Attribute Prediction (2015, background): “They address such assumptions by simultaneously capturing the shared features among tasks and identifying outliers through introducing an outlier matrix [16].”
• Multi-Task Learning and Algorithmic Stability (2015, background): “…and Ye 2010; Chen, Zhou, and Ye 2011; Chen, Liu, and Ye 2012) one component is to capture the low-rank property of the model parameters and another one learns the (group) sparse parameters but in (Jalali et al. 2010; Gong, Ye, and Zhang 2012) both components learn the (group) sparse parameters.”
• Learning Multi-Level Task Groups in Multi-Task Learning (2015, methodology): “Similar assumptions are commonly used in the MTL literature, e.g., (Chen, Zhou, and Ye 2011; Gong, Ye, and Zhang 2012).”
• Flexible Clustered Multi-Task Learning by Learning Representative Tasks (2016, methodology): “APG has been extensively used to solve machine learning problems [11], [12], [24], [59], [60], [61] due to the optimal convergence rate among all first-order methods.”
• No Matter Where You Are: Flexible Graph-Guided Multi-task Learning for Multi-view Head Pose Classification under Target Motion (2013, background): “in the form of a graph) defining task dependencies [6] or learn the task relationships simultaneously with task-specific parameters [11, 24, 10, 25, 9].”
• Robust Online Multi-Task Learning with Correlative and Personalized Structures (2017, methodology): “However, such a constraint may be too restrictive and rarely hold in the real-life applications, as the personalized taskswith individual traits often exist [13].”
• Tracking via Robust Multi-task Multi-view Joint Sparse Representation (2013, methodology): “Based on the fact that most of the particles are relevant and outliers often exist, we introduce a robust multitask learning scheme [9] to capture the underlying relationships shared by all tasks.”
• Online Multi-Task Learning Framework for Ensemble Forecasting (2017, background): “(a) Low-rank representation: This category of methods assumes that the models for different tasks share a low-rank representation, either additively [18], [19], [20] or multiplicatively [21], [22], [23].”
• Multi-Task Learning with Low Rank Attribute Embedding for Person Re-Identification (2015, methodology): “Representative work includes clustered MTL [51], Robust MTL [13] and trace norm regularization [20].”
• Robust Dynamic Trajectory Regression on Road Networks: A Multi-task Learning Framework (2014, background): “Now the goal is to learn the weights w ∈ Rd and then predict the total travel cost for an arbitrary trajectory.”
• On Multiplicative Multitask Feature Learning (2014, methodology): “Another method applies the (cid:96) 1 , 2 -norm both row-wisely to one component and column-wisely to the other [6].”
• Efficient multi-task feature learning with calibration (2014, background): “(18) has the following closed-form solution [14, 23]: respectively.”