Robust Online Multi-Task Learning with Correlative and Personalized Structures

By Peng Yang, Peilin Zhao, Xin Gao. 2017

19 citations recorded by the source corpus.

Multi-Task Learning (MTL) can enhance a classifier’s generalization performance by learning multiple related tasks simultaneously. Conventional MTL works under the offline or batch setting, and suffers from expensive training cost and poor scalability. To address such inefficiency issues, online learning techniques have been applied to solve MTL problems. However, most existing algorithms of online MTL constrain task relatedness into a presumed structure via a single weight matrix, which is a strict restriction that does not always hold in practice. In this paper, we propose a robust online MTL framework that overcomes this restriction by decomposing the weight matrix into two components: The first one captures the low-rank common structure among tasks via a nuclear norm and the second one identifies the personalized patterns of outlier tasks via a group lasso. Theoretical analysis shows the proposed algorithm can achieve a sub-linear regret with respect to the best linear model in hindsight. Even though the above framework achieves good performance, the nuclear norm that simply adds all nonzero singular values together may not be a good low-rank approximation. To improve the results, we use a log-determinant function as a non-convex rank approximation. The gradient scheme is applied to optimize log-determinant function and can obtain a closed-form solution for this refined problem. Experimental results on a number of real-world applications verify the efficacy of our method.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qiang Yang
2. Yu Zhang
3. Peilin Zhao
4. Peng Yang
5. Xin Gao
6. Jiayu Zhou
7. Mingsheng Long
8. Jianmin Wang

In Computational Mechanics

1. Qiang Yang
2. Yu Zhang
3. Xin Gao
4. Peilin Zhao
5. Peng Yang
6. Jiayu Zhou
7. Peter Richtárik
8. Jakub Konecný

Citations in the corpus

• A Survey on Multi-Task Learning (2017, methodology): “[183], an online multi-task model is proposed to learn both a low-rank component and a group sparse component to characterize task relations.”
• Confidence Weighted Multitask Learning (2019, methodology): “…in multitask learning focus on how to take advantage of the task relationship, either by sharing model parameters via regularization techniques (Argyriou, Evgeniou, and Pontil 2008; Zhang and Yeung 2010; Yang, Zhao, and Gao 2017) or learning cross-task data directly (Crammer and Mansour 2012).”