Multi-Stage Multi-Task Learning with Reduced Rank

By Lei Han, Yu Zhang. 2016

66 citations recorded by the source corpus.

Multi-task learning (MTL) seeks to improve the generalization performance by sharing information among multiple tasks. Many existing MTL approaches aim to learn the low-rank structure on the weight matrix, which stores the model parameters of all tasks, to achieve task sharing, and as a consequence the trace norm regularization is widely used in the MTL literature. A major limitation of these approaches based on trace norm regularization is that all the singular values of the weight matrix are penalized simultaneously, leading to impaired estimation on recovering the larger singular values in the weight matrix. To address the issue, we propose a Reduced rAnk MUlti-Stage multi-tAsk learning (RAMUSA) method based on the recently proposed capped norms. Different from existing trace-norm-based MTL approaches which minimize the sum of all the singular values, the RAMUSA method uses a capped trace norm regularizer to minimize only the singular values smaller than some threshold. Due to the non-convexity of the capped trace norm, we develop a simple but well guaranteed multi-stage algorithm to learn the weight matrix iteratively. We theoretically prove that the estimation error at each stage in the proposed algorithm shrinks and finally achieves a lower upper-bound as the number of stages becomes large enough. Empirical studies on synthetic and real-world datasets demonstrate the effectiveness of the RAMUSA method in comparison with the state-of-the-art methods.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xing Xie
2. Qiang Yang
3. Hongwei Wang
4. Yu Zhang
5. Wenjie Li
6. Miao Zhao
7. Fuzheng Zhang
8. Ying Wei

In Computational Mechanics

1. Qiang Yang
2. Yu Zhang
3. Ying Wei
4. Xing Xie
5. Jianhang Zhou
6. Bob Zhang
7. Shaoning Zeng

Citations in the corpus

• An Overview of Multi-task Learning (2018, neutral): “Similar to what the capped-(cid:2) p ,1 penalty did to the (cid:2) p ,1 norm, a variant of the trace-norm regularization called the capped-trace regularizer is proposed in [27] and defined as (cid”
• A Survey on Multi-Task Learning (2017, background): “(20) implies that WTW follows a Wishart distribution asW(·|0,Ω), Zhang and Yeung [91] generalize it as (WTW)t ∼ W(·|0,Ω), (22) where t is a positive integer to model high-order task relationships.”
• Learning Sparse Task Relations in Multi-Task Learning (2017, methodology): “The task clustering approach such as (Xue et al. 2007; Jacob, Bach, and Vert 2008; Kang, Grauman, and Sha 2011; Jawanpuria and Nath 2012; Han and Zhang 2015a) can detect the cluster structure among tasks where tasks from a cluster are similar to each other in terms of model parameters or feature rep”
• Learning to Multitask (2018, background): “This approach can be further classified into four categories: low-rank approach [1, 28, 17], task clustering approach [18, 20, 15], task relation learning approach [39, 40, 36, 35, 41, 42, 21, 37], and decomposition approach [10, 19, 44, 16].”
• Multi-Task Feature Learning for Knowledge Graph Enhanced Recommendation (2019, background): “performance improvement compared with learning them individually. In general, MTL algorithms can be classified into several categories, including feature learning approach [34, 41], low-rank approach [7, 16], task clustering approach [47], task relation learning approach [12], and decomposition appr”
• Two-stage Image Classification Supervised by a Single Teacher Single Student Model (2019, methodology): “The RAMUSA [8] algorithm performed multi-task learning by using a multi-stage method, which is similar to the idea of two-stage methods.”
• Graph‐guided multi‐task sparse learning model: a method for identifying antigenic variants of influenza A(H3N2) virus (2018, unknown): “We proposed to choose / from a candidate set 104; 103; . . . ; 10 2 which are common practice used by sparse learning methods (Han and Zhang, 2015, 2016; Han et al., 2016), and a from 0:1; 0:2; . . . ;0:9½ (because a is a value in the interval 0; 1½ ) via 10- fold cross-validation.”