Multi-Stage Multi-Task Feature Learning

By Pinghua Gong, Jieping Ye, Changshui Zhang. 2012

164 citations recorded by the source corpus.

Multi-task sparse feature learning aims to improve the generalization performance by exploiting the shared features among tasks. It has been successfully applied to many applications including computer vision and biomedical informatics. Most of the existing multi-task sparse feature learning algorithms are formulated as a convex sparse regularization problem, which is usually suboptimal, due to its looseness for approximating an ℓ0-type regularizer. In this paper, we propose a non-convex formulation for multi-task sparse feature learning based on a novel regularizer. To solve the non-convex optimization problem, we propose a Multi-Stage Multi-Task Feature Learning (MSMTFL) algorithm. Moreover, we present a detailed theoretical analysis showing that MSMTFL achieves a better parameter estimation error bound than the convex formulation. Empirical studies on both synthetic and real-world data sets demonstrate the effectiveness of MSMTFL in comparison with the state of the art multi-task sparse feature learning algorithms.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qiang Yang
2. Yu Zhang
3. Dacheng Tao
4. Lei Han
5. Yu Zhang
6. Jiayu Zhou
7. Tongliang Liu
8. Changshui Zhang

In Computer Vision and Pattern Recognition

1. Qiang Yang
2. Dacheng Tao
3. Yu Zhang
4. Tongliang Liu
5. Lei Han
6. Yu Zhang
7. Qiang Zhou
8. Qi Zhao

In Representation & Self-Supervised Learning

1. Qiang Yang
2. Yu Zhang
3. Dacheng Tao
4. Lei Han
5. Yu Zhang
6. Qiang Zhou
7. Qi Zhao
8. Jiayu Zhou

Citations in the corpus

• An Overview of Multi-task Learning (2018, neutral): “:3) di = 1 min( (cid:5) w i (cid:5) p , θ ), is proposed in [11].”
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, background): “Another work [28] which also mentioned the multi-stage concept used it to solve optimization problem in feature learning, which is very different from our multi-stage strategy here.”
• A Survey on Multi-Task Learning (2017, background): “[33] propose a capped-`p,1 penalty for multi-task feature selection where p = 1 or 2 and the objective function is formulated as”
• Multi-Stage Multi-Task Learning with Reduced Rank (2016, background): “Recently, the capped norms (Zhang 2009; 2010; 2011; Gong, Ye, and Zhang 2012; Sun, Xiang, and Ye 2013) have gained the popularity since those norms are capable of penalizing part of the parameters lower than a threshold.”
• On Multiplicative Multitask Feature Learning (2014, background): “Recent research applies the capped (cid:96) 1 , 1 norm as a nonconvex joint regularizer [5].”
• A General Iterative Shrinkage and Thresholding Algorithm for Non-convex Regularized Optimization Problems (2013, methodology): “In addition, we plan to apply the proposed algorithm to solve the multi-task feature learning problem (Gong et al., 2012a;b).”
• Regularize, Expand and Compress: NonExpansive Continual Learning (2020, background): “The sparsity-inducing norm [8] has been studied in this paper to select such discriminative parameter sub-set by incorporating inherent correlations among multiple tasks.”
• On Better Exploring and Exploiting Task Relationships in Multitask Learning: Joint Model and Feature Learning (2018, methodology): “In the experiment, the labels of English letters are treated as the regression values following the same setup as used in [38].”
• Flexible Clustered Multi-Task Learning by Learning Representative Tasks (2016, methodology): “Among various MTL methods, clustered multi-task learning (CMTL) assumes that all tasks can be clustered into groups and attempts to learn the underlying cluster structure from the training data.”
• Efficient multi-task feature learning with calibration (2014, background): “…shared features among tasks; examples include group lasso multi-task feature learning [1, 19, 23, 25, 29, 33, 34], tree-guided group lasso multi-task feature learning [18], composite regularized multi-task feature learning [14, 16] and non-convex regularized multi-task feature learning [13, 15].”