Flexible Modeling of Latent Task Structures in Multitask Learning

By Alexandre Passos, Piyush Rai, Jacques Wainer, Hal Daumé Iii. 2012

53 citations recorded by the source corpus.

Multitask learning algorithms are typically designed assuming some fixed, a priori known latent structure shared by all the tasks. However, it is usually unclear what type of latent task structure is the most appropriate for a given multitask learning problem. Ideally, the "right" latent task structure should be learned in a data-driven manner. We present a flexible, nonparametric Bayesian model that posits a mixture of factor analyzers structure on the tasks. The nonparametric aspect makes the model expressive enough to subsume many existing models of latent task structures (e.g, mean-regularized tasks, clustered tasks, low-rank or linear/non-linear subspace assumption on tasks, etc.). Moreover, it can also learn more general task structures, addressing the shortcomings of such models. We present a variational inference algorithm for our model. Experimental results on synthetic and real-world datasets, on both regression and classification problems, demonstrate the effectiveness of the proposed method.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Learning Task Grouping and Overlap in Multi-task Learning (2012, background): “Recently, (Passos et al., 2012) posited a mixture of sparse factor analyzers structure on the collection of the task weight vectors.”
Deep Multi-task Representation Learning: A Tensor Factorisation Approach (2016, methodology): “, 2007) assumes S·,i (the ith column of S) is a unit vector generated by a Dirichlet Process and (Passos et al., 2012) models W using linear factor analysis with Indian Buffet Process (Griffiths & Ghahramani, 2011) prior on S.”
SpotTune: Transfer Learning Through Adaptive Fine-Tuning (2018, methodology): “Early methods were designed for shallow classification models [52, 17, 36], while more recent approaches address the problem of “with whom” each task should share features using deep neural networks [29, 32].”
Fully-Adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification (2016, neutral): “A few methods have addressed the problem of “with whom” each task should share features [45, 16, 51, 18, 21, 26].”
Flexible Clustered Multi-Task Learning by Learning Representative Tasks (2016, methodology): “Among various MTL methods, clustered multi-task learning (CMTL) assumes that all tasks can be clustered into groups and attempts to learn the underlying cluster structure from the training data.”
A Unified Perspective on Multi-Domain and Multi-Task Learning (2014, methodology): “W = LS – can explain several models: Kumar & Daum ´ e III (2012) is L1/L2 regularised decomposition, Passos et al. (2012) is linear Gaussian model with IBP prior and an earlier study (Xue et al., 2007) assumes s i are unit vectors generated by a Dirichlet Process (DP).”
AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning (2019, background): “Early meth-ods have studied feature sharing among tasks using shallow classiﬁcation models [28, 20, 59, 61, 38].”
A Probabilistic Model for Dirty Multi-task Feature Selection (2015, background): “Finally, some works in the literature also consider modeling outlier tasks in multi-task learning, e.g., (Xue et al., 2007; Passos et al., 2012).”
Active Multitask Learning Using Both Latent and Supervised Shared Topics (2014, methodology): “In [30], the source data is first used to train a classifier, the parameters of which are later updated in an online manner with new examples actively selected from the target domain.”
On Multiplicative Multitask Feature Learning (2014, background): “Recent methods either explore the latent basis that is used to develop the entire set of tasks, or learn how to group the tasks [16, 11], or identify if certain tasks are outliers to other tasks [6].”