Task clustering and gating for bayesian multitask learning

2003

26 citations recorded by the source corpus.

Modeling a collection of similar regression or classification tasks can be improved by making the tasks 'learn from each other'. In machine learning, this subject is approached through 'multitask learning', where parallel tasks are modeled as multiple outputs of the same network. In multilevel analysis this is generally implemented through the mixed-effects linear model where a distinction is made between 'fixed effects', which are the same for all tasks, and 'random effects', which may vary between tasks. In the present article we will adopt a Bayesian approach in which some of the model parameters are shared (the same for all tasks) and others more loosely connected through a joint prior distribution that can be learned from the data. We seek in this way to combine the best parts of both the statistical multilevel approach and the neural network machinery. The standard assumption expressed in both approaches is that each task can learn equally well from any other task. In this article we extend the model by allowing more differentiation in the similarities between tasks. One such extension is to make the prior mean depend on higher-level task characteristics. More unsupervised clustering of tasks is obtained if we go from a single Gaussian prior to a mixture of Gaussians. This can be further generalized to a mixture of experts architecture with the gates depending on task characteristics. All three extensions are demonstrated through application both on an artificial data set and on two real-world problems, one a school problem and the other involving single-copy newspaper sales.

Subfields: General Health Professions, Philosophy

Citations in the corpus

• Discriminative k-shot learning using probabilistic models (2017, positive): “It subsumes two existing approaches in this vein [5, 6], and is motivated by similar ideas from multi-task learning [7].”
• Meta-Learning: A Survey (2018, neutral): “This is a generally applicable idea, and transfer learning approaches have been proposed for kernel methods [41, 42], parametric Bayesian models [139, 122, 8], Bayesian networks [107], clustering [167] and reinforcement learning [62, 36].”
• Bayesian Online Multitask Learning of Gaussian Processes (2010, unknown): “Further investigations demonstrated the potential advantage of multitask approaches against those that learn the single functions separately (single-task approach) [17], [18].”
• Inductive transfer with context-sensitive neural networks (2008, methodology): “Research on MTL has occurred using traditional machine learning methods (Bakker and Heskes 2003; Caruana 1997; Baxter 1997; Heskes 2000; Thrun and Pratt 1997; Ben-David and Schuller 2003), statistical regression methods (Greene 2002; Zellner 1962; Breiman and Friedman 1998), Bayesian methods involvi”
• Safe Screening for Multi-Task Feature Learning with Multiple Data Matrices (2015, unknown): “…have shown that learning multiple related tasks (MTL) simultaneously often provides superior predictive performance relative to learning each tasks independently (Ando and Zhang, 2005, Argyriou et al., 2008, Bakker and Heskes, 2003, Evgeniou et al., 2005, Zhang et al., 2006, Chen et al., 2013).”
• Learning Multiple Tasks with Kernel Methods (2005, methodology): “For comparison we also report on the performance of the task clustering method described in (Bakker and Heskes, 2003) – the dashed line in the figure.”
• Regularized multi--task learning (2004, methodology): “We selected this dataset to compare our method directly with the work of [4] where a number of multi-task learning methods are compared using this dataset.”