Task Clustering and Gating for Bayesian Multitask Learning

By Bart J. Bakker, Tom M. Heskes. 2003

659 citations recorded by the source corpus.

Modeling a collection of similar regression or classification tasks can be improved by making the tasks 'learn from each other'. In machine learning, this subject is approached through 'multitask learning', where parallel tasks are modeled as multiple outputs of the same network. In multilevel analysis this is generally implemented through the mixed-effects linear model where a distinction is made between 'fixed effects', which are the same for all tasks, and 'random effects', which may vary between tasks. In the present article we will adopt a Bayesian approach in which some of the model parameters are shared (the same for all tasks) and others more loosely connected through a joint prior distribution that can be learned from the data. We seek in this way to combine the best parts of both the statistical multilevel approach and the neural network machinery. The standard assumption expressed in both approaches is that each task can learn equally well from any other task. In this article we extend the model by allowing more differentiation in the similarities between tasks. One such extension is to make the prior mean depend on higher-level task characteristics. More unsupervised clustering of tasks is obtained if we go from a single Gaussian prior to a mixture of Gaussians. This can be further generalized to a mixture of experts architecture with the gates depending on task characteristics. All three extensions are demonstrated through application both on an artificial data set and on two real-world problems, one a school problem and the other involving single-copy newspaper sales.

Subfields: General Health Professions, Representation & Self-Supervised Learning, Philosophy

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In General Health Professions

In Representation & Self-Supervised Learning

In Philosophy

Citations in the corpus

An Overview of Multi-task Learning (2018, methodology): “Unlike[29,30],whichareBayesianmodels,there are several regularized methods [31–35] to do task clustering.”
Meta-Learning Probabilistic Inference for Prediction (2018, methodology): “First, we leverage shared statistical structure between tasks via hierarchical probabilistic models developed for multi-task and transfer learning (Heskes, 2000; Bakker and Heskes, 2003).”
A Survey on Multi-Task Learning (2017, background): “Bakker and Heskes [50] propose a multi-task Bayesian neural network model with the network structure similar to Fig.”
Variational Continual Learning (2017, background): “Standard practice in multi-task learning (Bakker and Heskes, 2003) uses networks that share parameters close to the inputs but with separate heads for each output, hence multi-head networks.”
Uncertainty in Multitask Transfer Learning (2018, background): “Hierarchical Bayes algorithms for multitask learning has a long history [8, 32, 2].”
Discriminative k-shot learning using probabilistic models (2017, methodology): “It subsumes two existing approaches in this vein [5, 6], and is motivated by similar ideas from multi-task learning [7].”
Overcoming catastrophic forgetting with hard attention to the task (2018, methodology): “The operation of layer L is equivalent to a multi-head output (Bakker & Heskes, 2003), which has been exploited in the context of catastrophic forgetting by a number of works (for example Rusu et al., 2016; Li & Hoiem, 2017; Nguyen et al., 2017).”
Continual Learning with Adaptive Weights (CLAW) (2019, methodology): “Our model is also related to the multi-task learning approach (Caruana, 1997; Heskes, 2000; Bakker & Heskes, 2003; Adel et al., 2017; Zhao et al., 2019a; Stickland & Murray, 2019; Zhao et al., 2019b).”
Learning Task Grouping and Overlap in Multi-task Learning (2012, methodology): “( Argyriou et al., 2008a; Agarwal et al., 2010; Bakker & Heskes, 2003).”
Gradient Surgery for Multi-Task Learning (2020, background): “Algorithms for multi-task learning typically consider how to train a single model that can solve a variety of different tasks (Caruana, 1997; Bakker & Heskes, 2003; Ruder, 2017).”
Meta-Learning: A Survey (2018, neutral): “, 2005; Evgeniou and Pontil, 2004), parametric Bayesian models (Rosenstein et al., 2005; Raina et al., 2006; Bakker and Heskes, 2003), Bayesian networks (Niculescu-Mizil and Caruana, 2005), clustering (Thrun, 1998) and reinforcement learning (Hengst, 2002; Dietterich et al.”
Clustered Multi-Task Learning: A Convex Formulation (2008, background): “More precisely, we consider the problem of multi-task learn ing, which has recently emerged as a very promising research direction for various application s [4].”
A Convex Formulation for Learning Task Relationships in Multi-Task Learning (2010, methodology): “Another widely studied approach for multi-task learning is the task clustering approach [25, 3, 26].”
GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks (2017, background): “Multitask learning has existed well before the advent of deep learning (Caruana (1998); Bakker & Heskes (2003)), but the robust learned features within deep networks have spurned renewed interest.”
Multi-task learning for cross-platform siRNA efficacy prediction: an in-silico study (2010, background): “Computational framework Multi-task learning has been developed in machine learning research to situations where multiple related learning tasks are accomplished together [38-46].”
Multi-Task Learning as Multi-Objective Optimization (2018, methodology): “In soft parameter sharing, all parameters are task-speciﬁc but they are jointly constrained via Bayesian priors (Xue et al., 2007; Bakker and Heskes, 2003) or a joint dictionary (Argyriou et al., 2007; Long and Wang, 2015; Yang and Hospedales, 2016; Ruder, 2017).”
Recent Advances in Bayesian Optimization (2022, background): “For example, Bakker and Heskes [11] proposed a Bayesian neural network for MTL in a hierarchical Bayesian perspective.”