Learning Task Grouping and Overlap in Multi-task Learning

By Abhishek Kumar, Hal Daumé Iii. 2012

541 citations recorded by the source corpus.

In the paradigm of multi-task learning, multiple related prediction tasks are learned jointly, sharing information across the tasks. We propose a framework for multi-task learning that enables one to selectively share the information across the tasks. We assume that each task parameter vector is a linear combination of a finite number of underlying basis tasks. The coefficients of the linear combination are sparse in nature and the overlap in the sparsity patterns of two tasks controls the amount of sharing across these. Our model is based on the assumption that task parameters within a group lie in a low dimensional subspace but allows the tasks in different groups to overlap with each other in one or more bases. Experimental results on four datasets show that our approach outperforms competing methods.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Multi-task Sequence to Sequence Learning (2015, background): “Such framework has been widely studied by Thrun (1996); Caruana (1997); Evgeniou & Pontil (2004); Ando & Zhang (2005); Argyriou et al. (2007); Kumar & III (2012), among many others.”
An Overview of Multi-task Learning (2018, methodology): “Both works [33,35] decomposeW asW = LS where columns in L consist of basis parameter vectors in different clusters and S contains combination coefficients.”
LEAF: A Benchmark for Federated Settings (2018, background): “This paradigm is also amenable to federated settings [30] but, contrary to realistic federated networks, is usually explored in regimes with small numbers of tasks and samples, e.g., the popular Landmine Detection [38, 26, 36, 30], Computer Survey [2, 1, 14] and Inner London Education Authority Scho”
A Survey on Multi-Task Learning (2017, methodology): “In [162], the Go-MTL method [58] is generalized to the multilinear multi-task setting [107], [108] for person-specific facial action unit prediction, where each facial action unit is treated as a task and in each task, a function is learned for a person.”
Federated Multi-Task Learning (2017, background): “This can be naturally captured through a multi-task learning (MTL) framework, where the goal is to consider ﬁtting separate but related models simultaneously [13, 2, 54, 27].”
Lifelong Learning with Dynamically Expandable Networks (2017, neutral): “Eaton & Ruvolo (2013) suggest an online lifelong learning framework (ELLA) that is based on an existing multi-task learning formulation (Kumar & Daume III, 2012) that efﬁciently updates latent parameter bases for a sequence of tasks, by removing dependency to previous tasks for the learning of each…”
Model Zoo: A Growing Brain That Learns Continually (2021, background): “There are a number of algorithmic tools to estimate task relatedness, e.g., (Evgeniou et al., 2005; Cavallanti et al., 2010; Kumar and Daume III, 2012), and although such methods are popular in transfer learning (Pentina and Lampert, 2015; Jaakkola and Haussler, 1999), one cannot apply them in…”
Online Coreset Selection for Rehearsal-based Continual Learning (2021, neutral): “In light of this need, continual learning (CL) [40], or lifelong learning, tackles a learning scenario where a model continuously learns over a sequence of tasks [18, 25] within a broad research area, such as classification [15, 9], image generation [47], language learning [24, 4], clinical applicat”
Federated Continual Learning with Weighted Inter-client Transfer (2020, methodology): “Continual learning (1; 2; 3; 4; 5) describes a learning scenario where a model continuously trains on a sequence of tasks; it is inspired by the human learning process, as a person learns to perform numerous tasks with large diversity over his/her lifespan, making use of the past knowledge to learn ”
A PAC-Bayesian bound for Lifelong Learning (2013, methodology): “We optimize (17), and due to the squared loss, the problem has a closed form solution: To make results comparable with the baseline algorithms, we report the squared error multiplied by the squared value of the maximum examination score.”
Task Agnostic Continual Learning via Meta Learning (2019, background): “Synaptic Intelligence (SI) by Zenke et al. (2017) proposed to estimate the importance of parameters by the path length of the updates on the previous task, then discourage future changes on important parameters by a quadratic penalty.”
Efficient Representations for Lifelong Learning and Autoencoding (2014, background): “Recent work of [19, 16] considers the problem of learning multiple linear separators that share a common low-dimensional subspace in the batch setting where all tasks are given up front.”