An Overview of Multi-task Learning

By Yu Zhang, Qiang Yang. 2018

1,909 citations recorded by the source corpus.

As a promising area in machine learning, multi-task learning (MTL) aims to improve the performance of multiple related learning tasks by leveraging useful information among them. In this paper, we give an overview of MTL by first giving a definition of MTL. Then several different settings of MTL are introduced, including multi-task supervised learning, multi-task unsupervised learning, multi-task semi-supervised learning, multi-task active learning, multi-task reinforcement learning, multi-task online learning and multi-task multi-view learning. For each setting, representative MTL models are presented. In order to speed up the learning process, parallel and distributed MTL models are introduced. Many areas, including computer vision, bioinformatics, health informatics, speech, natural language processing, web applications and ubiquitous computing, use MTL to improve the performance of the applications involved and some representative works are reviewed. Finally, recent theoretical analyses for MTL are presented.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Representation & Self-Supervised Learning

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, neutral): “There has been extensive prior work on multi-task learning (Caruana, 1997; Ruder, 2017).”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “An alternative approach, called “multi-task learning” [Ruder, 2017; Caruana, 1997], is to train the model on multiple tasks at a time.”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, background): “Training on a large number of tasks is known to help regularize multi-task models (Ruder, 2017).”
WT5?! Training Text-to-Text Models to Explain their Predictions (2020, background): “The “sentiment:” prefix tells the model what task it should perform, which is useful in multi-task models (Caruana, 1997; Ruder, 2017).”
Meta-learning with differentiable closed-form solvers (2018, background): “Our work – and more generally the few-shot learning literature as a whole – is related to the multi-task learning paradigm (Caruana, 1998; Ruder, 2017).”
A Survey on Multi-Task Learning (2017, methodology): “For an introduction to MTL without technical details, please refer to [6].”
Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions (2022, background): “…in the proliferation of new concepts and settings such as out-of-distribution generalization [15], domain generalization [3], multi-task learning [39], continual learning [23] and etc. Continual learning , which is the focus of this paper, concerns learning through a sequence of environments,…”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…are closely related to the multitask learning literature, where the relationship between task similarity and network performance in terms of accuracy and time has been studied (Zhang & Yang, 2021; Ruder, 2017; Standley et al., 2020; Liu, Johns, & Davison, 2019; Kendall, Gal, & Cipolla, 2018).”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “In modern day deep networks, MTL thus culminates in the question of how to exactly share the abundant amount of parameters in the architectural hierarchy, see e.g. the overview provided by Ruder (2017) for variants of sharing architecture portions.”
A Comprehensive Survey on Transfer Learning (2019, background): “The thought of multitask learning [18] is to jointly learn a group of related tasks.”
Multi-Passage Machine Reading Comprehension with Cross-Passage Answer Verification (2018, methodology): “Therefore, we propose to train them together as multi-task learning (Ruder, 2017).”
Multi-Task Learning as Multi-Objective Optimization (2018, methodology): “We summarize the work most closely related to ours and refer the interested reader to reviews by Ruder (2017) and Zhou et al. (2011b) for additional background.”