An Overview of Multi-Task Learning in Deep Neural Networks

By Sebastian Ruder. 2017

3,307 citations recorded by the source corpus.

Multi-task learning (MTL) has led to successes in many applications of machine learning, from natural language processing and speech recognition to computer vision and drug discovery. This article aims to give a general overview of MTL, particularly in deep neural networks. It introduces the two most common methods for MTL in Deep Learning, gives an overview of the literature, and discusses recent advances. In particular, it seeks to help ML practitioners apply MTL by shedding light on how MTL works and providing guidelines for choosing appropriate auxiliary tasks.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, neutral): “There has been extensive prior work on multi-task learning (Caruana, 1997; Ruder, 2017).”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “An alternative approach, called “multitask learning” (Ruder, 2017; Caruana, 1997), is to train the model on multiple tasks at a time.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “…2014; Bahdanau et al., 2014) have been widely adopted as the state-of-the-art approach for machine translation, both in the research community (Bojar et al., 2016a, 2017, 2018b) and for large-scale production systems (Wu et al., 2016; Zhou et al., 2016; Crego et al., 2016; Hassan et al., 2018).”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, background): “Training on a large number of tasks is known to help regularize multi-task models (Ruder, 2017).”
Finetuned Language Models Are Zero-Shot Learners (2021, background): “…be seen as a formulation of multitask learning (MTL), which is an established area within deep learning (Collobert et al., 2011; Luong et al., 2016; Ruder, 2017; Velay & Daniel, 2018; Clark et al., 2019b; Liu et al., 2019b, inter alia)—see Worsham & Kalita (2020) for a recent survey on MTL for…”
DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning (2021, neutral): “One of the key problems in MTL is how to share model parameters between tasks [31].”
Avoiding catastrophic forgetting in mitigating model biases in sentence-pair classification with elastic weight consolidation (2020, background): “parameters so that they do not deviate too much from those learned in the original task, which is also a motivation in multi-task training (Ruder, 2017).”
Learning from Dialogue after Deployment: Feed Yourself, Chatbot! (2019, background): “Identifying optimal task structure in multi-task learning (MTL) architectures is an open research problem (Ruder, 2017).”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “Our proposed approach is also motivated by learning using privileged information (LUPI) paradigm [45, 51] and multi-task learning strategies in deep neural networks [2, 43] that share representations between closely related tasks for enhanced learning performance.”
WT5?! Training Text-to-Text Models to Explain their Predictions (2020, background): “The “sentiment:” prefix tells the model what task it should perform, which is useful in multi-task models (Caruana, 1997; Ruder, 2017).”
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, background): “…with the aim of learning a shared representation that will enable the model to generalize better on each task (Caruana, 1997; Collobert and Weston, 2008; Nam et al., 2014; Liu et al., 2016, 2017; Zhang and Yang, 2017; Ruder, 2017; Ruder et al., 2019; Sanh et al., 2019; Pfeiffer et al., 2020a).”
MetaICL: Learning to Learn In Context (2021, methodology): “…objective, MetaICL achieves substantial improvements even with smaller LMs. Meta-training via multi-task learning Our work is broadly inspired by a large body of work in meta-learning (Vilalta and Drissi, 2002; Finn et al., 2017) and multi-task learning (Evgeniou and Pontil, 2004; Ruder, 2017).”
Meta-learning with differentiable closed-form solvers (2018, background): “Our work – and more generally the few-shot learning literature as a whole – is related to the multi-task learning paradigm (Caruana, 1998; Ruder, 2017).”
Toward Causal Representation Learning (2021, background): “Multitask learning refers to building a system that can solve multiple tasks across different environments [41], [209].”
Unsupervised Adversarial Invariance (2018, methodology): “Our method shares some properties with multi-task learning (MTL) [17] in the sense that the model is trained with multiple objectives.”
Towards Causal Representation Learning (2021, neutral): “multiple tasks across different environments [40, 209].”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…are closely related to the multitask learning literature, where the relationship between task similarity and network performance in terms of accuracy and time has been studied (Zhang & Yang, 2021; Ruder, 2017; Standley et al., 2020; Liu, Johns, & Davison, 2019; Kendall, Gal, & Cipolla, 2018).”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “+ positive transfer – negative transfer – task boundaries – off-line learning (Caruana, 1997) (Zhang and Yang, 2017) (Ruder, 2017) (McCann et al.”