Multi-Task Learning with Deep Neural Networks: A Survey

By Michael Crawshaw. 2020

804 citations recorded by the source corpus.

Multi-task learning (MTL) is a subfield of machine learning in which multiple tasks are simultaneously learned by a shared model. Such approaches offer advantages like improved data efficiency, reduced overfitting through shared representations, and fast learning by leveraging auxiliary information. However, the simultaneous learning of multiple tasks presents new design and optimization challenges, and choosing which tasks should be learned jointly is in itself a non-trivial problem. In this survey, we give an overview of multi-task learning methods for deep neural networks, with the aim of summarizing both the well-established and most recent directions within the field. Our discussion is structured according to a partition of the existing deep MTL techniques into three groups: architectures, optimization methods, and task relationship learning. We also provide a summary of common multi-task benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, background): “It is crucial to choose which tasks should be learned jointly, and learning multiple tasks simultaneously presents new challenges of design and optimization [25].”
ImageNet-21K Pretraining for the Masses (2021, background): “• Single-task training - training with a softmax (a single loss) makes convergence easy and efficient, and avoids many optimization problems associated with multi-loss learning, such as different gradient magnitudes and gradient interference [52, 5, 7].”
UniT: Multimodal Multitask Learning with a Unified Transformer (2021, background): “There has been a long history of work on multi-task learning [6, 12] in vision ( e.g .”
Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch (2023, neutral): “The superiority of model merging over multi-task learning (Crawshaw, 2020; Zhang & Yang, 2022) (which also intends to obtain one model with several abilities) is that model merging pays attention to the fusion of model parameters without accessing the original training data (Matena & Raffel, 2022;…”
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022, background): “Multitask Learning In multitask learning, a model is trained simultaneously with multiple tasks and a shared representation is learned to capture the commonalities among the related tasks (Crawshaw, 2020; Gao et al., 2022; Chen et al., 2021).”
Efficient Training of Visual Transformers with Small Datasets (2021, background): “, image labels) in order to regularize VT training, hence our framework is a multi-task learning approach [13].”
Efficient Diffusion Training via Min-SNR Weighting Strategy (2023, background): “One of the main challenges in MTL is negative transfer [9], means the joint training of tasks hurts learning instead of helping it.”
Learning a Universal Template for Few-shot Dataset Generalization (2021, background): “The aim in multi-task learning (Crawshaw, 2020), for example, is to learn multiple tasks simultaneously with a shared model, similar to FLUTE’s approach for training on multiple diverse datasets jointly.”
Transformer is All You Need: Multimodal Multitask Learning with a Unified Transformer (2021, background): “There has been a long history of work on multi-task learning [6, 12] in vision ( e.g .”
A Review on Deep Learning in UAV Remote Sensing (2021, background): “Several advantages are mentioned in [42], including fast learning and the minimization of overfitting problems.”
BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers (2022, methodology): “However, the jointly trained model does not perform as well as individually trained models for road and lane segmentation, which is a common phenomenon called negative transfer [11, 13] in multi-task learning.”
Differential Privacy for Deep and Federated Learning: A Survey (2022, background): “As the multi-task learning paradigm [126], [127] is to leverage useful knowledge in multiple tasks to improve the generalization performance of all tasks, the authors propose that each device in FL learns a task-speciﬁc parameter ω i with the objective function f i .”
Planning-oriented Autonomous Driving (2022, background): “However, such a scheme may cause undesirable “negative transfer” [23,64].”
Task Agnostic Representation Consolidation: a Self-supervised based Continual Learning Approach (2022, background): “However, simultaneous learning of multiple objectives poses new design and optimization challenges, and choosing which objectives that should be learned together is in itself a non-trivial problem (Crawshaw, 2020).”