Multi-Task Learning for Dense Prediction Tasks: A Survey

By Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, Luc Van Gool. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020

994 citations recorded by the source corpus.

With the advent of deep learning, many dense prediction tasks, i.e., tasks that produce pixel-level predictions, have seen significant performance improvements. The typical approach is to learn these tasks in isolation, that is, a separate neural network is trained for each individual task. Yet, recent multi-task learning (MTL) techniques have shown promising results w.r.t. performance, computations and/or memory footprint, by jointly tackling multiple tasks through a learned shared representation. In this survey, we provide a well-rounded view on state-of-the-art deep learning approaches for MTL in computer vision, explicitly emphasizing on dense prediction tasks. Our contributions concern the following. First, we consider MTL from a network architecture point-of-view. We include an extensive overview and discuss the advantages/disadvantages of recent popular MTL models. Second, we examine various optimization methods to tackle the joint learning of multiple tasks. We summarize the qualitative elements of these works and explore their commonalities and differences. Finally, we provide an extensive experimental evaluation across a variety of dense prediction benchmarks to examine the pros and cons of the different methods, including both architectural and optimization based strategies.

View Paper: https://www.semanticscholar.org/paper/00e6834700e9805cb1618433f05915c261a6ba08

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

DeFRCN: Decoupled Faster R-CNN for Few-Shot Object Detection (2021, methodology): “Specifically, with utilizing the feature maps extracted from hard-parameter shared [42] backbone, RPN aims at generating class-agnostic region proposals to tell the network where to look, while RCNN targets to perform region-based detection category by category to identify what to look.”
Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans (2021, background): “Recent work has demonstrated that MTL methods developed on existing datasets seem to specialize to their respective developement set and do not perform well on large, realistic datasets, or on other tasks [62, 63, 73].”
FairMOT: On the Fairness of Detection and Re-identification in Multiple Object Tracking (2020, result): “Similar results can be found in Wang et al. (2020b), Vandenhende et al. (2021).”
Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task (2021, background): “Multitask learning aims to improve generalization by leveraging domain-speciﬁc information contained in the training signals of related tasks (Vandenhende et al., 2020).”
MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022, background): “A common approach for multi-task learning is to use a single encoder to learn a shared representation followed by multiple task-specific decoders [31, 84].”
Rethinking the Competition Between Detection and ReID in Multiobject Tracking (2020, background): “Multi-object tracking (MOT), aiming to estimate the locations and scales of multiple targets in a video sequence, is one of the most fundamental yet challenging tasks in computer vision [1].”
Conflict-Averse Gradient Descent for Multi-task Learning (2021, methodology): “However, learning multiple tasks simultaneously can be a challenging optimization problem because it involves multiple objectives [38].”