Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics

By Alex Kendall, Yarin Gal, Roberto Cipolla. 2017

4,210 citations recorded by the source corpus.

Numerous deep learning applications benefit from multitask learning with multiple regression and classification objectives. In this paper we make the observation that the performance of such systems is strongly dependent on the relative weighting between each task's loss. Tuning these weights by hand is a difficult and expensive process, making multi-task learning prohibitive in practice. We propose a principled approach to multi-task deep learning which weighs multiple loss functions by considering the homoscedastic uncertainty of each task. This allows us to simultaneously learn various quantities with different units or scales in both classification and regression settings. We demonstrate our model learning per-pixel depth regression, semantic and instance segmentation from a monocular input image. Perhaps surprisingly, we show our model can learn multi-task weightings and outperform separate models trained individually on each task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, methodology): “Recently, PanopticDeepLab [20], a simple, fast, and strong approach for bottom-up panoptic segmentation, employs a class-agnostic instance segmentation branch involving a simple instance center regression [45,82,66], coupled with DeepLab semantic segmentation outputs [13,15,16].”
Panoptic Feature Pyramid Networks (2019, background): “In general, using a single network to solve multiple diverse tasks degrades performance [32], but various strategies can mitigate this [29, 42].”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “Then, instance segments are obtained by grouping ‘thing’ pixels with various methods, such as instance center regression [44, 86, 70, 100, 20], Watershed transform [88, 3, 8], Hough-voting [4, 53, 8], or pixel affinity [45, 66, 81, 30, 8].”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…are closely related to the multitask learning literature, where the relationship between task similarity and network performance in terms of accuracy and time has been studied (Zhang & Yang, 2021; Ruder, 2017; Standley et al., 2020; Liu, Johns, & Davison, 2019; Kendall, Gal, & Cipolla, 2018).”
Do not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting (2021, background): “However, work by Gal and Ghahramani [8] and other related subsequent works [14, 15] suggest that it is possible to obtain the uncertainty of a model in its parameters or the data distribution during prediction.”
Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, methodology): “Several works proposed to dynamically re-weight the loss function of each task [6,21,59,56], sort the order of task learning [16], or adapt the feature sharing between ‘related’ and ‘unrelated’ tasks [69], to mitigate the effect of negative transfer.”
Predictive Uncertainty Estimation via Prior Networks (2018, methodology): “This technique has been successfully applied to tasks in computer vision [19, 20].”
Learning to Drive in a Day (2018, background): “Beyond pixel-space autoencoders is a wealth of computer vision research addressing effective compression of images: here existing work in areas such as semantic segmentation, depth, egomotion and pixel-flow provide an excellent prior for what is important in driving scenes [35], [1], [36].”
Gradient Surgery for Multi-Task Learning (2020, methodology): “Similarly to our work, a number of prior approaches have observed the difficulty of optimization in multi-task learning [26, 29, 52, 55].”
Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding (2015, background): “Further Applications • Distinguish Aleatoric (sensor) uncertainty and Epistemic (model) uncertainty [3] • Use uncertainty to improve multi-task learning [4] • Semantic segmentation, instance segmentation and depth regression from a single input image [4] Model Standard Bayesian DilationNet 71.”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, methodology): “We note that there are several instance segmentation works [77, 68, 76, 2, 48, 35, 56, 20, 18, 44, 37, 30, 51, 54, 6], which could be potentially extended to bottom-up panoptic segmentation.”
DCN+: Mixed Objective and Deep Residual Coattention for Question Answering (2017, methodology): “In a similar fashion to Kendall et al. (2017), we combine the two losses using homoscedastic uncertainty as task-dependent weightings.”
GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks (2017, background): “Perhaps the most relevant to the current work, (Kendall et al., 2017) uses a joint likelihood formulation to derive task weights based on the intrinsic uncertainty in each task.”
Learning to Fuse Things and Stuff (2018, methodology): “This task has drawn attention from the computer vision community as a key next step in dense scene understanding [14, 18, 28], and several publicly available benchmark datasets have started to provide labels supporting this task, including Cityscapes [8], Mapillary Vistas [27], ADE20k [44], and COCO”
Which Tasks Should Be Learned Together in Multi-task Learning? (2019, methodology): “Hard parameter sharing is also used in many other works such as [42, 4, 27, 12, 17, 2, 33, 10, 45, 19, 23, 8, 47, 36].”