PAD-Net: Multi-tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing

By Dan Xu, Wanli Ouyang, Xiaogang Wang, Nicu Sebe. 2018

571 citations recorded by the source corpus.

Depth estimation and scene parsing are two particularly important tasks in visual scene understanding. In this paper we tackle the problem of simultaneous depth estimation and scene parsing in a joint CNN. The task can be typically treated as a deep multi-task learning problem [42]. Different from previous methods directly optimizing multiple tasks given the input training data, this paper proposes a novel multi-task guided prediction-and-distillation network (PAD-Net), which first predicts a set of intermediate auxiliary tasks ranging from low level to high level, and then the predictions from these intermediate auxiliary tasks are utilized as multi-modal input via our proposed multi-modal distillation modules for the final tasks. During the joint learning, the intermediate tasks not only act as supervision for learning more robust deep representations but also provide rich multi-modal information for improving the final tasks. Extensive experiments are conducted on two challenging datasets (i.e. NYUD-v2 and Cityscapes) for both the depth estimation and scene parsing tasks, demonstrating the effectiveness of the proposed approach.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, methodology): “On the other hand, decoder-focused approaches [59, 61, 62, 58] utilize the multi-task feature representations learned at the encoding stage to distill cross-task information at the decoding stage, thus refining the original feature representations.”
Branched Multi-Task Networks: Deciding what layers to share (2019, background): “Finally, multi-task networks may outperform their single-task counterparts (Kendall et al., 2018; Xu et al., 2018; Sener & Koltun, 2018; Maninis et al., 2019).”
DSNet: Joint Learning for Scene Segmentation and Disparity Estimation (2019, methodology): “Moreover, in module C, we combine module A and B by introducing attention mechanism [36] which has been proven to be effective.”
Categorical Depth Distribution Network for Monocular 3D Object Detection (2021, methodology): “Further, many methods jointly perform depth estimation and segmentation [63, 66, 58, 14] in an end-to-end manner.”
GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image (2024, background): “On the other hand, several works [12,28,67,79] create multiple branches for depth and normal, and enforce information exchange through propagating latent features.”
ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding (2021, background): “Other works focused on real-time performance [54], leveraging different modalities such as depth [52], and deﬁning neighborhood-based supervision [20] for segmentation.”
Pattern-Affinitive Propagation Across Depth, Surface Normal and Semantic Segmentation (2019, background): “In contrast to the single-task methods, recently, several joint-task learning methods [58, 62, 46, 32] on these tasks have shown a promising direction to improve the predictions by utilizing task-correlative information to boost for each other.”
Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and More (2019, methodology): “TargetNet yields results better than all but one compared methods, PADNet-ResNet [36], for which the size is almost three times as that of TargetNet.”
MTL-NAS: Task-Agnostic Neural Architecture Search Towards General-Purpose Multi-Task Learning (2020, background): “A great success has been witnessed in various areas such as detection [15, 16, 23, 50, 54, 55], depth prediction and semantic segmentation (also surface normal prediction) [10, 65], human-related tasks [21,48,60,62,68], etc.”
Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans (2021, background): “Instead of predicting mid-level cues as the downstream task (i.e. multi-task learning), multiple cues could also be used as inputs (if relevant sensors are available) or specified as intermediate representation (with the labels being used as supervision only during training, i.e. PADNet [68]).”
MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning (2020, methodology): “[47] proposed to distil information from the initial predictions of other tasks, by means of spatial attention, before adding it as a residual to the task of interest.”
Rethinking the Competition Between Detection and ReID in Multiobject Tracking (2020, methodology): “The idea of designing CCN is inspired by recent self-attention [13] and multi-task decoding [14] mechanisms, as shown in Fig.”