Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations

By Vladimir Nekrasov, Thanuja Dharmasiri, Andrew Spek, Tom Drummond, Chunhua Shen, Ian David Reid. 2018

161 citations recorded by the source corpus.

Deployment of deep learning models in robotics as sensory information extractors can be a daunting task to handle, even using generic GPU cards. Here, we address three of its most prominent hurdles, namely, i) the adaptation of a single model to perform multiple tasks at once (in this work, we consider depth estimation and semantic segmentation crucial for acquiring geometric and semantic understanding of the scene), while ii) doing it in real-time, and iii) using asymmetric datasets with uneven numbers of annotations per each modality. To overcome the first two issues, we adapt a recently proposed real-time semantic segmentation network, making changes to further reduce the number of floating point operations. To approach the third issue, we embrace a simple solution based on hard knowledge distillation under the assumption of having access to a powerful ‘teacher’ network. We showcase how our system can be easily extended to handle more tasks, and more datasets, all at once, performing depth estimation and segmentation both indoors and outdoors with a single model. Quantitatively, we achieve results equivalent to (or better than) current state-of-the-art approaches with one forward pass costing just 13ms and 6.5 GFLOPs on 640×480 inputs. This efficiency allows us to directly incorporate the raw predictions of our network into the SemanticFusion framework [1] for dense 3D semantic reconstruction of the scene.33The models are available here: https://github.com/drsleep/ multi–task–refinenet

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Which Tasks Should Be Learned Together in Multi-task Learning? (2019, methodology): “Hard parameter sharing is also used in many other works such as [42, 4, 27, 12, 17, 2, 33, 10, 45, 19, 23, 8, 47, 36].”
Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cells (2018, methodology): “Following previous work [25], we only use 25K training images with depth annotations from the Kinect sensor, and report validation results on 654 images in Table 3.”
Multi-Task Learning for Dense Prediction Tasks: A Survey (2020, methodology): “Prior work [112] showed that the standard update rule from Equation 5 gives suboptimal results if we do not take precautions when annotations are missing.”
Enforcing Geometric Constraints of Virtual Normal for Depth Prediction (2019, unknown): “Metrics CReaM [41] RF-LW[34] Ours-B Ours-VN”
Multi-Task Collaborative Network for Joint Referring Expression Comprehension and Segmentation (2020, background): “MTL has been widely deployed in a variety of computer vision tasks [8, 5, 27, 7, 10, 15].”