Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance

By Marvin Klingner, Jan-Aike Termöhlen, Jonas Mikolajczyk, Tim Fingscheidt. European Conference on Computer Vision, 2020

396 citations recorded by the source corpus.

Self-supervised monocular depth estimation presents a powerful method to obtain 3D scene information from single camera images, which is trainable on arbitrary image sequences without requiring depth labels, e.g., from a LiDAR sensor. In this work we present a new self-supervised semantically-guided depth estimation (SGDepth) method to deal with moving dynamic-class (DC) objects, such as moving cars and pedestrians, which violate the static-world assumptions typically made during training of such models. Specifically, we propose (i) mutually beneficial cross-domain training of (supervised) semantic segmentation and self-supervised depth estimation with task-specific network heads, (ii) a semantic masking scheme providing guidance to prevent moving DC objects from contaminating the photometric loss, and (iii) a detection method for frames with non-moving DC objects, from which the depth of DC objects can be learned. We demonstrate the performance of our method on several benchmarks, in particular on the Eigen split, where we exceed all baselines without test-time refinement.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/49f78a8d46ddd1224c178d20f1d8acd2e4ba0aaf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Neural Window Fully-connected CRFs for Monocular Depth Estimation (2022, background): “Also, other approaches bring in auxiliary information to help the training of the depth network, such as sparse depth [10] or segmentation information [15,24,27,44].”
Is Pseudo-Lidar needed for Monocular 3D Object detection? (2021, background): “It is itself a thriving research area: the community has pushed toward accurate dense depth prediction via supervised [35, 29, 19, 50, 26, 18, 32, 13, 31] as well as self-supervised [48, 75, 62, 16, 15, 64] methods.”
Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimation (2022, background): “In addition to increasing the accuracy of monocular training by introducing improved loss functions [15] and semantic information [5, 21] to mitigate the occlusion and moving objects problems, many works focused on designing more effective CNN architectures [17, 33, 39, 41, 46].”
MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer (2022, methodology): “Therefore, novel constraints [17] and additional cues [22, 64] (like semantic segmentation, optical ﬂow and surface normals) are often used to reduce the shortcomings mentioned above.”
Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (2024, background): “There exist some works [9, 21, 28, 71] improving depth estimation with an auxiliary semantic segmentation task.”