Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos

By Vincent Casser, Sören Pirk, Reza Mahjourian, Anelia Angelova. AAAI Conference on Artificial Intelligence, 2018

505 citations recorded by the source corpus.

Learning to predict scene depth from RGB inputs is a challenging task both for indoor and outdoor robot navigation. In this work we address unsupervised learning of scene depth and robot ego-motion where supervision is provided by monocular videos, as cameras are the cheapest, least restrictive and most ubiquitous sensor for robotics. Previous work in unsupervised image-to-depth learning has established strong baselines in the domain. We propose a novel approach which produces higher quality results, is able to model moving objects and is shown to transfer across data domains, e.g. from outdoors to indoor scenes. The main idea is to introduce geometric structure in the learning process, by modeling the scene and the individual objects; camera ego-motion and object motions are learned from monocular videos as input. Furthermore an online refinement method is introduced to adapt learning on the fly to unknown domains. The proposed approach outperforms all state-of-the-art approaches, including those that handle motion e.g. through learned flow. Our results are comparable in quality to the ones which used stereo as supervision and significantly improve depth prediction on scenes and datasets which contain a lot of object motion. The approach is of practical relevance, as it allows transfer across environments, by transferring models trained on data collected for robot navigation in urban scenes to indoor navigation settings. The code associated with this paper can be found at https://sites.google.com/view/struct2depth.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2887d97aaff6513106b99a917f2305eb182294c7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

iGibson, a Simulation Environment for Interactive Tasks in Large Realistic Scenes (2020, background): “as: depth/segmentation/normal/affordance prediction [45, 46, 47], action-conditioned flow prediction [48], multi-modal pose estimation [49, 50, 51], and visuomotor policy training assuming perfect vision systems [35, 52].”
Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance (2020, methodology): “To ensure comparability to existing work [5,24,20,58], we choose an Imagenet [51] pretrained ResNet18 encoder [25].”
Consistent video depth estimation (2020, methodology): “Note that any selfsupervised method can be trained at test time (as in [Casser et al. 2019; Chen et al. 2019b]).”
The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth (2021, methodology): “Unlike similar works e.g. [50, 97, 9, 59, 5], we do not use future frames at test time e.g.”
D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry (2020, unknown): “45 Struct2Depth [6] 10.”
On the Uncertainty of Self-Supervised Monocular Depth Estimation (2020, background): “One of the shortcomings of these approaches is represented by moving objects appearing in the training images, addressed in [8, 75] employing instance segmentation and subsequent motion estimation of the segmented dynamic objects.”
Monocular depth estimation based on deep learning: An overview (2020, methodology): “Besides, the recent work [81] has proven that it is more efficient to calculate the minimum value of the reconstruction error than the mean, which has been applied in [82], [83].”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, unknown): “While adapting image models to a video input [367], [416],”
iGibson 1.0: A Simulation Environment for Interactive Tasks in Large Realistic Scenes (2020, background): “These modalities can support research topics such as: depth/segmentation/normal/affordance prediction [48, 49, 50], action-conditioned flow prediction [51], multi-modal pose estimation [52, 53, 54], and visuomotor policy training assuming perfect vision systems [35, 55].”
Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding (2018, methodology): “[55] use Mask R-CNN [56] to generate possible moving regions and apply scale constraints on the moving objects.”
3D Packing for Self-Supervised Monocular Depth Estimation (2019, methodology): “estimation have mostly focused on engineering the loss function [5, 33, 47, 53], we show that performance critically depends on the model architecture, in line with the observations of [27] for other self-supervised tasks.”