Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video

By Jiawang Bian, Zhichao Li, Naiyan Wang, Huangying Zhan, Chunhua Shen, Ming-Ming Cheng, Ian David Reid. Neural Information Processing Systems, 2019

595 citations recorded by the source corpus.

Recent work has shown that CNN-based depth and ego-motion estimators can be learned using unlabelled monocular videos. However, the performance is limited by unidentified moving objects that violate the underlying static scene assumption in geometric image reconstruction. More significantly, due to lack of proper constraints, networks output scale-inconsistent results over different samples, i.e., the ego-motion network cannot provide full camera trajectories over a long video sequence because of the per-frame scale ambiguity. This paper tackles these challenges by proposing a geometry consistency loss for scale-consistent predictions and an induced self-discovered mask for handling moving objects and occlusions. Since we do not leverage multi-task learning like recent works, our framework is much simpler and more efficient. Comprehensive evaluation results demonstrate that our depth estimator achieves the state-of-the-art performance on the KITTI dataset. Moreover, we show that our ego-motion network is able to predict a globally scale-consistent camera trajectory for long video sequences, and the resulting visual odometry accuracy is competitive with the recent model that is trained using stereo videos. To the best of our knowledge, this is the first work to show that deep networks trained using unlabelled monocular videos can predict globally scale-consistent camera trajectories over a long video sequence.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/605fa5990fa23dac0066f97f9f5b41dfaba495ee

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

On the Uncertainty of Self-Supervised Monocular Depth Estimation (2020, background): “Seminal work in this direction is [82], extended by leveraging on point-cloud alignment [45], differentiable DVO [69], optical flow [78, 83, 11, 3], semantic [66] or scale consistency [5].”
Conditional Convolutions for Instance Segmentation (2020, background): “FCNs also have shown excellent performance on many other per-pixel prediction tasks ranging from low-level image processing such as denoising, super-resolution; to mid-level tasks such as optical flow estimation and contour detection; and high-level tasks including recent single-shot object detectio”
SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving (2023, methodology): “While early methods require full depth annotations to supervise the depth estimation model [15, 25, 66], later research focuses on self-supervised depth estimation as it does not require intensive human annotations [70, 17, 5, 67, 62, 56, 69, 43, 9].”
Learning to Recover 3D Scene Shape from a Single Image (2020, methodology): “We then align the unknown scale [3, 15] of the original depth and our shifted depth to the ground truth, and evaluate both using the AbsRel error.”
The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth (2021, background): “more robust image reconstruction losses [26, 72], discrete rather than continuous depth predictions [56, 25, 40], feature space reconstruction losses [93, 72], sparse automatically generated depth supervision [49, 83], occlusion handling [26], improved network architectures [28], and moving objects ”
Consistent video depth estimation (2020, background): “…et al. 2019], better network architecture and training loss design [Gordon et al. 2019; Guizilini et al. 2019], scale-consistent ego-motion network [Bian et al. 2019], incorporating 3D geometric constraints [Mahjourian et al. 2018], and learning from unknown camera intrinsics [Chen et al. 2019b;…”
Monocular depth estimation based on deep learning: An overview (2020, methodology): “Besides, a scale consistency loss term is proposed in [44], and it significantly tackles the problem of scale inconsistency between different depth maps.”