Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints

By Reza Mahjourian, Martin Wicke, Anelia Angelova. 2018

773 citations recorded by the source corpus.

We present a novel approach for unsupervised learning of depth and ego-motion from monocular video. Unsupervised learning removes the need for separate supervisory signals (depth or ego-motion ground truth, or multi-view video). Prior work in unsupervised depth learning uses pixel-wise or gradient-based losses, which only consider pixels in small local neighborhoods. Our main contribution is to explicitly consider the inferred 3D geometry of the whole scene, and enforce consistency of the estimated 3D point clouds and ego-motion across consecutive frames. This is a challenging task and is solved by a novel (approximate) backpropagation algorithm for aligning 3D structures. We combine this novel 3D-based loss with 2D losses based on photometric quality of frame reconstructions using estimated depth and ego-motion from adjacent frames. We also incorporate validity masks to avoid penalizing areas in which no useful information exists. We test our algorithm on the KITTI dataset and on a video dataset captured on an uncalibrated mobile phone camera. Our proposed approach consistently improves depth estimates on both datasets, and outperforms the state-of-the-art for both depth and ego-motion. Because we only require a simple video, learning depth and ego-motion on large and varied datasets becomes possible. We demonstrate this by training on the low quality uncalibrated video dataset and evaluating on KITTI, ranking among top performing prior methods which are trained on KITTI itself.1

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Self-Supervised Correspondence in Visuomotor Policy Learning (2019, background): “These methods, however, have not benefitted from the rich sources of self-supervision that may be provided by dense threedimensional computer vision techniques [4]–[6], for example correspondence learning which robots can automate without human input [7].”
Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer (2019, background): “Various approaches that leverage self-supervision have been proposed, but they either require stereo images [15, 47] or are based on apparent motion [49, 31, 2], and are thus challenging to apply to highly dynamic scenes.”
DeepV2D: Video to Depth with Differentiable Structure from Motion (2018, methodology): “Depth and Motion: Geometric information has served as a self-supervisory signal for many recent works (Vijayanarasimhan et al., 2017; Zhou et al., 2017; Wang et al., 2018; Yin & Shi, 2018; Yang et al., 2018; Godard et al., 2017; Mahjourian et al., 2018).”
Depth From Videos in the Wild: Unsupervised Monocular Depth Learning From Unknown Cameras (2019, background): “Unsupervised learning of depth, where the only supervision is obtained from the monocular video itself and no depth sensors are needed, has also been popularized recently [47, 10, 12, 33, 24, 34, 44].”
Empowering Things With Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things (2020, background): “10) Depth Estimation/Localization/SLAM: Estimating depth using cameras is a long-standing research topic [109], [129]–[131].”
Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video (2019, neutral): “015 Mahjourian et al. [6] 0 .”
Self-Supervised Monocular Depth Hints (2019, background): “Depth regression is optimized and relative poses come from stereo camera calibration in a training-from-stereo setting [9, 7, 25, 27, 26], while depth values and camera poses can be optimized jointly when training on videos [42, 20, 37, 22, 38, 32, 44, 36, 28].”
From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation (2019, background): “[30] present an approach that explicitly consider the inferred 3D geometry of the whole scene, and enforce consistency of the estimated 3D point clouds and ego-motion across consecutive frames.”
Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR and Monocular Camera (2018, methodology): “[29] improved the accuracy by using 3D geometric constraints, and Yin and Shi [30] extended the framework for optical flow estimation.”
Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation (2018, methodology): “’s split [5] that is consistent across related works [5, 20, 22, 37, 38, 39].”
3D Packing for Self-Supervised Monocular Depth Estimation (2019, methodology): “Several methods [5, 26, 33, 43, 46, 47, 51, 53] have advanced this line terms,of work by incorporatingthese methods, ad,ditional loss and constraints.”
MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer (2022, background): “Several frameworks have advanced this line of research by incorporating additional losses and constraints such as those based on direct visual odometry [49], adversarial learning[68], ICP [31], normal consistency [63, 62], semantic segmentation [23, 19] and uncertainty [35, 60].”
Learning the Depths of Moving People by Watching Frozen People (2019, background): “Numerous algorithms, based on both supervised and unsupervised learning, have recently been proposed for predicting dense depth from a single RGB image [46, 17, 7, 6, 3, 19, 33, 8, 52, 49, 21, 41].”
SuperDepth: Self-Supervised, Super-Resolved Monocular Depth Estimation (2018, methodology): “Input image SuperDepth MonoDepth [6] GeoNet [12] SfMLearner [7] Vid2Depth [15]”
On the Uncertainty of Self-Supervised Monocular Depth Estimation (2020, background): “Self-supervised [19, 82, 45, 56, 53] or weakly supervised [76, 65, 72] paradigms, leveraging on image reprojection and noisy labels respectively, have removed this issue and yield accuracy close to supervised methods [15], neglecting at all the deployment of additional depth sensors for labeling pur”
Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance (2020, methodology): “Other approaches weigh the loss functions by a factor [38,71] inducing the downside that the gradients inside the decoders are also scaled.”
Softmax Splatting for Video Frame Interpolation (2020, background): “(cid:1) Prominent examples where this approach has been used include unsupervised depth estimation [13, 31, 54], unsupervised optical ﬂow prediction [32, 47, 52], optical ﬂow prediction [18, 42, 45], novel view synthesis [8, 27, 55], video frame interpolation [3, 22, 28, 29], and video enhancement…”