Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction

By Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, Ian David Reid. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

694 citations recorded by the source corpus.

Despite learning based methods showing promising results in single view depth estimation and visual odometry, most existing approaches treat the tasks in a supervised manner. Recent approaches to single view depth estimation explore the possibility of learning without full supervision via minimizing photometric error. In this paper, we explore the use of stereo sequences for learning depth and visual odometry. The use of stereo sequences enables the use of both spatial (between left-right pairs) and temporal (forward backward) photometric warp error, and constrains the scene depth and camera motion to be in a common, real-world scale. At test time our framework is able to estimate single view depth and two-view odometry from a monocular sequence. We also show how we can improve on a standard photometric warp loss by considering a warp of deep features. We show through extensive experiments that: (i) jointly training for single view depth and visual odometry improves depth prediction because of the additional constraint imposed on depths and achieves competitive results for visual odometry; (ii) deep feature-based warping loss improves upon simple photometric warp loss for both single view depth estimation and visual odometry. Our method outperforms existing learning based methods on the KITTI driving dataset in both tasks. The source code is available at https://github.com/Huangying-Zhan/Depth-VO-Feat.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/f58592051db1f5d300d096f2cd7869258d2de9d8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Refine and Distill: Exploiting Cycle-Inconsistency and Knowledge Distillation for Unsupervised Monocular Depth Estimation (2019, background): “Recently, several works proposed to tackle the depth estimation problem within an unsupervised learning framework [20, 28, 34, 42, 32].”
Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance (2020, methodology): “Since then this idea was complemented by improved loss functions [2,38], specialized network architectures [23,58,70], a hybrid approach utilizing video and stereo data [65], and refinement strategies [5,6] to optimize the network or the prediction at test time.”
Unsupervised Event-Based Learning of Optical Flow, Depth, and Egomotion (2018, methodology): “We train a second network to predict the egomotion of the camera and the structure of the scene, in a similar manner to [22, 18].”
Digging Into Self-Supervised Monocular Depth Estimation (2018, methodology): “Finally, inspired by [72], [73] use ground truth depth to train an appearance matching term.”
The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth (2021, background): “…robust image reconstruction losses [26, 72], discrete rather than continuous depth predictions [56, 25, 40], feature space reconstruction losses [93, 72], sparse automatically generated depth supervision [49, 83], occlusion handling [26], improved network architectures [28], and moving objects…”
DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency (2018, methodology): “To alleviate the dependency on large-scale annotated datasets, several works have been proposed to exploit the classical assumptions of brightness constancy and spatial smoothness on the disparity map or the flow field [17,21,28,43,71].”
Distilling Knowledge From a Deep Pose Regressor Network (2019, methodology): “In particular, we employ DNN-based VO methods [33, 34, 37, 38, 26, 1] which replaces the conventional VO pipeline based on multiple-view geometry [15, 27] to DNN-based pipeline which automatically learns useful features for estimating 6 Degree-of-Freedom (DoF) camera poses.”
Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos (2018, background): “Many subsequent works have improved the initial results in both the monocular setting (Yang et al. 2017; Yin 2018) and when using stereo during training (Godard, Aodha, and Brostow 2017; Ummen-hofer et al. 2017; Zhan et al. 2018; Yang et al. 2018a).”
On the Uncertainty of Self-Supervised Monocular Depth Estimation (2020, background): “Finally, approaches leveraging both kind of supervisions have been proposed in [80, 77, 41, 20].”
Depth From Videos in the Wild: Unsupervised Monocular Depth Learning From Unknown Cameras (2019, background): “A common metric for localization is average relative translational drift trel [31, 46] – the distance between the predicted location and the groundtruth location divided by distance traveled and averaged over the trajectory.”
MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer (2022, background): “Pre-cisely, training single-view depth estimation network with stereo images [14, 16], monocular videos [71], or a combination of both [65, 17].”
Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video (2019, methodology): “We compare with SfMLearner [7] and the methods trained with stereo videos [17].”
D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry (2020, background): “More recent works start to tackle the problem in a self-and unsupervised way by learning the depth map using the photometric error [27, 28, 49, 73, 81, 82, 86] and adopting differentiable interpolation [32].”
Monocular depth estimation based on deep learning: An overview (2020, neutral): “[14], Yin et al.”