D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry

By Nan Yang, Lukas von Stumberg, Rui Wang, Daniel Cremers. Computer Vision and Pattern Recognition, 2020

464 citations recorded by the source corpus.

We propose D3VO as a novel framework for monocular visual odometry that exploits deep networks on three levels -- deep depth, pose and uncertainty estimation. We first propose a novel self-supervised monocular depth estimation network trained on stereo videos without any external supervision. In particular, it aligns the training image pairs into similar lighting condition with predictive brightness transformation parameters. Besides, we model the photometric uncertainties of pixels on the input images, which improves the depth estimation accuracy and provides a learned weighting function for the photometric residuals in direct (feature-less) visual odometry. Evaluation results show that the proposed network outperforms state-of-the-art self-supervised depth estimation networks. D3VO tightly incorporates the predicted depth, pose and uncertainty into a direct visual odometry method to boost both the front-end tracking as well as the back-end non-linear optimization. We evaluate D3VO in terms of monocular visual odometry on both the KITTI odometry benchmark and the EuRoC MAV dataset. The results show that D3VO outperforms state-of-the-art traditional monocular VO methods by a large margin. It also achieves comparable results to state-of-the-art stereo/LiDAR odometry on KITTI and to the state-of-the-art visual-inertial odometry on EuRoC MAV, while using only a single camera.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/61bbec51fa32571925c17efe8a2c7da48473f419

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual–Inertial, and Multimap SLAM (2020, background): “Otherwise, recent advances on depth estimation from a single image with CNNs offer good promise for reliable and true-scale monocular SLAM [85], at least in the same type of environments where the CNN has been trained.”
Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimation (2022, methodology): “To model dynamic scenes other works introduced multi-task learning, such as optical ﬂow estimation [41] and semantic segmentation [5,20], or introduced additional constraints, such as uncertainty estimation [30, 40].”
Co-SLAM: Joint Coordinate and Sparse Parametric Encodings for Neural Real-Time SLAM (2023, background): “Several recent works [11, 29, 30, 38] leverage deep learning to improve the accuracy and robustness of traditional SLAM, and even achieve dense reconstruction with monocular SLAM.”
NICE-SLAM: Neural Implicit Scalable Encoding for SLAM (2021, background): “The modified loss down-weights less certain regions in the reconstructed geometry [46, 62], e.”
A survey of uncertainty in deep neural networks (2021, background): “[207], [330], [331] developed deep learning based localization systems along with uncertainty estimates.”
Deep Learning Sensor Fusion for Autonomous Vehicle Perception and Localization: A Review (2020, unknown): “In [162], the network was trained in an unsupervised manner, but it was further modiﬁed to predict the depth and pose, also considering the uncertainty for VO optimization.”
Continuous 3D Perception Model with Persistent State (2025, unknown): “…replacing hand-crafted components with learning-based alternatives (e.g., substituting conventional feature descriptors [6, 47] with learned ones [17, 20, 63, 73]), integrating data-driven priors [13, 77–79, 93, 97, 105] into the systems, or optimizing entire systems end-to-end [76, 78, 83, 94].”
Empowering Things With Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things (2020, background): “The aforementioned camera pose estimation is also related to visual odometry (VO) and visual-inertial odometry (VIO) [133], [134], which aim to calculate sequential camera poses of an agent based on the camera and inertial measurement unit (IMU) sensors.”
DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras (2021, methodology): “D3VO [58] is able to achieve both good robustness and accuracy by combining a neural network frontend with DSO as a backend, using 6 of the 11 sequences for evaluation and performing unsupervised training on the remaining ones, which contain the same scenes used for evaluation.”
MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer (2022, background): “Several frameworks have advanced this line of research by incorporating additional losses and constraints such as those based on direct visual odometry [49], adversarial learning[68], ICP [31], normal consistency [63, 62], semantic segmentation [23, 19] and uncertainty [35, 60].”
ConceptFusion: Open-set Multimodal 3D Mapping (2023, background): “One of the major catalysts of continued progress in 3D spatial perception [1, 2, 3, 4] has been the advent of deep convolutional networks trained on large datasets of images.”
A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving (2020, background): “2), optical flow [43], [60], depth estimation [50], [57], [61], [62], visual odometry [35], [63], [64], image annotation [41], [65]– [67], visual tracking [52], trajectory prediction [49], [68]–[70] and end-to-end perception [47], [71], [72].”