Deep Virtual Stereo Odometry: Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometry

By Nan Yang, Rui Wang, Jörg Stückler, Daniel Cremers. European Conference on Computer Vision, 2018

356 citations recorded by the source corpus.

Monocular visual odometry approaches that purely rely on geometric cues are prone to scale drift and require sufficient motion parallax in successive frames for motion estimation and 3D reconstruction. In this paper, we propose to leverage deep monocular depth prediction to overcome limitations of geometry-based monocular visual odometry. To this end, we incorporate deep depth predictions into Direct Sparse Odometry (DSO) as direct virtual stereo measurements. For depth prediction, we design a novel deep network that refines predicted depth from a single image in a two-stage process. We train our network in a semi-supervised way on photoconsistency in stereo images and on consistency with accurate sparse depth reconstructions from Stereo DSO. Our deep predictions excel state-of-the-art approaches for monocular depth on the KITTI benchmark. Moreover, our Deep Virtual Stereo Odometry clearly exceeds previous monocular and deep-learning based methods in accuracy. It even achieves comparable performance to the state-of-the-art stereo methods, while only relying on a single camera.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/86a891b3209984562c72fd49c7fd26737cb58c5d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

A2D2: Audi Autonomous Driving Dataset (2020, methodology): “As mentioned in Section 2.2, these data are useful for research in end-to-end autonomous driving, depth prediction from mono/stereo images or videos [26, 27], and SLAM.”
On the Uncertainty of Self-Supervised Monocular Depth Estimation (2020, background): “Self-supervised [19, 82, 45, 56, 53] or weakly supervised [76, 65, 72] paradigms, leveraging on image reprojection and noisy labels respectively, have removed this issue and yield accuracy close to supervised methods [15], neglecting at all the deployment of additional depth sensors for labeling pur”
3D Packing for Self-Supervised Monocular Depth Estimation (2019, methodology): “Several methods [5, 26, 33, 43, 46, 47, 51, 53] have advanced this line terms,of work by incorporatingthese methods, ad,ditional loss and constraints.”
Digging Into Self-Supervised Monocular Depth Estimation (2018, background): “Recent work has shown that conventional structure-from-motion (SfM) pipelines can generate sparse training signal for both camera pose and depth [35, 28, 68], where SfM is typically run as a pre-processing step decoupled from learning.”
SuperDepth: Self-Supervised, Super-Resolved Monocular Depth Estimation (2018, methodology): “Disparity Smoothness Loss In order to regularize the disparities in textureless low-image gradient regions, we incorporate an edge-aware term (Equation 4), similar to [6], [13], [23].”
D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry (2020, methodology): “Inspired by [78], we introduce a virtual stereo term E † p to Eq.”
Monocular depth estimation based on deep learning: An overview (2020, background): “[111] follow this idea and leverage the depth estimation based on deep learning to recover the absolute scale of monocular”
BA-Net: Dense Bundle Adjustment Network (2018, methodology): “deal with these problems, we use the convolutional network for monocular image depth estimation as a compact parameterization, rather than using it as an initialization as in Tateno et al. (2017) and Yang et al. (2018). We use a standard encoder-decoder architecture for monocular depth learning as i”
DeepV2D: Video to Depth with Differentiable Structure from Motion (2018, background): “Goddard et. al [12] Yang et al [47] trains single-image depth networks supervised on photo-consistency measure between stereo pairs.”
DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras (2021, methodology): “Previous work has investigated replacing hand-crafted with learned features[13, 7, 29, 26, 35], using neural 3D representations[46, 1, 9, 45, 44, 25, 22], and combining learned energy terms with classical optimization backends[58, 57].”
CubeSLAM: Monocular 3-D Object SLAM (2018, background): “Recently, there is also some end-to-end deep learning-based SLAM without object representations, such as DVSO [28], DeepVO [29].”