Learning Depth from Monocular Videos Using Direct Methods

By Chaoyang Wang, José Miguel Buenaposada, Rui Zhu, Simon Lucey. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017

639 citations recorded by the source corpus.

The ability to predict depth from a single image - using recent advances in CNNs - is of increasing interest to the vision community. Unsupervised strategies to learning are particularly appealing as they can utilize much larger and varied monocular video datasets during learning without the need for ground truth depth or stereo. In previous works, separate pose and depth CNN predictors had to be determined such that their joint outputs minimized the photometric error. Inspired by recent advances in direct visual odometry (DVO), we argue that the depth CNN predictor can be learned without a pose CNN predictor. Further, we demonstrate empirically that incorporation of a differentiable implementation of DVO, along with a novel depth normalization strategy - substantially improves performance over state of the art that use monocular videos for training.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/71acad655c616df320faa16234c7221a41869679

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

BA-Net: Dense Bundle Adjustment Network (2018, methodology): “Our method outperforms the supervised methods (Eigen et al., 2014) as well as recent unsupervised methods (Zhou et al., 2017; Wang et al., 2018; Godard et al., 2017).”
Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding (2018, methodology): “Same as in [10], [82], the evaluation is performed on the test set of 134 images.”
Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video (2019, background): “Furthermore, comprehensive evaluation results on the KITTI [15] dataset show that our depth network outperforms state-of-the-art models that are trained in more complicated multi-task learning frameworks [9–11, 16].”
Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos (2018, methodology): “In addition to the reconstruction loss, the baseline uses an SSIM (Wang et al. 2004) loss, a depth smoothness loss and applies depth normalization during training, which demonstrated success in prior works (Zhou et al. 2017; Godard, Aodha, and Brostow 2017; Wang et al. 2018).”
Single-View View Synthesis With Multiplane Images (2020, methodology): “This idea has been used in classical computer vision, notably in stereo correspondence [25], and also in a variety of different smoothness losses for learning depth prediction [9, 17, 33].”
From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation (2019, background): “[42] adopt a differentiable pose predictor and train a monocular depth estimation network in an end-to-end fashion while benefited from the pose predictor.”
On the Uncertainty of Self-Supervised Monocular Depth Estimation (2020, background): “Seminal work in this direction is [82], extended by leveraging on point-cloud alignment [45], differentiable DVO [69], optical flow [78, 83, 11, 3], semantic [66] or scale consistency [5].”
BARF: Bundle-Adjusting Neural Radiance Fields (2021, background): “They can also be naturally integrated into deep learning frameworks through image reconstruction losses [70, 58, 66].”
SuperDepth: Self-Supervised, Super-Resolved Monocular Depth Estimation (2018, background): “Recent advances in Convolutional Neural Networks (CNNs), especially for depth and pose estimation [1], [2], [3], [4] from a monocular camera have dramatically shifted the landscape of single-image 3D reconstruction.”
Depth From Videos in the Wild: Unsupervised Monocular Depth Learning From Unknown Cameras (2019, methodology): “Other novel techniques include the use of motion [41, 35, 44, 6, 40].”
Self-Supervised Monocular Depth Hints (2019, background): “Depth regression is optimized and relative poses come from stereo camera calibration in a training-from-stereo setting [9, 7, 25, 27, 26], while depth values and camera poses can be optimized jointly when training on videos [42, 20, 37, 22, 38, 32, 44, 36, 28].”
Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance (2020, methodology): “roughout our experimental evaluation. Network Topology: Our topology is based on [20], where an encoderdecoder architecture with skip connections is employed. To ensure comparability to existing work [5,24,20,58], we choose an Imagenet [51] pretrained ResNet18 encoder [25]. The depth head has a sigm”
3D Packing for Self-Supervised Monocular Depth Estimation (2019, methodology): “Several methods [5, 26, 33, 43, 46, 47, 51, 53] have advanced this line terms,of work by incorporatingthese methods, ad,ditional loss and constraints.”
MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer (2022, background): “Several frameworks have advanced this line of research by incorporating additional losses and constraints such as those based on direct visual odometry [49], adversarial learning[68], ICP [31], normal consistency [63, 62], semantic segmentation [23, 19] and uncertainty [35, 60].”
Refine and Distill: Exploiting Cycle-Inconsistency and Knowledge Distillation for Unsupervised Monocular Depth Estimation (2019, background): “Other recent works demonstrated that temporal information and, in particular, considering multiple consecutive frames contribute to improve depth estimation [35, 41, 12, 43].”
D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry (2020, background): “More recent works start to tackle the problem in a self-and unsupervised way by learning the depth map using the photometric error [27, 28, 49, 73, 81, 82, 86] and adopting differentiable interpolation [32].”
Monocular depth estimation based on deep learning: An overview (2020, background): “n et al. [98] 2019 Stereo images p Multi-task Fei et al. [17] 2019 Stereo images + IMU + Semantic Label p Multi-task,physical information Feng et al. [65] 2019 Stereo images p Stacked-GAN Wang et al. [16] 2018 Mono. sequences p Direct VO Zhan et al. [14] 2018 Stereo sequences p Deep feature reconstr”