Unsupervised Monocular Depth Estimation with Left-Right Consistency

By Clément Godard, Oisin Mac Aodha, Gabriel J. Brostow. 2016

3,248 citations recorded by the source corpus.

Learning based methods have shown very promising results for the task of depth estimation in single images. However, most existing approaches treat depth prediction as a supervised regression problem and as a result, require vast quantities of corresponding ground truth depth data for training. Just recording quality depth data in a range of environments is a challenging problem. In this paper, we innovate beyond existing approaches, replacing the use of explicit depth data during training with easier-to-obtain binocular stereo footage. We propose a novel training objective that enables our convolutional neural network to learn to perform single image depth estimation, despite the absence of ground truth depth data. Ex-ploiting epipolar geometry constraints, we generate disparity images by training our network with an image reconstruction loss. We show that solving for image reconstruction alone results in poor quality depth images. To overcome this problem, we propose a novel training loss that enforces consistency between the disparities produced relative to both the left and right images, leading to improved performance and robustness compared to existing approaches. Our method produces state of the art results for monocular depth estimation on the KITTI driving dataset, even outperforming supervised methods that have been trained with ground truth depth.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, methodology): “More recently, higher-order cycle consistency has been used in structure from motion [56], 3D shape matching [19], co-segmentation [51], dense semantic alignment [59, 60], and depth estimation [12].”
Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency (2017, methodology): “[15] used stereo images to learn a single image depth prediction system by minimizing the inconsistency as measured by pixelwise reprojection error.”
The limits and potentials of deep learning for robotics (2018, background): “Estimating the depth of the scene from a single image has become a widely researched topic [31], [32], [66].”
Unsupervised Learning of Depth and Ego-Motion from Video (2017, methodology): “[16] that uses calibrated stereo images (i.”
Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer (2019, methodology): “Following [10], we cap predictions at an appropriate maximum value for datasets that are evaluated in depth space.”
Kornia: an Open Source Differentiable Computer Vision Library for PyTorch (2019, background): “Similar to [39, 18, 43], depth is solved for by minimizing a photometric error between the views warped to the reference view, (equation 2 and 3).”
Learning Correspondence From the Cycle-Consistency of Time (2019, methodology): “es. Forward-backward consistency is a speciﬁc case of cycle-consistency, which has been widely applied as a learning objective for 3D shape matching [21], image alignment [87,89,88], depth estimation [86,14,84], and image-to-image translation [90,4]. For example Zhou et al. [88] used 3D CAD models t”
Weakly-Supervised Salient Object Detection via Scribble Annotations (2020, background): “Inspired by the smoothness loss [9, 38], we also impose such constraint inside the salient regions.”
DeepV2D: Video to Depth with Differentiable Structure from Motion (2018, background): “Goddard et. al [12] Yang et al [47] trains single-image depth networks supervised on photo-consistency measure between stereo pairs.”
Synthetic Data for Deep Learning (2019, background): “• L s (f task) = ∣∣∂xfT2 task(xT )∣∣−|∂xxT + ∣∣∂yfT2 task(xT )∣∣−|∂yxT , where ∂x and ∂y are image gradients, is the task loss for depth estimation on real images; since ground truth depth maps are not available now, this regularizer is a locally smooth loss intended to optimize object boundaries, a”
Unsupervised Event-Based Learning of Optical Flow, Depth, and Egomotion (2018, methodology): “Table 2: Quantitative evaluation of our depth network compared to Monodepth [6].”
SfM-Net: Learning of Structure and Motion from Video (2017, methodology): “Differentiable warping [16] has been used to learn end-to-end unsupervised optical ﬂow [34], disparity ﬂow in a stereo rig [13] and video prediction [24].”
HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation (2020, methodology): “Recent works (Zhou et al. 2017; Godard et al. 2017) in self-supervised depth estimation are limited to training in low-resolution inputs due to the large memory requirements Figure 1: Depth prediction from single image on KITTI.”
Monocular Fisheye Camera Depth Estimation Using Sparse LiDAR Supervision (2018, background): “Full scene understanding with our capability to precisely estimate depth appears to bolster from the combination of both top-down and bottom-up cues [5].”
Real-Time Self-Adaptive Deep Stereo (2018, methodology): “Then, adaptation can be effectively achieved by fine-tuning the model offline on samples from the target domain by relying on expensive annotations or unsupervised loss functions [6, 10, 40, 45].”
SO-Net: Self-Organizing Network for Point Cloud Analysis (2018, background): “Similar to common practice in many depth estimation networks [14, 15], the convolution branch is designed as an up-convolution (upconv) chain in a pyramid style.”
High Quality Monocular Depth Estimation via Transfer Learning (2018, background): “[12] 0.”