Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue

By Ravi Garg, B. V. Kumar, Gustavo Carneiro, Ian David Reid. 2016

1,652 citations recorded by the source corpus.

A significant weakness of most current deep Convolutional Neural Networks is the need to train them using vast amounts of manually labelled data. In this work we propose a unsupervised framework to learn a deep convolutional neural network for single view depth prediction, without requiring a pre-training stage or annotated ground-truth depths. We achieve this by training the network in a manner analogous to an autoencoder. At training time we consider a pair of images, source and target, with small, known camera motion between the two such as a stereo pair. We train the convolutional encoder for the task of predicting the depth map for the source image. To do so, we explicitly generate an inverse warp of the target image using the predicted depth and known inter-view displacement, to reconstruct the source image; the photometric error in the reconstruction is the reconstruction loss for the encoder. The acquisition of this training data is considerably simpler than for equivalent systems, requiring no manual annotation, nor calibration of depth sensor to camera. We show that our network trained on less than half of the KITTI dataset gives comparable performance to that of the state-of-the-art supervised methods for single view depth estimation.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Learning Features by Watching Objects Move (2016, background): “Animals and robots can also sense their own motion (proprioception), and a possible task is to predict this signal from the visual input alone [2, 14, 21].”
Learning a Multi-View Stereo Machine (2017, background): “A line of recent works, complementary to ours, has proposed to incorporate ideas from multi-view geometry in a learning framework to train single view prediction systems [14, 56, 53, 42, 61] using multiple views as supervisory signal.”
Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency (2017, background): “rect [8], multi-view [11, 15] or even no supervision [10] predict detailed 2.”
The limits and potentials of deep learning for robotics (2018, background): “Estimating the depth of the scene from a single image has become a widely researched topic [31], [32], [66].”
Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics (2017, background): “Depth by recognition has been demonstrated using dense prediction networks with supervised [15] and unsupervised [16] deep learning.”
Unsupervised Learning of Depth and Ego-Motion from Video (2017, methodology): “[14]), but falls short of concurrent work by Godard et al.”
Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer (2019, background): “Stereo cameras are a promising source of data [13, 15], but collecting stereo images in diverse environments at scale remains a challenge.”
Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age (2016, positive): “An online semantic mapping system was later proposed by Pronobis et al. [233, 234], who combine three layers of reasoning (sensory, categorical and place) to build a semantic map of the environment using laser and camera sensors.”
Concrete Problems for Autonomous Vehicle Safety: Advantages of Bayesian Deep Learning (2017, methodology): “However, visual solutions have been demonstrated using supervised [Eigen and Fergus, 2015] and unsupervised [Garg et al., 2016] deep learning (see Figure 3).”
Learning Monocular 3D Human Pose Estimation from Multi-view Images (2018, background): “[6], the multi-view visual hull constraint used for reconstruction of static objects by Yan et al.”
Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations (2018, methodology): “using supervised learning [9], [26]–[28], unsupervised learning [29], [30] and semi-supervised learning [31].”
Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints (2018, methodology): “Previous approaches have either ignored this problem, or tried to tackle it by adding a general-purpose mask to the network [9, 30, 25], which is expected to exclude regions that are unexplainable due to any reason.”
Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimation (2022, background): “[12] regarded the depth estimation as a novel view synthesis problem, and proposed to minimize a photometric loss between an input left image and the synthesized right image.”
Single-View View Synthesis With Multiplane Images (2020, background): “At the same time, depth and view synthesis are highly intertwined, and many recent depth prediction methods use view synthesis as implicit supervision [7, 9, 36].”
MegaDepth: Learning Single-View Depth Prediction from Internet Photos (2018, background): “Multiple views of a scene can also be used as an implicit source of training data for single-view depth prediction, by utilizing view synthesis as a supervisory signal [38, 10, 13, 43].”
The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth (2021, methodology): “Self-supervised methods remove the limitation of requiring ground truth depth supervision, instead training with image-reconstruction losses using stereo pairs [87, 21, 23] or monocular video sequences [99].”
Unsupervised object learning from dense equivariant image labelling (2017, methodology): “[34, 11] use a warping-based loss to learn depth from video.”