Depth Map Prediction from a Single Image using a Multi-Scale Deep Network

By David Eigen, Christian Puhrsch, Rob Fergus. 2014

4,752 citations recorded by the source corpus.

Predicting depth is an essential component in understanding the 3D geometry of a scene. While for stereo images local correspondence suffices for estimation, finding depth relations from a single image is less straightforward, requiring integration of both global and local information from various cues. Moreover, the task is inherently ambiguous, with a large source of uncertainty coming from the overall scale. In this paper, we present a new method that addresses this task by employing two deep network stacks: one that makes a coarse global prediction based on the entire image, and another that refines this prediction locally. We also apply a scale-invariant error to help measure depth relations rather than scale. By leveraging the raw datasets as large sources of training data, our method achieves state-of-the-art results on both NYU Depth and KITTI, and matches detailed depth boundaries without the need for superpixelation.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Fully convolutional networks for semantic segmentation (2014, background): “We also discuss in-network upsampling 3.3, of which the fully connected prediction by Eigen et al .”
Objects as Points (2019, neutral): “[13] and d = 1 /σ ( ˆ d ) − 1 , where σ is the sigmoid function.”
Learning to generate chairs with convolutional neural networks (2014, background): “Our work is also loosely related to applications of CNNs to non-discriminative tasks, such as super-resolution [6] or inferring depth from a single image [8].”
Perceptual Losses for Real-Time Style Transfer and Super-Resolution (2016, methodology): “Recent methods for depth [5,6,18] and surface normal estimation [6,19] are similar, transforming color input images into geometrically meaningful output images using a feed-forward convolutional network trained with per-pixel regression [5,6] or classification [19] losses.”
Conditional Random Fields as Recurrent Neural Networks (2015, background): “[6] propose a coarse-to-fine approach to achieve depth prediction from single images.”
Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture (2014, methodology): “Ladicky[20]Karsch[18] Baig [1] Liu [23] Eigen[8] Ours(A) Ours(VGG)”
Attribute2Image: Conditional Image Generation from Visual Attributes (2015, methodology): “We augment the training data with the following image transformations [22, 9]: 1) ﬂipping images horizontally with probability 0 .”
Designing deep networks for surface normal estimation (2014, background): “Unfortunately, while all of these local primitives work well on things like blinds, cabinets, and tile floors, they tend to be stymied by local ambiguities at less-textured regions.”
A Point Set Generation Network for 3D Object Reconstruction from a Single Image (2016, background): “As commodity 3D sensors become popular, RGBD database has been built and used to train learningbased systems [6, 8].”
Variational Information Distillation for Knowledge Transfer (2019, background): “, depth estimation [8], pose estimation [26], optical flow [7], object classification [11], detection [10], and segmentation [25].”
SafetyNet: Detecting and Rejecting Adversarial Examples Robustly (2017, methodology): ““Regression” methods used in both train and test are: random swaps of depth and image planes; single image predicted depth [5]; rectangle cropped region insertion and random shifted or scaled misaligned depth and image.”
DEEPCON: Protein Contact Prediction using Dilated Convolutional Neural Networks with Dropout (2019, background): “From the perspective of input and output data format, the protein contact prediction problem is similar to depth prediction [13] in computer vision, i.e. predicting 3D depth from 2D images.”
Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, background): “Over the last decade, convolutional neural networks (CNNs) have been established as the standard approach for many computer vision tasks, like image classification [26,58,17], object detection [15,52,34], semantic segmentation [35,3,67], and monocular depth estimation [12,27].”
Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos (2016, background): “Although some works are done towards this direction [10], recovering depth from RGB in general is an underdetermined problem.”
Learning a Multi-View Stereo Machine (2017, background): “Approaches which predict shape from a single image have been proposed in form of direct depth map regression [46, 31, 10], generating multiple depth maps from novel viewpoints [51], producing voxel occupancies [5, 16], geometry images [49] and point clouds [11].”
Hierarchical Surface Prediction for 3D Object Reconstruction (2017, background): “5D setting where the goal is, given a color image to predict a depth map [11], [21], [30].”
3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction (2016, methodology): “Inspired by the success of Long Short-Term Memory (LSTM) [33] networks [34,35] as well as recent progress in single-view 3D reconstruction using Convolutional Neural Networks [36,37], we propose a novel architecture that we call the 3D Recurrent Reconstruction Neural Network (3D-R2N2).”