Deep convolutional neural fields for depth estimation from a single image

By Fayao Liu, Chunhua Shen, Guosheng Lin. 2014

934 citations recorded by the source corpus.

We consider the problem of depth estimation from a single monocular image in this work. It is a challenging task as no reliable depth cues are available, e.g., stereo correspondences, motions etc. Previous efforts have been focusing on exploiting geometric priors or additional sources of information, with all using hand-crafted features. Recently, there is mounting evidence that features from deep convolutional neural networks (CNN) are setting new records for various vision applications. On the other hand, considering the continuous characteristic of the depth values, depth estimations can be naturally formulated into a continuous conditional random field (CRF) learning problem. Therefore, we in this paper present a deep convolutional neural field model for estimating depths from a single image, aiming to jointly explore the capacity of deep CNN and continuous CRF. Specifically, we propose a deep structured learning scheme which learns the unary and pairwise potentials of continuous CRF in a unified deep CNN framework. The proposed method can be used for depth estimations of general scenes with no geometric priors nor any extra information injected. In our case, the integral of the partition function can be analytically calculated, thus we can exactly solve the log-likelihood optimization. Moreover, solving the MAP problem for predicting depths of a new image is highly efficient as closed-form solutions exist. We experimentally demonstrate that the proposed method outperforms state-of-the-art depth estimation methods on both indoor and outdoor scene datasets.

Subfields: Computer Vision and Pattern Recognition, Media Technology, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

In 3D Vision & Pose

Citations in the corpus

Towards fairer datasets: filtering and balancing the distribution of the people subtree in the ImageNet hierarchy (2019, methodology): “In addition, the feature representation learned on ImageNet images has been used as a backbone for a variety of computer vision tasks such as object detection [33, 61], human activity understanding [68], image captioning [74], and recovering depth from a single RGB image [49], to name a few.”
Perceptual Losses for Real-Time Style Transfer and Super-Resolution (2016, methodology): “Some methods move beyond per-pixel losses by penalizing image gradients [5] or using a CRF loss layer [16] to enforce local consistency in the output image.”
Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture (2014, background): “Ladicky[20]Karsch[18] Baig [1] Liu [23] Eigen[8] Ours(A) Ours(VGG)”
Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation (2015, background): “FCNNs have also been applied to a range of other dense-prediction tasks recently, such as image restoration [10], image super-resolution [8] and depth estimation [11, 29].”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, background): “normal estimation [2, 25, 103], depth estimation [25, 63, 64], intrinsic image decomposition [73] shadow detection [108], texture classification [19], material classification [4], saliency estimation [58, 112], semantic segmentation [14, 27,67], region proposal generation [30,82,85], instance segmen”
Synthetic Data for Text Localisation in Natural Images (2016, methodology): “The normals are estimated automatically by first predicting a dense depth map using the CNN of [30] for the regions segmented above, and then fitting a planar facet to it using RANSAC [10].”
Deep Level Sets: Implicit Surface Representations for 3D Shape Inference (2019, methodology): “Convolutional neural networks [9,12,16,22,28] and generative adversarial models (GANs) [3, 24] have been successfully applied to 3D reconstruction problems by using either volumetric or point cloud representations.”
3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction (2016, methodology): “Inspired by the success of Long Short-Term Memory (LSTM) [33] networks [34,35] as well as recent progress in single-view 3D reconstruction using Convolutional Neural Networks [36,37], we propose a novel architecture that we call the 3D Recurrent Reconstruction Neural Network (3D-R2N2).”
(CAD)$^2$RL: Real Single-Image Flight without a Single Real Image (2016, background): “in the hallways, more detailed depth estimation [36, 24, 23] could also be used without loss of generality.”
RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation (2016, background): “, depth estimation [15, 13, 33], image restoration [14], image super-resolution [12].”
Marr Revisited: 2D-3D Alignment via Surface Normal Prediction (2016, background): “Most similar to our surface normal prediction approach is recent work that trains a CNN to directly predict depth [27], jointly predicts surface normals, depth, and object labels [9], or combines CNN features with the global room layout via a predicted 3D box [46].”
Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer (2019, background): “Various architectural innovations have been proposed to enhance prediction accuracy [25, 36, 29, 12, 26].”
Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age (2016, neutral): “Note that the list of benchmarking problems is usually restricted to a small set of sensors, while some of the datasets (i.e., the ones used to exercise the back-end) are agnostic to the sensor type.”
Deeper Depth Prediction with Fully Convolutional Residual Networks (2016, methodology): “[19] propose to learn the unary and pairwise potentials during CNN training in the form of a CRF loss, while Li et al.”
A Study of Face Obfuscation in ImageNet (2021, background): “…on the original/obfuscated images and ﬁnetuning on 4 down-stream tasks: object recognition on CIFAR-10 (Krizhevsky et al., 2009), scene recognition on SUN (Xiao et al., 2010), object detection on PASCAL VOC (Everingham et al., 2010), and face attribute classiﬁcation on CelebA (Liu et al., 2015b).”
Learning to rank in person re-identification with metric ensembles (2015, background): “Future works include learning mid-level features [21] for person re-id, incorporating depth from a single monocular image [23], integrating person re-id with person detector [29, 30] and improving multiple target tracking of [26] with the proposed approach.”
Learning from Synthetic Humans (2017, methodology): “In this section, we present our approach for human body part segmentation [5, 24] and human depth estimation [10, 11, 18], which we train with synthetic and/or real data, see Section 5 for the evaluation.”
DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency (2018, background): “The availability of RGB-D datasets and deep learning leads to a line of work on single-view depth estimation [13,14,35,38,62,72].”