Deeper Depth Prediction with Fully Convolutional Residual Networks

By Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, Nassir Navab. 2016

2,021 citations recorded by the source corpus.

This paper addresses the problem of estimating the depth map of a scene given a single RGB image. We propose a fully convolutional architecture, encompassing residual learning, to model the ambiguous mapping between monocular images and depth maps. In order to improve the output resolution, we present a novel way to efficiently learn feature map up-sampling within the network. For optimization, we introduce the reverse Huber loss that is particularly suited for the task at hand and driven by the value distributions commonly present in depth maps. Our model is composed of a single architecture that is trained end-to-end and does not rely on post-processing techniques, such as CRFs or other additional refinement steps. As a result, it runs in real-time on images or videos. In the evaluation, we show that the proposed model contains fewer parameters and requires fewer training data than the current state of the art, while outperforming all approaches on depth estimation. Code and models are publicly available.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “The setup is based on [83] and detailed in Appendix E.”
Multi-task Self-Supervised Visual Learning (2017, methodology): “We use a modiﬁed version of the architecture proposed in Laina et al. [19].”
Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, background): “Over the last decade, convolutional neural networks (CNNs) have been established as the standard approach for many computer vision tasks, like image classification [26,58,17], object detection [15,52,34], semantic segmentation [35,3,67], and monocular depth estimation [12,27].”
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? (2017, methodology): “We evaluate our method using the same standard as [26], resizing images to 345 × 460 pixels and evaluating on pixels with depth less than 70m.”
Taskonomy: Disentangling Task Transfer Learning (2018, methodology): “released models of Fully Convolutional Residual Network [19].”
IQA: Visual Question Answering in Interactive Environments (2017, methodology): “First, we use the ground truth object detections and depth instead of YOLO [56] and FRCN depth [38].”
Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cells (2018, methodology): “We upsample the logits to the size of the target mask and use the inverse Huber loss [4] for optimisation, ignoring pixels with missing depth measurements.”
Learning 3D Object Categories by Looking Around Them (2017, methodology): “BerHu-Net is a variant of the state-of-theart depth prediction network from [18] based on the same Φdepth subnetwork as VpDR-Net (but dropping Φpcl and Φvp).”
DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene From Sparse LiDAR Data and Single Color Image (2018, methodology): “[25] proposed a single-scale but deeper fully convolutional architecture.”
Im2Pano3D: Extrapolating 360° Structure and Semantics Beyond the Field of View (2017, background): “Current methods for direct regressing raw depth values produce blurred results [34, 20, 7], partly due to the viewpointdependent nature of depth maps and the large value variance of depth values even for nearby pixels on the same 3D plane.”
Refine and Distill: Exploiting Cycle-Inconsistency and Knowledge Distillation for Unsupervised Monocular Depth Estimation (2019, background): “The vast majority of methods focus on a supervised setting and the problem of predicting depth maps is cast as a pixellevel regression problem [5, 25, 44, 22, 38, 40, 6].”
Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies (2018, background): “depth estimation [18, 43], object classification [42, 32], detection [64, 25], segmentation [70, 30, 35], pose estimation [92, 9, 87], etc.”
Consistent video depth estimation (2020, methodology): “Deep learning basedmodels have been successfully applied to single image depth estimation [Eigen and Fergus 2015; Eigen et al. 2014; Fu et al. 2018; Laina et al. 2016; Liu et al. 2015].”
D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry (2020, background): “Laina et al. [43] propose a robust loss function to improve the estimation accuracy.”
Estimating Depth From Monocular Images as Classification Using Deep Fully Convolutional Residual Networks (2016, methodology): “Lastly, in order to obtain high resolution predictions, [27] applied an up-sampling scheme while we simply use bilinear interpolation.”
From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation (2019, methodology): “[37] was introduced, considerable improvements have been made along with rapid advances in deep learning [12, 11, 28, 29, 43, 35, 21, 25].”