Deep Depth Completion of a Single RGB-D Image

By Yinda Zhang, Thomas A. Funkhouser. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

445 citations recorded by the source corpus.

The goal of our work is to complete the depth channel of an RGB-D image. Commodity-grade depth cameras often fail to sense depth for shiny, bright, transparent, and distant surfaces. To address this problem, we train a deep network that takes an RGB image as input and predicts dense surface normals and occlusion boundaries. Those predictions are then combined with raw depth observations provided by the RGB-D camera to solve for depths for all pixels, including those missing in the original observation. This method was chosen over others (e.g., inpainting depths directly) as the result of extensive experiments with a new depth completion benchmark dataset, where holes are filled in training data through the rendering of surface reconstructions created from multiview RGB-D scans. Experiments with different network inputs, depth representations, loss functions, optimization methods, inpainting methods, and deep depth estimation networks show that our proposed approach provides better depth completions than these alternatives.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9766552d5d25519ed50bcafaf88b7701d2533477

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Surface Normal Estimation With Hierarchical RGB-D Fusion (2019, methodology): “For the corresponding ground-truth normal data, we use the render normal provided by [32] which was generated with multiview reconstruction.”
DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene From Sparse LiDAR Data and Single Color Image (2018, methodology): “[55] performs well on close regions, but worse than our method in far areas and where surface normal estimation fails, e.”
Learning Guided Convolutional Network for Depth Completion (2019, background): “[31] and ‘DeepLiDAR’ [6] are obtained from DeepLiDAR [6].”
Learning Depth with Convolutional Spatial Propagation Network (2018, background): “Other works [75] complete depth with an RGB image in a manner of in-painting, where dense depth values are partially available.”
3D Photography Using Context-Aware Layered Depth Inpainting (2020, background): “, transparent/reflective/distant surfaces) [35, 70, 36] or performing image editing tasks such as object removal on stereo images [57, 40].”
Clear Grasp: 3D Shape Estimation of Transparent Objects for Manipulation (2019, methodology): “…image (with all pixels corresponding to transparent surface removed) and predictions of surface normals and occlusion and contact edges, ClearGrasp reconstructs the 3D surfaces of transparent objects (missing depth region) via the global optimization algorithm proposed by Zhang and Funkhouser [59].”
Neural RGB-D Surface Reconstruction (2021, methodology): “To reduce artifacts from classical reconstruction methods, a series of methods was proposed that use learned spatial priors to predict depth maps from color images [24, 28, 39], to learn multiview stereo using 3D CNNs on voxel grids [34, 68, 82], or multi-plane images [22], to reduce the influence o”
Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR and Monocular Camera (2018, background): “, Microsoft Kinect) [11] is sometimes also referred to as depth inpainting [12], or depth enhancement [5–7] when noise is taken into account.”