High quality depth map upsampling for 3D-TOF cameras

By Jaesik Park, Hyeongwoo Kim, Yu-Wing Tai, Michael S. Brown, In-So Kweon. 2011

540 citations recorded by the source corpus.

This paper describes an application framework to perform high quality upsampling on depth maps captured from a low-resolution and noisy 3D time-of-flight (3D-ToF) camera that has been coupled with a high-resolution RGB camera. Our framework is inspired by recent work that uses nonlocal means filtering to regularize depth maps in order to maintain fine detail and structure. Our framework extends this regularization with an additional edge weighting scheme based on several image features based on the additional high-resolution RGB input. Quantitative and qualitative results show that our method outperforms existing approaches for 3D-ToF upsampling. We describe the complete process for this system, including device calibration, scene warping for input alignment, and even how the results can be further processed using simple user markup.

Subfields: Computer Vision and Pattern Recognition, Instrumentation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Instrumentation

Andreas Geiger

Citations in the corpus

Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018, background): “This research also extends to depth map super-resolution [22, 28, 11].”
OctNetFusion: Learning Depth Fusion from Data (2017, methodology): “To simulate real data, we added depth dependent Gaussian noise to the inputs as proposed in [33].”
Sparsity Invariant CNNs (2017, background): “More advanced approaches are based on global energy minimization [1, 6, 12, 49, 51], compressive sensing [22], or incorporate semantics for improved performance [58].”
The Fast Bilateral Solver (2015, methodology): “With the advent of consumer depth sensors, techniques have been proposed for the task of upsampling the noisy depth maps produced by these sensors using a highresolution RGB reference image [8, 17, 22, 24, 26, 27].”
Deep Joint Image Filtering (2016, methodology): “Among them, JBU [4], GF [6] and Ham [13] are generic methods for joint image upsampling while MRF [16], TGV [3] and Park [2] are algorithms specifically designed for image guided depth upsampling.”
Color-Guided Depth Recovery From RGB-D Data Using an Adaptive Autoregressive Model (2014, methodology): “Without resorting to higher level tools such as segmentation used in [11], our proposed method achieves the best quality among several state-of-the-art depth recovery methods.”
Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation (2013, methodology): “In contrast, our method builds on the success of recently introduced upsampling methods using MRF and least squares optimization [5] [15].”
Deep Depth Completion of a Single RGB-D Image (2018, background): “They have exploited a variety of approaches, including Markov random ﬁelds [48, 15, 46, 56, 63], shape-from-shading [27, 76], segmentation [45], and dictionary methods [21, 34, 49, 69].”
Learning Guided Convolutional Network for Depth Completion (2019, background): “…information, edge information, or surface information, etc. Earlier works mainly address depth super-resolution with bilateral ﬁltering [16], or global energy minimization [26]–[28], where the depth completion is guided by image [16], [26]–[28], semantic segmentation [29] or edge information [30].”
Depth Map Super-Resolution by Deep Multi-Scale Guidance (2016, methodology): “We do not explicitly formulate an optimization problem as the global methods [2, 4, 5, 15] or design a fixed filter as the local methods [6, 13, 14] because CNN can be trained to address the upsampling problem.”
Patch Based Synthesis for Single Depth Image Super-Resolution (2012, background): “This is consistent with the same observation for bilinear interpolation reported in [7].”