State of the Art on 3D Reconstruction with RGB‐D Cameras

By Michael Zollhöfer, P. Stotko, A. Görlitz, Christian Theobalt, Matthias Nießner, Reinhard Klein, Andreas Kolb. Computer graphics forum (Print), 2018

342 citations recorded by the source corpus.

The advent of affordable consumer grade RGB‐D cameras has brought about a profound advancement of visual scene reconstruction methods. Both computer graphics and computer vision researchers spend significant effort to develop entirely new algorithms to capture comprehensive shape models of static and dynamic scenes with RGB‐D cameras. This led to significant advances of the state of the art along several dimensions. Some methods achieve very high reconstruction detail, despite limited sensor resolution. Others even achieve real‐time performance, yet possibly at lower quality. New concepts were developed to capture scenes at larger spatial and temporal extent. Other recent algorithms flank shape reconstruction with concurrent material and lighting estimation, even in general scenes and unconstrained conditions. In this state‐of‐the‐art report, we analyze these recent developments in RGB‐D scene reconstruction in detail and review essential related work. We explain, compare, and critically analyze the common underlying algorithmic concepts that enabled these recent advancements. Furthermore, we show how algorithms are designed to best exploit the benefits of RGB‐D data while suppressing their often non‐trivial data distortions. In addition, this report identifies and discusses important open research questions and suggests relevant directions for future work.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/484024a91c1cb97b358c2cabd9fa427fcc6d52e3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

Unmanned Aircraft System (UAS) Technology and Applications in Agriculture (2019, background): “for imaging purposes include: altitude monitoring during spraying [92]; phenotyping [90]; and 3D modeling [93,94].”
Space-time Neural Irradiance Fields for Free-Viewpoint Video (2020, background): “A line of research work focuses on fusing a sequence of RGB-Depth images in a video into a global with voxel-, point-based, signed distance field-based representation [80].”
RTAB‐Map as an open‐source lidar and visual simultaneous localization and mapping library for large‐scale and long‐term online operation (2018, methodology): “Regarding visual SLAM, many open-source approaches exist but not many can be easily used on a robot (consult [Zollhöfer et al., 2018] for a review on 3D reconstruction focused approaches).”
Neural RGB-D Surface Reconstruction (2021, methodology): “It is related to classical fusion-based 3D reconstruction methods [9, 14, 50, 53, 92], learned 3D reconstruction [7, 15, 47, 58, 79], as well as to recent coordinate-based scene representation models [48, 69, 74].”
Resolving 3D Human Pose Ambiguities With 3D Scene Constraints (2019, background): “Traditionally, human pose estimation methods [43] estimate bodies in isolation ignoring the surrounding world, while 3D reconstruction methods focus on acquiring the dense 3D shape of the scene only [76] or performing semantic analysis [7, 13, 54], assuming no humans are present.”
ReFusion: 3D Reconstruction in Dynamic Environments for RGB-D Cameras Exploiting Residuals (2019, methodology): “With the advent of inexpensive RGB-D cameras, many approaches for mapping using such sensors were proposed [15], [20].”
Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video (2020, neutral): “Some approaches allow for the reconstruction of general non-rigid objects [104, 17, 33, 72], but most methods only reconstruct the geometry without the appearance of the objects in the scene.”
BAD SLAM: Bundle Adjusted Direct RGB-D SLAM (2019, methodology): “In practice, we encountered problems when using this parameterization in combination with the point-to-plane residuals typically used for RGB-D SLAM [73].”