Neural RGB-D Surface Reconstruction

By Dejan Azinovi'c, Ricardo Martin-Brualla, Dan B. Goldman, Matthias Nießner, Justus Thies. Computer Vision and Pattern Recognition, 2021

423 citations recorded by the source corpus.

Obtaining high-quality 3D reconstructions of room-scale scenes is of paramount importance for upcoming applications in AR or VR. These range from mixed reality applications for teleconferencing, virtual measuring, virtual room planing, to robotic applications. While current volume-based view synthesis methods that use neural radiance fields (NeRFs) show promising results in reproducing the appearance of an object or scene, they do not reconstruct an actual surface. The volumetric representation of the surface based on densities leads to artifacts when a surface is extracted using Marching Cubes, since during optimization, densities are accumulated along the ray and are not used at a single sample point in isolation. Instead of this volumetric representation of the surface, we propose to represent the surface using an implicit function (truncated signed distance function). We show how to incorporate this representation in the NeRF framework, and extend it to use depth measurements from a commodity RGB-D sensor, such as a Kinect. In addition, we propose a pose and camera re-finement technique which improves the overall reconstruction quality. In contrast to concurrent work on integrating depth priors in NeRF which concentrates on novel view synthesis, our approach is able to reconstruct high-quality, metrical 3D reconstructions.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/03074d197dda05614f93509c0938d66bf5993236

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass (2025, neutral): “This is critical for allowing the model to implicitly reason about camera pose jointly for all images from an otherwise permutationally invariant set of tokens.”
NICE-SLAM: Neural Implicit Scalable Encoding for SLAM (2021, background): “A few recent papers [1, 4, 9, 27, 48, 57, 61] attempt to predict scene-level geometry with RGB-(D) inputs, but they all assume given camera poses.”
Vox-Fusion: Dense Tracking and Mapping with Voxel-based Neural Implicit Representation (2022, methodology): “Unlike methods such as [23] that force the network to learn a negative truncation value−tr for points behind the truncation region, we simply mask out these points during rendering to avoid solving surface intersection ambiguities [36] as proposed in [1].”
MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction (2022, background): “Joint optimization of scene representations and camera parameters [4, 83] is another interesting direction, especially for multi-resolution grids, in order to better handle noisy camera poses.”
Continuous 3D Perception Model with Persistent State (2025, neutral): “3D reconstruction comparison on 7-Scenes [67] and NRGBD [4] datasets.”
Depth-supervised NeRF: Fewer Views and Faster Training for Free (2021, background): “For example, recent works have demonstrated how depth extracted from sensors like time-of-flight cameras [1] or RGB-D Kinect sensor [2] can be applied to fit implicit functions.”
NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields (2022, background): “Further, differently than other depth-supervised approaches [9], [33], [27], [34], [35], our depth loss is weighted by the depth’s marginal covariance, and the depth is estimated from optical-flow rather than measured by an RGB-D camera.”
Urban Radiance Fields (2021, methodology): “An alternative is to provide the exposure code β i directly as an input to the network, as done in NeRF-W [39] and [3].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, neutral): “The scans were performed using RGB-D sensors attached to handheld devices such as iPhone/iPad.”