UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction

By Michael Oechsle, Songyou Peng, Andreas Geiger. IEEE International Conference on Computer Vision, 2021

907 citations recorded by the source corpus.

Neural implicit 3D representations have emerged as a powerful paradigm for reconstructing surfaces from multi-view images and synthesizing novel views. Unfortunately, existing methods such as DVR or IDR require accurate per-pixel object masks as supervision. At the same time, neural radiance fields have revolutionized novel view synthesis. However, NeRF’s estimated volume density does not admit accurate surface reconstruction. Our key insight is that implicit surface models and radiance fields can be formulated in a unified way, enabling both surface and volume rendering using the same model. This unified perspective enables novel, more efficient sampling procedures and the ability to reconstruct accurate surfaces without input masks. We compare our method on the DTU, BlendedMVS, and a synthetic indoor dataset. Our experiments demonstrate that we outperform NeRF in terms of reconstruction quality while performing on par with IDR without requiring masks.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f184a47adced84a1f7f99bd052d8f30f0587d6db

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

NICE-SLAM: Neural Implicit Scalable Encoding for SLAM (2021, methodology): “Similar to [34], we model the ray termination probability at point pi asw i = o 0 pi ∏i−1 j=1(1−o0pj ) for coarse level, andw i = opi ∏i−1 j=1(1−opj ) for fine level.”
Vox-Fusion: Dense Tracking and Mapping with Voxel-based Neural Implicit Representation (2022, background): “They propose to identify the surface via iterative root-finding [22], weighting the rendered color with the associated SDF values [36], forcing the network to learn more details near the surface within a pre-defined truncation distance [1].”
NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction (2021, methodology): “UNISURF [30], a concurrent work, also learns an implicit surface via volume rendering.”
Multi-View Mesh Reconstruction with Neural Deferred Shading (2022, positive): “Other methods instead model the rendering operation with neural networks [41,43,64], i.”
SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering (2023, methodology): “Retrieving such an editable mesh for realistic rendering is done within minutes with our method, compared to hours with the state-of-the-art method on SDFs, while providing a better rendering quality.”
Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering (2023, background): “They directly use spatial coordinates (and viewing direction) as input to the MLP and predict point-wise attribute, e.g . signed distance to scene surface (SDF) [33, 34, 46, 54], or density and color of that point [2, 30, 49].”
Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation (2022, background): “Neural radiance fields (NeRF) is a family of inverse rendering algorithms that have excelled at multiview 3D reconstruction tasks including view synthesis and surface geometry estimation [31, 34, 40, 57, 61].”
NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction (2022, background): “Neural implicit representations have made remarkable achievements in novel view synthesis [53, 33, 22, 35, 30, 52] and 3D/4D reconstruction [47, 48, 66, 37, 23, 20, 32, 39, 59, 65].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, neutral): “UNISERF [138] (April 2021) reconstructed scene surfaces by replacing the alpha value ai at the i-th sample point used in the discretized volume rendering equation, given by (5), with a discrete occupancy function o(x) = 1 in occupied”
Shape As Points: A Differentiable Poisson Solver (2021, background): “However, as such representations implicitly encode surface information, extracting the underlying surface is typically slow as they require numerous network evaluations in 3D space for extracting complete surfaces using marching cubes [10, 40, 50], or along rays for intersection detection in the con”
Urban Radiance Fields (2021, methodology): “Radiance ﬁelds are also used for surface extraction, either using an SDF representation [66,70,71] or solid surfaces [46].”
2D Gaussian Splatting for Geometrically Accurate Radiance Fields (2024, positive): “For instance, Mip-NeRF [Barron et al. 2021] and subsequent works [Barron et al. 2022a, 2023; Hu et al. 2023] tackle NeRF’s aliasing issues.”
DUSt3R: Geometric 3D Vision Made Easy (2023, neutral): “In the end, modern SfM and MVS pipelines boil down to solving a series of minimal problems : matching points, finding essential matrices, triangulating points, sparsely re-constructing the scene, estimating cameras and finally performing dense reconstruction.”
MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction (2022, methodology): “More specifically, we investigate the following architectures: a single, large MLP [46, 71, 75, 76], a dense SDF grid [30], a single feature grid [27, 36, 50, 51] and multi-resolution feature grids [10, 21, 42, 65, 83].”
Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields (2021, positive): “This goal is shared by recent works that combine NeRF’s neural volumetric representation with neural implicit surface representations [23,37,41].”