NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction
By Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, Wenping Wang. Neural Information Processing Systems, 2021
2,405 citations recorded by the source corpus.
We present a novel neural surface reconstruction method, called NeuS, for reconstructing objects and scenes with high fidelity from 2D image inputs. Existing neural surface reconstruction approaches, such as DVR and IDR, require foreground mask as supervision, easily get trapped in local minima, and therefore struggle with the reconstruction of objects with severe self-occlusion or thin structures. Meanwhile, recent neural methods for novel view synthesis, such as NeRF and its variants, use volume rendering to produce a neural scene representation with robustness of optimization, even for highly complex objects. However, extracting high-quality surfaces from this learned implicit representation is difficult because there are not sufficient surface constraints in the representation. In NeuS, we propose to represent a surface as the zero-level set of a signed distance function (SDF) and develop a new volume rendering method to train a neural SDF representation. We observe that the conventional volume rendering method causes inherent geometric errors (i.e. bias) for surface reconstruction, and therefore propose a new formulation that is free of bias in the first order of approximation, thus leading to more accurate surface reconstruction even without the mask supervision. Experiments on the DTU dataset and the BlendedMVS dataset show that NeuS outperforms the state-of-the-arts in high-quality surface reconstruction, especially for objects and scenes with complex structures and self-occlusion.
Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/cf5647cb2613f5f697729eab567383006dcd4913
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
- Andreas Geiger
- Jonathan T. Barron
- Matthias Nießner
- Ben Mildenhall
- Pratul P. Srinivasan
- Andrea Tagliasacchi
- Michael Niemeyer
- Songyou Peng
In Computer Graphics and Computer-Aided Design
- Andreas Geiger
- Jonathan T. Barron
- Ben Mildenhall
- Pratul P. Srinivasan
- Matthias Nießner
- Peter Hedman
- Anpei Chen
- Zehao Yu
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Andreas Geiger
- Jitendra Malik
- Trevor Darrell
- Xiaogang Wang
- Andrea Vedaldi
Citations in the corpus
- SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes (2023, neutral): “However, extracting scene geometry and appearance from limited observations [30, 31, 49] is challenging.”
- Neuralangelo: High-Fidelity Neural Surface Reconstruction (2023, methodology): “Neuralangelo adopts Instant NGP as a neural SDF representation of the underlying 3D scene, optimized from multi-view image observations via neural surface rendering [36].”
- UniSim: A Neural Closed-Loop Sensor Simulator (2023, methodology): “To learn high-quality geometry [74,86], we adopt the signed distance function (SDF) as our implicit geometry representation s.”
- OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation (2023, methodology): “NeuS with sparseview input achieves a surprisingly good performance.”
- Continuous 3D Perception Model with Persistent State (2025, neutral): “…traditional methods such as Structure from Motion (SfM) [1, 2, 28, 47, 65, 66, 70, 71, 80] and Simultaneous Localization and Mapping (SLAM) [11, 16, 19, 22, 37, 51, 53], as well as more recent approaches such as Neural Radiance Fields (NeRF) [12, 24, 49, 50, 84] and 3D Gaussian Splatting [36].”
- Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (2023, background): “NeRF does not store any 3D geometry explicitly (only the density field), and several works propose to use a signed distance function to recover a scene’s surface [98, 89, 97, 90, 13], including in the few-shot setting as well (e.”
- Neural RGB-D Surface Reconstruction (2021, methodology): “NeuS [76] is also a concurrent work on novel view synthesis which uses a signed distance function to represent the geometry, but takes only RGB images as input, and thus fails to reconstruct the geometry of featureless surfaces, like white walls.”
- VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, neutral): “…becomes a popular 3D representation for photo-realistic novel-view synthesis in recent years [35], many variants are proposed to improve quality [2–4, 24, 45, 47–49, 57], increase speed [8, 9, 11, 14, 20, 32, 36, 37, 40, 43, 46, 58, 60], extend to dynamic scenes [7, 15, 18, 25, 27, 50], and so on.”
- Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation (2022, background): “Neural radiance fields (NeRF) is a family of inverse rendering algorithms that have excelled at multiview 3D reconstruction tasks including view synthesis and surface geometry estimation [31, 34, 40, 57, 61].”
- Geo-Neus: Geometry-Consistent Neural Implicit Surfaces Learning for Multi-view Reconstruction (2022, methodology): “Inspired by neural volume rendering [21, 42] that simultaneously learns volume density and radiance field from input images, recent works [33, 39] use signed distance functions (SDF) [25] for surface representation and introduce the SDF-induced density function to enable the volume rendering to…”
- Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering (2023, background): “They directly use spatial coordinates (and viewing direction) as input to the MLP and predict point-wise attribute, e.g . signed distance to scene surface (SDF) [33, 34, 46, 54], or density and color of that point [2, 30, 49].”
- NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, neutral): “3D Reconstruction SDF Neural Surface [136], Neural RGB-D [137]”
- PGSR: Planar-Based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction (2024, positive): “Consequently, several approaches have transformed NeRF-based network architectures into surface reconstruction frameworks by incorporating intermediate representations such as occupancy [50] or signed distance fields [19], [51].”
- Extracting Triangular 3D Models, Materials, and Lighting From Images (2021, background): “Unisurf [47] is a hybrid approach that gradually reduces the sampling region, encouraging a volumetric representation to converge to a surface, and NeuS [59] provides an unbiased conversion from SDF into density for volume rendering.”
- SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering (2023, methodology): “Retrieving such an editable mesh for realistic rendering is done within minutes with our method, compared to hours with the state-of-the-art method on SDFs, while providing a better rendering quality.”
- Co-SLAM: Joint Coordinate and Sparse Parametric Encodings for Neural Real-Time SLAM (2023, methodology): “While these methods focus on novel view synthesis, others instead focus on surface reconstruction from RGB images, with implicit surface representations and differentiable renderers [8, 33, 39, 40].”
- Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (2023, background): “…through the use of MLPs. Subsequent research has expanded the utility of NeRF to various applications, encompassing tasks such as mesh reconstruction from a collection of images [20, 45], inverse rendering [5, 25, 54], optimization of camera parameters [21, 47, 48], and few-shot learning [10, 51].”