Volume Rendering of Neural Implicit Surfaces

By Lior Yariv, Jiatao Gu, Yoni Kasten, Yaron Lipman. Neural Information Processing Systems, 2021

1,381 citations recorded by the source corpus.

Neural volume rendering became increasingly popular recently due to its success in synthesizing novel views of a scene from a sparse set of input images. So far, the geometry learned by neural volume rendering techniques was modeled using a generic density function. Furthermore, the geometry itself was extracted using an arbitrary level set of the density function leading to a noisy, often low fidelity reconstruction. The goal of this paper is to improve geometry representation and reconstruction in neural volume rendering. We achieve that by modeling the volume density as a function of the geometry. This is in contrast to previous work modeling the geometry as a function of the volume density. In more detail, we define the volume density function as Laplace's cumulative distribution function (CDF) applied to a signed distance function (SDF) representation. This simple density representation has three benefits: (i) it provides a useful inductive bias to the geometry learned in the neural volume rendering process; (ii) it facilitates a bound on the opacity approximation error, leading to an accurate sampling of the viewing ray. Accurate sampling is important to provide a precise coupling of geometry and radiance; and (iii) it allows efficient unsupervised disentanglement of shape and appearance in volume rendering. Applying this new density representation to challenging scene multiview datasets produced high quality geometry reconstructions, outperforming relevant baselines. Furthermore, switching shape and appearance between scenes is possible due to the disentanglement of the two.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/eded1f3acaba853499fd5a6b3de63fa9d5e0cef2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior (2022, methodology): “A number of techniques are proposed to improve NeRF’s performance with additional regularisation [13, 26, 54], depth priors [7,32,47,53], surface enhancements [27,43,49] or latent codes [40, 44, 52].”
Efficient Geometry-aware 3D Generative Adversarial Networks (2021, methodology): “To further improve the quality of the learned shapes, we could instill a stronger geometry prior or regularize the density component of the radiance field following methods proposed by [51, 67, 69].”
NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction (2022, background): “Neural implicit representations have made remarkable achievements in novel view synthesis [53, 33, 22, 35, 30, 52] and 3D/4D reconstruction [47, 48, 66, 37, 23, 20, 32, 39, 59, 65].”
MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes (2023, background): “On the other hand, increasing sparsity and moving closer [Wang et al. 2021; Yariv et al. 2021] or completely [Chen et al.”
Advances in Neural Rendering (2021, background): “Recently, [OPG21] and [YGKL21b] addressed this issue by formulating the surface function into the volumetric rendering formulation (introduced below); on the other hand [ZYQ21] use off-the-shelf depth estimation methods to generate pseudo ground truth signed distance values to assist the training…”
Urban Radiance Fields (2021, methodology): “Radiance ﬁelds are also used for surface extraction, either using an SDF representation [66,70,71] or solid surfaces [46].”
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, neutral): “…becomes a popular 3D representation for photo-realistic novel-view synthesis in recent years [35], many variants are proposed to improve quality [2–4, 24, 45, 47–49, 57], increase speed [8, 9, 11, 14, 20, 32, 36, 37, 40, 43, 46, 58, 60], extend to dynamic scenes [7, 15, 18, 25, 27, 50], and so on.”
Tensor4D: Efficient Neural 4D Decomposition for High-Fidelity Dynamic Reconstruction and Rendering (2022, methodology): “More recently, several methods unify surface and volumetric rendering, enabling accurate geometry reconstruction and high-quality novel view synthesis [36, 59, 63].”
Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering (2023, background): “They directly use spatial coordinates (and viewing direction) as input to the MLP and predict point-wise attribute, e.g . signed distance to scene surface (SDF) [33, 34, 46, 54], or density and color of that point [2, 30, 49].”
SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering (2023, methodology): “Retrieving such an editable mesh for realistic rendering is done within minutes with our method, compared to hours with the state-of-the-art method on SDFs, while providing a better rendering quality.”
Geo-Neus: Geometry-Consistent Neural Implicit Surfaces Learning for Multi-view Reconstruction (2022, methodology): “Inspired by neural volume rendering [21, 42] that simultaneously learns volume density and radiance ﬁeld from input images, recent works [33, 39] use signed distance functions (SDF) [25] for surface representation and introduce the SDF-induced density function to enable the volume rendering to…”
StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation (2021, methodology): “As shown in VolSDF [72], the SDF can be serve as a proxy for the density function used for the traditional volume rendering [46].”
BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis (2023, methodology): “Our representation combines the benefits of mip-NeRF 360 for representing unbounded scenes with the well-behaved surface properties of VolSDF’s hybrid volume-surface representation [Yariv et al. 2021].”