Neural Sparse Voxel Fields
By Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-seng Chua, Christian Theobalt. 2020
1,503 citations recorded by the source corpus.
Photo-realistic free-viewpoint rendering of real-world scenes using classical computer graphics techniques is challenging, because it requires the difficult step of capturing detailed appearance and geometry models. Recent studies have demonstrated promising results by learning scene representations that implicitly encode both geometry and appearance without 3D supervision. However, existing approaches in practice often show blurry renderings caused by the limited network capacity or the difficulty in finding accurate intersections of camera rays with the scene geometry. Synthesizing high-resolution imagery from these representations often requires time-consuming optical ray marching. In this work, we introduce Neural Sparse Voxel Fields (NSVF), a new neural scene representation for fast and high-quality free-viewpoint rendering. NSVF defines a set of voxel-bounded implicit fields organized in a sparse voxel octree to model local properties in each cell. We progressively learn the underlying voxel structures with a diffentiable ray-marching operation from only a set of posed RGB images. With the sparse voxel octree structure, rendering novel views can be accelerated by skipping the voxels containing no relevant scene content. Our method is over 10 times faster than the state-of-the-art (namely, NeRF) at inference time while achieving higher quality results. Furthermore, by utilizing an explicit sparse voxel representation, our method can easily be applied to scene editing and scene composition. We also demonstrate several challenging tasks, including multi-scene learning, free-viewpoint rendering of a moving human, and large-scale scene rendering.
Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
- Andreas Geiger
- Jonathan T. Barron
- Matthias Nießner
- Gerard Pons-Moll
- Michael Niemeyer
- Ben Mildenhall
- Pratul P. Srinivasan
- Andrea Tagliasacchi
In Computer Graphics and Computer-Aided Design
- Andreas Geiger
- Jonathan T. Barron
- Ben Mildenhall
- Pratul P. Srinivasan
- Matthias Nießner
- Peter Hedman
- Anpei Chen
- Matthew Tancik
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Jitendra Malik
- Dacheng Tao
- Ming-Hsuan Yang
- Andrea Vedaldi
- Andreas Geiger
- Trevor Darrell
- Xiaogang Wang
In 3D Vision & Pose
- Andreas Geiger
- Matthias Nießner
- Jonathan T. Barron
- Jitendra Malik
- Michael Niemeyer
- Justin Johnson
- Angela Dai
- Songyou Peng
Citations in the corpus
- Multiview Compressive Coding for 3D Reconstruction (2023, background): “Re-construction by analysis [11] or synthesis via volume rendering [24] of implicit [37, 82] and explicit [33, 56] representations have shown to produce strong results.”
- Mip-Splatting: Alias-Free 3D Gaussian Splatting (2023, background): “Further advancements have been made to improve the training and rendering of NeRF with advanced scene representations [4, 6, 11, 18, 19, 24, 32, 46, 51].”
- Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields (2021, neutral): “We report the NSVF [19] and NeRF [24] results on the Blender dataset from the tables in the mip-NeRF paper.”
- BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering (2021, background): “Neural volumetric representations [21,24,16,22,15,37] have demonstrated remarkable capability in representing 3D objects and scenes from images.”
- PlenOctrees for Real-time Rendering of Neural Radiance Fields (2021, methodology): “The Tanks and Temples sub-set is from NSVF [23] and contains 5 scenes of real objects captured by an inward facing camera circling the scene.”
- GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting (2024, methodology): “While NeRF [41] initially represents a radiance field as a coordinate-based MLP, this area has expanded with various models, including voxel-based [34, 59, 72], factorization-based [7, 9–11], hash grid-based [42], and point-(or primitive-) based [24, 30, 35, 67] representations among others [4–6 ,…”
- Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video (2020, neutral): “Neural Sparse Voxel Fields [39] are a promising direction to speed up NeRF-like methods.”
- Unsupervised Learning of 3D Object Categories from Videos in the Wild (2021, methodology): “Next, Voxel+MLP is inspired by Neural Sparse Voxel fields [31] and marries NeRF [36] with voxel grids.”
- 3D Neural Field Generation Using Triplane Diffusion (2022, neutral): “For example, concurrently developed single-latent approaches [2,14] generate a global latent for conditioning the neural field, relying on a 3D decoder to transform the scene representation from 1D to 3D without directly performing 3D diffusion.”
- 2D Gaussian Splatting for Geometrically Accurate Radiance Fields (2024, neutral): “Nevertheless, it falls short in capturing intricate geometry since the volumetric 3D Gaussian, which models the complete angular radiance, conflicts with the thin nature of surfaces.”
- Structured 3D Latents for Scalable and Versatile 3D Generation (2024, neutral): “More Implementation Details”
- TensoRF: Tensorial Radiance Fields (2022, methodology): “Recent methods [47,18,12] have leveraged a voxel grid of features in radiance field modeling, achieving fast rendering.”
- Block-NeRF: Scalable Large Scene Neural View Synthesis (2022, methodology): “NeRF caching techniques [20,25,73] or sparse voxel grids [34] could enable real-time Block-NeRF rendering.”
- iNeRF: Inverting Neural Radiance Fields for Pose Estimation (2020, methodology): “NSVF [13] improves NeRF by incorporating a sparse voxel octree structure into the scene representation, which accelerates rendering by allowing voxels without scene content to be omitted during rendering.”
- NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction (2022, background): “Neural implicit representations have made remarkable achievements in novel view synthesis [53, 33, 22, 35, 30, 52] and 3D/4D reconstruction [47, 48, 66, 37, 23, 20, 32, 39, 59, 65].”
- Efficient Geometry-aware 3D Generative Adversarial Networks (2021, background): “Block-based sparse volume representations overcome some of these issues, but are applicable to mostly empty scenes [19,35] and difficult to generalize across scenes.”
- Compact 3D Gaussian Representation for Radiance Field (2023, methodology): “These approaches using explicit voxel grid structures [4, 9, 10, 25, 26, 39] have demonstrated a significant improvement in training speed compared to traditional MLP-based NeRF methods.”
- Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering (2021, methodology): “We follow the standard metric in [12, 17, 6] by using PSNR, SSIM and LPIPS to measure the rendering quality.”
- Advances in Neural Rendering (2021, background): “Neural Sparse Voxel Fields [LGL ∗ 20] builds and dynamically updates an oc-tree structure while the MLP is optimized, allowing for aggressive empty space skipping and early ray termination (when the transmittance along the ray approaches zero).”