Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction

By Cheng Sun, Min Sun, Hwann-Tzong Chen. Computer Vision and Pattern Recognition, 2021

1,369 citations recorded by the source corpus.

We present a super-fast convergence approach to reconstructing the per-scene radiance field from a set of images that capture the scene with known poses. This task, which is often applied to novel view synthesis, is recently revolution-ized by Neural Radiance Field (NeRF) for its state-of-the-art quality and fiexibility. However, NeRF and its variants require a lengthy training time ranging from hours to days for a single scene. In contrast, our approach achieves NeRF-comparable quality and converges rapidly from scratch in less than 15 minutes with a single GPU. We adopt a representation consisting of a density voxel grid for scene geometry and a feature voxel grid with a shallow network for complex view-dependent appearance. Modeling with explicit and discretized volume representations is not new, but we propose two simple yet non-trivial techniques that contribute to fast convergence speed and high-quality output. First, we introduce the post-activation interpolation on voxel density, which is capable of producing sharp surfaces in lower grid resolution. Second, direct voxel density optimization is prone to suboptimal geometry solutions, so we robustify the optimization process by imposing several priors. Finally, evaluation on five inward-facing benchmarks shows that our method matches, if not surpasses, NeRF's quality, yet it only takes about 15 minutes to train from scratch for a new scene. Code: https://github.com/sunset1995/DirectVoxGO.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4f7eb65f8d3c1eeb97e30f7ac68977ff16e1e942

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (2023, methodology): “Hybrid methods [4, 7, 24, 27, 41, 46] incorporate a neural component within the explicit grid.”
TensoRF: Tensorial Radiance Fields (2022, methodology): “DVGO [37] and Plenoxels [46] also optimize voxel grids of (neural or SH) features for fast radiance field reconstruction.”
Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (2023, neutral): “Another group trades space for speed by incorporating explicit and localized neural representations [8, 15, 17, 27, 36, 60, 67, 75, 82, 83, 96, 101].”
Nerfstudio: A Modular Framework for Neural Radiance Field Development (2023, background): “…for high-quality 3D mesh exports [Oechsle et al. 2021; Wang et al. 2021a; Yariv et al. 2021], speed improvements for real-time rendering and training [Fridovich-Keil et al. 2022; Müller et al. 2022; Sun et al. 2022], 3D object generation [Poole et al. 2023], and more [Xie et al. 2022].”
Block-NeRF: Scalable Large Scene Neural View Synthesis (2022, positive): “Recent work has demonstrated techniques to speed up NeRF training by multiple orders of magnitude [41, 59, 72].”
SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering (2023, methodology): “Retrieving such an editable mesh for realistic rendering is done within minutes with our method, compared to hours with the state-of-the-art method on SDFs, while providing a better rendering quality.”
Co-SLAM: Joint Coordinate and Sparse Parametric Encodings for Neural Real-Time SLAM (2023, background): “Given that coordinate encoding-based methods require lengthy training, many follow-up works [7, 10, 27] propose parametric encodings that increase parameter size but speed up the training.”
Advances in Neural Rendering (2021, methodology): “Recently, there has also been a ﬂood of new approaches [PC21, FXW ∗ 21,YFKT ∗ 21,WLB ∗ 21,SSC21,KIT ∗ 21] that employ classical data structures, such as grids, sparse grids, trees, and hashes, for acceleration of rendering speed as well as faster training times.”
PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics (2023, background): “Building upon the achievements of NeRF, further studies have focused on enhancing reconstruction quality and improving training speeds [1, 7, 20, 26, 40, 46].”
Text-to-3D using Gaussian Splatting (2023, background): “, 2021), improving training (Yu et al., 2021a; Chen et al., 2022a; Sun et al., 2022; Müller et al., 2022) and rendering (Reiser et al.”
MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures (2022, background): “This embedding can either be a dense voxel grid [23, 43], a sparse voxel grid [41], a low-rank decomposition of a voxel grid [9], a point-based representation [50], or a multi-resolution hash map [34].”
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, neutral): “…becomes a popular 3D representation for photo-realistic novel-view synthesis in recent years [35], many variants are proposed to improve quality [2–4, 24, 45, 47–49, 57], increase speed [8, 9, 11, 14, 20, 32, 36, 37, 40, 43, 46, 58, 60], extend to dynamic scenes [7, 15, 18, 25, 27, 50], and so on.”
Gaussian Splatting SLAM (2023, neutral): “Recent developments have introduced explicit volume structures such as multi-resolution voxel grids [6, 14, 34] or hash functions [19] to improve performance.”
LRM: Large Reconstruction Model for Single Image to 3D (2023, neutral): “Different NeRF variants (Chan et al., 2022; Chen et al., 2022a; M¨uller et al., 2022; Sun et al., 2022) typically differ from each other in terms of the choice of the spatial encoding and the size of the MLP.”
Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation (2022, methodology): “We represent a 3D asset θ as a voxel radiance field [6, 54, 62], which is much faster to access and update compared to a vanilla NeRF parameterized by a neural network [34].”
GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting (2024, methodology): “…NeRF [41] initially represents a radiance field as a coordinate-based MLP, this area has expanded with various models, including voxel-based [34, 59, 72], factorization-based [7, 9–11], hash grid-based [42], and point-(or primitive-) based [24, 30, 35, 67] representations among others [4–6 , 74…”