PlenOctrees for Real-time Rendering of Neural Radiance Fields

By Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, Angjoo Kanazawa. IEEE International Conference on Computer Vision, 2021

1,321 citations recorded by the source corpus.

We introduce a method to render Neural Radiance Fields (NeRFs) in real time using PlenOctrees, an octree-based 3D representation which supports view-dependent effects. Our method can render 800×800 images at more than 150 FPS, which is over 3000 times faster than conventional NeRFs. We do so without sacrificing quality while preserving the ability of NeRFs to perform free-viewpoint rendering of scenes with arbitrary geometry and view-dependent effects. Real-time performance is achieved by pre-tabulating the NeRF into a PlenOctree. In order to preserve view-dependent effects such as specularities, we factorize the appearance via closed-form spherical basis functions. Specifically, we show that it is possible to train NeRFs to predict a spherical harmonic representation of radiance, removing the viewing direction as an input to the neural network. Furthermore, we show that PlenOctrees can be directly optimized to further minimize the reconstruction loss, which leads to equal or better quality compared to competing methods. Moreover, this octree optimization step can be used to reduce the training time, as we no longer need to wait for the NeRF training to converge fully. Our real-time neural rendering approach may potentially enable new applications such as 6-DOF industrial and product visualizations, as well as next generation AR/VR systems. PlenOctrees are amenable to in-browser rendering as well; please visit the project page for the interactive online demo, as well as video and code: https://alexyu.net/plenoctrees.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/5744fcc21b40327f7ad710de7d947d4584c53012

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Plenoxels: Radiance Fields without Neural Networks (2021, methodology): “Spherical harmonics are a standard basis for functions over the sphere, and have been used previously to represent view dependence [4, 36, 47, 58, 59].”
Instant neural graphics primitives with a multiresolution hash encoding (2022, methodology): “NSVF [Liu et al. 2020] and several concurrent works [Sun et al. 2021; Yu et al. 2021a] adopt a multistage, coarse to fine strategy in which regions of the feature grid are progressively refined and culled away as necessary.”
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, neutral): “…becomes a popular 3D representation for photo-realistic novel-view synthesis in recent years [35], many variants are proposed to improve quality [2–4, 24, 45, 47–49, 57], increase speed [8, 9, 11, 14, 20, 32, 36, 37, 40, 43, 46, 58, 60], extend to dynamic scenes [7, 15, 18, 25, 27, 50], and so on.”
HexPlane: A Fast Representation for Dynamic Scenes (2023, methodology): “Some methods improve inference speeds of trained NeRFs by optimizing the computation [19,21,60,86].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, methodology): “The method was evaluated on the NeRF Synthetic [1], BlendedMVS [94] and Tanks and Temple datasets [30], outperforming methods such as PlenOctrees [51], FastNeRF [52] and KiloNeRF [53] in terms of quality, at comparable a rendering speed.”
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (2023, methodology): “[12, 29, 43] use pure explicit representations or the baking method to promote the rendering speed but mainly focus on the static scenes.”
Decomposing NeRF for Editing via Feature Field Distillation (2022, background): “Regular structures such as voxelgrids or octrees [13, 48, 14, 88, 94, 89, 43, 81, 107, 59, 60] or unsupervised decomposition [73, 85, 109, 83] enable editability via manipulation of localized parameters.”
SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis (2023, neutral): “proaches [47, 38, 22, 43, 15, 32, 16, 28] attempted to speed up the standard NeRF.”
UniSim: A Neural Closed-Loop Sensor Simulator (2023, methodology): “We believe the run-time can be further optimized with recent advancements [11,97] and code optimization ( e.g ., customized CUDA kernels).”
Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction (2021, methodology): “Several follow-up methods [15,18,29,30,42,43,66] have shown significant speedup of FPS in testing phase, some of which even achieve real-time rendering.”
Depth-supervised NeRF: Fewer Views and Faster Training for Free (2021, background): “Many works [20,32] have found that the limiting factor is not learning the radiance itself, but rather oversampling the empty space during training.”
SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes (2023, background): “Most recently, Gaussian splatting [13] has shown remarkable performance in terms of rendering quality, resolution, and speed.”
DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization (2024, background): “The following improvements mainly pursue either higher quality [2, 3] or efficiency [6, 12, 15, 21, 26, 37, 53], but hard to achieve both.”
Advances in Neural Rendering (2021, methodology): “…along with weights for combining a set of learned spherical basis functions that produce view-varying colors at each point in 3D. PlenOctrees [YLT ∗ 21] queries the MLP to produce a sparse voxel octree of volume density and spherical harmonic coefﬁcients and further ﬁnetunes this octree…”
DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision (2023, neutral): “Additionally, it in-a spires waves of innovative developments such as NeRF variants [3, 4, 8, 27, 32, 37] and 3D Gaussian Splatting (3DGS) [15], significantly enhancing experiences in virtual reality, augmented reality, and advanced simulations.”
Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly- Throughs (2021, methodology): “Current renderers (such as Plenoctree [35]) cache precomputed model outputs into a fixed octree, limiting the resolution of rendered images (a).”