Baking Neural Radiance Fields for Real-Time View Synthesis

By Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, Paul E. Debevec. IEEE International Conference on Computer Vision, 2021

646 citations recorded by the source corpus.

Neural volumetric representations such as Neural Radiance Fields (NeRF) have emerged as a compelling technique for learning to represent 3D scenes from images with the goal of rendering photorealistic images of the scene from unobserved viewpoints. However, NeRF’s computational requirements are prohibitive for real-time applications: rendering views from a trained NeRF requires querying a multilayer perceptron (MLP) hundreds of times per ray. We present a method to train a NeRF, then precompute and store (i.e. "bake") it as a novel representation called a Sparse Neural Radiance Grid (SNeRG) that enables real-time rendering on commodity hardware. To achieve this, we introduce 1) a reformulation of NeRF’s architecture, and 2) a sparse voxel grid representation with learned feature vectors. The resulting scene representation retains NeRF’s ability to render fine geometric details and view-dependent appearance, is compact (averaging less than 90 MB per scene), and can be rendered in real-time (higher than 30 frames per second on a laptop GPU). Actual screen captures are shown in our video.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8bd70d3dcfa295d9710922c34c1a9eeb0be48b94

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly- Throughs (2021, methodology): “Plenoctree [45], SNeRG [13], and FastNeRF [12] speed up the process by storing precomputed non-view dependent model outputs into a separate data structure such as a sparse voxel octree.”
SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis (2023, neutral): “proaches [47, 38, 22, 43, 15, 32, 16, 28] attempted to speed up the standard NeRF.”
Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies (2021, background): “It is could be solved with recent caching-based techniques [66, 17].”
Advances in Neural Rendering (2021, background): “SNeRG [HSM ∗ 21] stores volume density and a small spatially-varying feature vector in a sparse 3D texture atlas, using fast shader for compositing these values along a ray and running a tiny MLP decoder to produce view-dependent color for each ray.”
RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs (2021, methodology): “Employing the sparsity prior from Hedman et al. [17] leads to better quantitative results, but novel views still contain ﬂoating artifacts and the optimized geometry has holes.”
PlenOctrees for Real-time Rendering of Neural Radiance Fields (2021, background): “Also concurrently, Lombardi et al. [25] propose to model data using geometric primitives, and [13, 9, 38] also distill NeRFs to enable real-time rendering.”
KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs (2021, background): “Other works show that real-time rendering can be achieved by converting the neural representation into a discrete one after training [14, 16, 74].”
Nerfstudio: A Modular Framework for Neural Radiance Field Development (2023, background): “Additionally, Plenoxels [Fridovich-Keil et al. 2022], KiloNeRF [Reiser et al. 2021], SNerg [Hedman et al. 2021], FastNerf [Garbin et al. 2021], DONeRF [Neff et al. 2021], TensoRF [Chen et al. 2022] and InstantNGP [Müller et al. 2022] propose different approaches to address the problem of…”
NeRF-Editing: Geometry Editing of Neural Radiance Fields (2022, background): “However, NeRF still has shortcomings and plenty of work has extended the original NeRF, including better synthesis effects [36, 71, 83], applicable to dynamic scenes [14, 33, 44–48, 67, 72, 82], faster rendering speed [18, 22, 49, 77], generalization to different scenes [8, 69], relighting [4, 60, 8”
Fast Dynamic Radiance Fields with Time-Aware Neural Voxels (2022, background): “Representing scenes with explicit data structures shows great success in dramatically accelerating NeRF training and rendering [Hedman et al. 2021; Müller et al. 2022; Sun et al. 2022; Yu et al. 2022].”
Point-NeRF: Point-based Neural Radiance Fields (2022, methodology): “Voxel grids with per-voxel neural features [8, 15, 28] are also a local neural radiance representation.”
TensoRF: Tensorial Radiance Fields (2022, methodology): “Recent methods [47,18,12] have leveraged a voxel grid of features in radiance field modeling, achieving fast rendering.”
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, methodology): “Although NeRF has become widely used in both 3D reconstruction (Barron et al., 2022; Li et al., 2023d; Chen et al., 2022; Hedman et al., 2021) and generation (Poole et al., 2022; Lin et al., 2023; Chan et al., 2022), optimizing NeRF can be time-consuming.”
HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video (2022, background): “Neural radiance fields: NeRF [41] and its extensions [3, 23, 44, 58, 62, 75, 77] enable high quality rendering of novel views of static scenes.”
GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting (2023, neutral): “However, optimizing NeRF can be time-consuming, despite its wide usage in 3D reconstruction [3, 6, 13, 21, 44] and generation [22, 35] tasks.”
Text-to-3D using Gaussian Splatting (2023, neutral): “After works have emerged to improve NeRF in reconstruction quality [7, 8, 76], handling large-scale [13, 40, 68, 81] and dynamic scenes [3, 47, 51, 60, 73], improving training [10, 44, 67, 79], rendering [25, 56, 80] speed and facilitating down-stream tasks [17, 46, 78, 83, 84].”
Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (2023, methodology): “Pre-computed methods [12, 35] utilize spatial acceleration structures such as spherical harmonics coefficients [52] and feature vectors [13], cached or distilled from implicit neural representation, as opposed to directly employing the neural representations themselves.”