FastNeRF: High-Fidelity Neural Rendering at 200FPS

By Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, Julien P. C. Valentin. IEEE International Conference on Computer Vision, 2021

787 citations recorded by the source corpus.

Recent work on Neural Radiance Fields (NeRF) showed how neural networks can be used to encode complex 3D environments that can be rendered photorealistically from novel viewpoints. Rendering these images is very computationally demanding and recent improvements are still a long way from enabling interactive rates, even on high-end hardware. Motivated by scenarios on mobile and mixed reality devices, we propose FastNeRF, the first NeRF-based system capable of rendering high fidelity photorealistic images at 200Hz on a high-end consumer GPU. The core of our method is a graphics-inspired factorization that allows for (i) compactly caching a deep radiance map at each position in space, (ii) efficiently querying that map using ray directions to estimate the pixel values in the rendered image. Extensive experiments show that the proposed method is 3000 times faster than the original NeRF algorithm and at least an order of magnitude faster than existing work on accelerating NeRF, while maintaining visual quality and extensibility.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c0ccfbaf073c91e68ccbc57af2114c72c0d0427d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly- Throughs (2021, methodology): “Plenoctree [45], SNeRG [13], and FastNeRF [12] speed up the process by storing precomputed non-view dependent model outputs into a separate data structure such as a sparse voxel octree.”
NeRF-Editing: Geometry Editing of Neural Radiance Fields (2022, background): “In particular, the recently proposed neural radiance field (NeRF) [43] has inspired a large number of follow-up works aiming to achieve better visual effects [36], faster rendering speed [18, 77],”
MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures (2022, methodology): “FastNeRF [17] uses a dense voxel grid and represents view-dependence with a global spherical basis function.”
DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes (2023, neutral): “…[14] 23.30 0.844 DNMP [8] 23.41 0.846 3DGS [4] 22.93 0.847 Ours-S 25.18 0.862 Ours-L 25.62 0.868 In contrast, although our method is not specifically designed for monocular scenarios, it still shows good adaptability and robustness in representing monocular driving scenes with detail areas and…”
Plenoxels: Radiance Fields without Neural Networks (2021, background): “Multiple subsequent papers [9, 10, 21, 37, 38, 59] reduced this computational cost for rendering, but single GPU training still requires multiple hours, a bottleneck that limits the practical application of photorealistic volumetric reconstruction.”
Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (2023, methodology): “Pre-computed methods [12, 35] utilize spatial acceleration structures such as spherical harmonics coefficients [52] and feature vectors [13], cached or distilled from implicit neural representation, as opposed to directly employing the neural representations themselves.”
FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting (2023, background): “Tremendous works focus on improving its efficiency [7, 15, 17, 26, 31, 38, 43], quality [1, 3, 4, 9, 20, 42, 48, 52], generalizing to unseen scenes [8, 10, 24, 44, 53, 59], applying artistic effects [14, 22, 49, 60] and 3D genera-2 tion [5, 6, 18, 21, 25, 27, 28, 34, 41, 45].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, positive): “This resulted in rendering speed comparable to FastNeRF [52], and exceeding that of baseline NeRF by thousands fold.”
Nerfstudio: A Modular Framework for Neural Radiance Field Development (2023, background): “Additionally, Plenoxels [Fridovich-Keil et al. 2022], KiloNeRF [Reiser et al. 2021], SNerg [Hedman et al. 2021], FastNerf [Garbin et al. 2021], DONeRF [Neff et al. 2021], TensoRF [Chen et al. 2022] and InstantNGP [Müller et al. 2022] propose different approaches to address the problem of…”
Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations (2021, neutral): “N e R F [ 25 ] F a s t N e R F [ 9 ] N e R F -”
HeadNeRF: A Realtime NeRF-based Parametric Head Model (2021, background): “Besides, many works are proposed to further improve NeRF, including speeding up training [1,2] and inference [13,18,44,56,59], improving rendering quality [55], and reducing the number of required inputs [8, 60].”
Baking Neural Radiance Fields for Real-Time View Synthesis (2021, background): “[15] instead factorize the view-dependent computation as a dot product between a global spherical function and a per-sample feature vector.”
LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS (2023, methodology): “Follow-ups either explore ray re-parameterizations[4, 5], explicit spatial data structures [10, 21, 31, 39, 45, 55, 66], caching and distillation [22, 28, 48, 60], or ray-based representations [2, 54] for speeding up.”