KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs

By Christian Reiser, Songyou Peng, Yiyi Liao, Andreas Geiger. IEEE International Conference on Computer Vision, 2021

995 citations recorded by the source corpus.

NeRF synthesizes novel views of a scene with unprecedented quality by fitting a neural radiance field to RGB images. However, NeRF requires querying a deep Multi-Layer Perceptron (MLP) millions of times, leading to slow rendering times, even on modern GPUs. In this paper, we demonstrate that real-time rendering is possible by utilizing thousands of tiny MLPs instead of one single large MLP. In our setting, each individual MLP only needs to represent parts of the scene, thus smaller and faster-to-evaluate MLPs can be used. By combining this divide-and-conquer strategy with further optimizations, rendering is accelerated by three orders of magnitude compared to the original NeRF model without incurring high storage costs. Further, using teacher-student distillation for training, we show that this speed-up can be achieved without sacrificing visual quality.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c041aaed581616e122e790dd2769337216df3d8d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (2023, methodology): “[12, 29, 43] use pure explicit representations or the baking method to promote the rendering speed but mainly focus on the static scenes.”
3D Neural Field Generation Using Triplane Diffusion (2022, neutral): “Moreover, GANs can be difficult to train and suffer from issues like mode collapse [75] whereas diffusion models train stably and have been shown to better capture the full training distribution.”
MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures (2022, background): “For example, by reducing the network queries per pixel with learned sampling [36], by evaluating the network for larger ray segments [48], or by subdividing the scene into smaller networks [38, 39, 49].”
Advances in Neural Rendering (2021, background): “KiloNeRF [RPLG21] combines empty space skipping and early termination with a dense 3D grid of MLPs, each with a much smaller number of weights than a standard NeRF network.”
LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS (2023, methodology): “R2L [60] and Kilo-NeRF [48] distill Neural Radiance Fields (NeRF) into more efficient representations such as light fields or multiple tiny MLPs.”
Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (2023, methodology): “Pre-computed methods [12, 35] utilize spatial acceleration structures such as spherical harmonics coefficients [52] and feature vectors [13], cached or distilled from implicit neural representation, as opposed to directly employing the neural representations themselves.”
FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting (2023, background): “In particular, Reiser et al. [38] accelerate NeRF’s training by splitting a big MLP into thousands of tiny MLPs. MVSNeRF [8] constructs a 3D cost volume [19, 56] and renders high-quality images from novel viewpoints.”
MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes (2023, methodology): “2022] or by splitting the scene into voxels and representing each voxel with a small separate MLP [Reiser et al. 2021].”
BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis (2023, background): “The SDF MLP’s values in both of these types of regions are not supervised during training, so we must cull any surface crossings that would show up as spurious content in the reconstructed mesh.”
Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (2023, neutral): “Another group trades space for speed by incorporating explicit and localized neural representations [8, 15, 17, 27, 36, 60, 67, 75, 82, 83, 96, 101].”
NeRF: Representing scenes as neural radiance ﬁelds for view synthesis (neutral): “Similarly, KiloNeRF [ 54 ] breaks the scene into regions, each ﬁt with small MLPs.”
Mip-Splatting: Alias-Free 3D Gaussian Splatting (2023, methodology): “Subsequent methods [16, 40, 41, 52, 54] distill a pretrained NeRF into a sparse representation, enabling real-time rendering of NeRFs.”
HeadNeRF: A Realtime NeRF-based Parametric Head Model (2021, background): “Besides, many works are proposed to further improve NeRF, including speeding up training [1,2] and inference [13,18,44,56,59], improving rendering quality [55], and reducing the number of required inputs [8, 60].”
Efficient Geometry-aware 3D Generative Adversarial Networks (2021, background): “Local implicit representations [3, 5, 23, 56] and hybrid explicit–implicit representations [11,35,39,53] combine the beneﬁts of both types of representations by offering computationally and memory-efﬁcient architectures.”
TensoRF: Tensorial Radiance Fields (2022, background): “This representation has been quickly extended and applied in diverse graphics and vision applications, including generative models [6,26], appearance acquisition [2,4], surface reconstruction [40,27], fast rendering [33,47,12], appearance editing [42,19], dynamic capture [17,28].”