DONeRF: Towards Real‐Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks

By Thomas Neff, P. Stadlbauer, Mathias Parger, A. Kurz, J. Mueller, Chakravarty Reddy Alla Chaitanya, Anton Kaplanyan, M. Steinberger. Computer graphics forum (Print), 2021

342 citations recorded by the source corpus.

The recent research explosion around implicit neural representations, such as NeRF, shows that there is immense potential for implicitly storing high‐quality scene and lighting information in compact neural networks. However, one major limitation preventing the use of NeRF in real‐time rendering applications is the prohibitive computational cost of excessive network evaluations along each view ray, requiring dozens of petaFLOPS. In this work, we bring compact neural representations closer to practical rendering of synthetic content in real‐time applications, such as games and virtual reality. We show that the number of samples required for each view ray can be significantly reduced when samples are placed around surfaces in the scene without compromising image quality. To this end, we propose a depth oracle network that predicts ray sample locations for each view ray with a single network evaluation. We show that using a classification network around logarithmically discretized and spherically warped depth values is essential to encode surface locations rather than directly estimating depth. The combination of these techniques leads to DONeRF, our compact dual network design with a depth oracle network as its first step and a locally sampled shading network for ray accumulation. With DONeRF, we reduce the inference costs by up to 48× compared to NeRF when conditioning on available ground truth depth information. Compared to concurrent acceleration methods for raymarching‐based neural representations, DONeRF does not require additional memory for explicit caching or acceleration structures, and can render interactively (20 frames per second) on a single GPU.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/792f25e5a2f167df962e0d25b6bee5474a86605f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Neural RGB-D Surface Reconstruction (2021, background): “[49] restrict the volumetric rendering to near surface regions.”
Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields (2021, methodology): “Our online distillation approach is similar in spirit to the “sampling oracle networks” used in DONeRF, though that approach uses ground-truth depth for supervision [31].”
Efficient Geometry-aware 3D Generative Adversarial Networks (2021, neutral): “Emerging neural scene representations use differentiable 3D-aware representations [1, 3, 6, 8, 13, 17, 43, 44, 52, 65] that can be optimized using 2D multi-view images via neural rendering [15, 20, 24, 30, 34–37, 40, 45, 46, 50, 51, 54, 62, 63, 70– 72].”
Baking Neural Radiance Fields for Real-Time View Synthesis (2021, background): “While fully baked representations [15, 54] perform the least computation per 3D sample point, our representation, KiloNeRF [39] and DONeRF [32] compress the scene more effectively, as they represent view-dependence with MLPs instead of precomputed spherical functions.”
3D Neural Field Generation Using Triplane Diffusion (2022, neutral): “…[1,4,5,10,11,15,21,23,45, 47,48,57,65,73] or use posed images to jointly optimize geometry and appearance [6,7,18,25,29,33,36–39, 44, 49, 50, 54, 55 Triplanesareefficient, scalingwiththesurfacearearatherthan volume, and naturally integrate with expressive, fine-tuned 2D generator architectures.”
SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis (2023, positive): “proaches [47, 38, 22, 43, 15, 32, 16, 28] attempted to speed up the standard NeRF.”
MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures (2022, background): “For example, by reducing the network queries per pixel with learned sampling [36], by evaluating the network for larger ray segments [48], or by subdividing the scene into smaller networks [38, 39, 49].”
Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (2023, neutral): “One group of methods focuses on improving the sampling strategy to reduce the number of point queries [4, 68, 73] or applies light field-based formulation [3, 25, 53, 54, 79, 81, 89].”
Depth-supervised NeRF: Fewer Views and Faster Training for Free (2021, methodology): “Several prior works have explored ways to leverage depth information for view synthesis [24,27] and NeRF training [9, 10, 16, 18, 30].”
Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields (2023, methodology): “(5) After resampling, we evaluate our loss L prop as an element-wise function of w ˆ s and ˆ w , as they share a common coordinate space.”
BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis (2023, methodology): “While rendering can be accelerated with a sampling network that reduces the MLP queries per ray [Neff et al. 2021], recent methods have improved both training and render time by replacing the large MLP with a voxel grid [Karnewar et al.”
MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes (2023, background): “Alternatively, faster rendering can be achieved by extending the network to work with ray segments rather than points [Lindell et al. 2021; Wu et al. 2022a] or by training a separate sampling network [Barron et al. 2022; Kurz et al. 2022; Neff et al. 2021; Piala and Clark 2021].”
Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly- Throughs (2021, background): “DONeRF [25] accelerates rendering by significantly reducing the number of samples queried per ray.”
Fast Dynamic Radiance Fields with Time-Aware Neural Voxels (2022, methodology): “Some works propose to reduce inference times for acceleration by improving sampling strategies [Arandjelović and Zisserman 2021; Fang et al. 2021; Lindell et al. 2021; Neff et al. 2021; Piala and Clark 2021] or introducing efficient rendering techniques [Sitzmann et al. 2021].”