Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes

By Towaki Takikawa, Joey Litalien, K. Yin, Karsten Kreis, Charles T. Loop, Derek Nowrouzezahrai, Alec Jacobson, M. McGuire, Sanja Fidler. Computer Vision and Pattern Recognition, 2021

642 citations recorded by the source corpus.

Neural signed distance functions (SDFs) are emerging as an effective representation for 3D shapes. State-of-the-art methods typically encode the SDF with a large, fixed-size neural network to approximate complex shapes with implicit surfaces. Rendering with these large networks is, however, computationally expensive since it requires many forward passes through the network for every pixel, making these representations impractical for real-time graphics. We introduce an efficient neural representation that, for the first time, enables real-time rendering of high-fidelity neural SDFs, while achieving state-of-the-art geometry reconstruction quality. We represent implicit surfaces using an octree-based feature volume which adaptively fits shapes with multiple discrete levels of detail (LODs), and enables continuous LOD with SDF interpolation. We further develop an efficient algorithm to directly render our novel neural SDF representation in real-time by querying only the necessary LODs with sparse octree traversal. We show that our representation is 2–3 orders of magnitude more efficient in terms of rendering speed compared to previous works. Furthermore, it produces state-of-the-art reconstruction quality for complex shapes under both 3D geometric and 2D image-space metrics.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design

View Paper: https://www.semanticscholar.org/paper/6ca0a80111fed68b8a5b7eac5ecdc3d258f0fea6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

Citations in the corpus

MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction (2022, methodology): “More speciﬁcally, we investigate the following architectures: a single, large MLP [49,76,81,82], a dense SDF grid [31], a single feature grid [28,38,53,54] and multi-resolution feature grids [10, 22, 45, 69, 92].”
Plenoxels: Radiance Fields without Neural Networks (2021, methodology): “Neural representations were first used to model occupancy [6, 23, 26] and signed distance to an object’s surface [33, 50], and perform novel view synthesis from 3D point clouds [2, 18, 42, 57].”
BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering (2021, methodology): “[34] proposed to represent implicit surfaces using an octree-based feature volume which adaptively fits shapes with multiple discrete LODs.”
Vox-Fusion: Dense Tracking and Mapping with Voxel-based Neural Implicit Representation (2022, background): “E.g. NGLOD [32] leverages a hierarchical data structure by concatenating features from each level to achieve scene representation with different levels of details.”
Block-NeRF: Scalable Large Scene Neural View Synthesis (2022, methodology): “This approach has also been applied to compress detailed 3D shapes into neural signed distance functions [62] and to represent large scenes using occupancy networks [46].”
Efficient Geometry-aware 3D Generative Adversarial Networks (2021, neutral): “Emerging neural scene representations use differentiable 3D-aware representations [1, 3, 6, 8, 13, 17, 43, 44, 52, 65] that can be optimized using 2D multi-view images via neural rendering [15, 20, 24, 30, 34–37, 40, 45, 46, 50, 51, 54, 62, 63, 70– 72].”
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields (2021, background): “[42] proposes a multiscale representation based on sparse voxel octrees for continuous neural representations of implicit surfaces, but their approach requires that the scene geometry be known a priori, as opposed to our view synthesis setting where the only inputs are observed images.”
NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields (2022, methodology): “In particular, NGLOD [17] proposes to use tiny MLPs in a volumetric grid, leading to faster reconstructions, but not quite real-time.”
FastNeRF: High-Fidelity Neural Rendering at 200FPS (2021, methodology): “One approach is to use a voxel grid in combination with the implicit function learned by the MLP, as proposed in Neural Sparse Voxel Fields (NSVFs) [16] and Neural Geometric Level of Detail [38] (for SDFs only), where a dynamically constructed sparse octree is used to represent scene occupancy.”
Volume Rendering of Neural Implicit Surfaces (2021, background): “Recent studies have been focusing on model implicit functions with multi-layer perceptron (MLP) due to its expressive representation power and low memory foot-print, including scene (geometry & appearance) representation [9, 20, 19, 23, 25, 29, 36, 28, 35] and free-view rendering [33, 16, 30, 26, 17”
Advances in Neural Rendering (2021, background): “Takikawa et al. [TLY ∗ 21] improved the rendering performance by adapting the ray-tracing algorithm to the sparse octree data structure.”
Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (2023, neutral): “Another group trades space for speed by incorporating explicit and localized neural representations [8, 15, 17, 27, 36, 60, 67, 75, 82, 83, 96, 101].”
Co-SLAM: Joint Coordinate and Sparse Parametric Encodings for Neural Real-Time SLAM (2023, methodology): “Subsequent works focused on addressing the scalability issue by adopting more efficient data structures, such as surface elements (surfels) [35, 36], VoxelHashing [3, 9, 19] or Octrees [31,41].”
Instant neural graphics primitives with a multiresolution hash encoding (2022, methodology): “Most successful among these encodings are trainable, task-specific data structures [Liu et al. 2020; Takikawa et al. 2021] that take on a large portion of the learning task.”
3D Neural Field Generation Using Triplane Diffusion (2022, neutral): “They either solely learn geometry [1,4,5,10,11,15,21,23,45, 47,48,57,65,73] or use posed images to jointly optimize geometry and appearance [6,7,18,25,29,33,36–39, 44, 49, 50, 54, 55 Triplanesareefficient, scalingwiththesurfacearearatherthan volume, and naturally integrate with expressive,…”