Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields

By Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, Peter Hedman. IEEE International Conference on Computer Vision, 2023

867 citations recorded by the source corpus.

Neural Radiance Field training can be accelerated through the use of grid-based representations in NeRF’s learned mapping from spatial coordinates to colors and volumetric density. However, these grid-based approaches lack an explicit understanding of scale and therefore often introduce aliasing, usually in the form of jaggies or missing scene content. Anti-aliasing has previously been addressed by mip-NeRF 360, which reasons about sub-volumes along a cone rather than points along a ray, but this approach is not natively compatible with current grid-based techniques. We show how ideas from rendering and signal processing can be used to construct a technique that combines mip-NeRF 360 and grid-based models such as Instant NGP to yield error rates that are 8% – 77% lower than either prior technique, and that trains 24× faster than mip-NeRF 360.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9eafa581e0268d471b9c61c87db79710dab9274e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting (2023, background): “Tremendous works focus on improving its efficiency [7, 15, 17, 26, 31, 38, 43], quality [1, 3, 4, 9, 20, 42, 48, 52], generalizing to unseen scenes [8, 10, 24, 44, 53, 59], applying artistic effects [14, 22, 49, 60] and 3D genera-2 tion [5, 6, 18, 21, 25, 27, 28, 34, 41, 45].”
DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision (2023, neutral): “Additionally, it in-a spires waves of innovative developments such as NeRF variants [3, 4, 8, 27, 32, 37] and 3D Gaussian Splatting (3DGS) [15], significantly enhancing experiences in virtual reality, augmented reality, and advanced simulations.”
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, neutral): “…becomes a popular 3D representation for photo-realistic novel-view synthesis in recent years [35], many variants are proposed to improve quality [2–4, 24, 45, 47–49, 57], increase speed [8, 9, 11, 14, 20, 32, 36, 37, 40, 43, 46, 58, 60], extend to dynamic scenes [7, 15, 18, 25, 27, 50], and so on.”
Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (2023, neutral): “Another group trades space for speed by incorporating explicit and localized neural representations [8, 15, 17, 27, 36, 60, 67, 75, 82, 83, 96, 101].”
PGSR: Planar-Based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction (2024, positive): “Recently, neural radiance ﬁelds [11], [12], [13], [14] have been widely used to tackle this task, achieving high-ﬁdelity novel view synthesis [15], [16], [17] and 3D geometry reconstruction [18], [19].”
Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians (2024, methodology): “The concept of LOD finds extensive applications in geometry reconstruction [46]–[48] and neural rendering [16], [27], [30], [49], [50].”
Compact 3D Gaussian Representation for Radiance Field (2023, methodology): “These approaches using explicit voxel grid structures [4, 9, 10, 25, 26, 39] have demonstrated a significant improvement in training speed compared to traditional MLP-based NeRF methods.”
GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting (2024, methodology): “…a radiance field as a coordinate-based MLP, this area has expanded with various models, including voxel-based [34, 59, 72], factorization-based [7, 9–11], hash grid-based [42], and point-(or primitive-) based [24, 30, 35, 67] representations among others [4–6 , 74 Feed-forward Reconstruction.”
Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering (2023, background): “In contrast, volumetric representations and neural radiance fields utilize learning-based parametric models [3, 5, 30], hence can produce continuous rendering results with more details preserved.”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, methodology): “In Table 1, we compare to the state-of-the-art for dense-view NeRF reconstruction with no learned priors (Zip-NeRF [71]) and methods that heavily leverage generative priors such as ZeroNVS [41] and ReconFusion [7].”
Mip-Splatting: Alias-Free 3D Gaussian Splatting (2023, methodology): “We use the official implementation for Mip-NeRF 360 [1] and 3DGS [18] and use a community reim-plementation for Zip-NeRF [3] § as the code is not available.”
Text-to-3D using Gaussian Splatting (2023, neutral): “After works have emerged to improve NeRF in reconstruction quality [7, 8, 76], handling large-scale [13, 40, 68, 81] and dynamic scenes [3, 47, 51, 60, 73], improving training [10, 44, 67, 79], rendering [25, 56, 80] speed and facilitating down-stream tasks [17, 46, 78, 83, 84].”
Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (2023, methodology): “Hybrid methods [4, 7, 24, 27, 41, 46] incorporate a neural component within the explicit grid.”
SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes (2023, methodology): “Recent advancements in this area are primarily driven by neural radiance fields (NeRF) [19, 30, 37, 66], which utilizes an implicit func-1 tion to simultaneously learn scene geometry [26, 29] and textures [12, 57] from multi-view images.”
Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields (2023, methodology): “Zip-NeRF [2] utilized an anti-aliased grid-based technique to boost the radiance field performance.”