Point-NeRF: Point-based Neural Radiance Fields

By Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, Ulrich Neumann. Computer Vision and Pattern Recognition, 2022

755 citations recorded by the source corpus.

Volumetric neural rendering methods like NeRF [34] generate high-quality view synthesis results but are optimized per-scene leading to prohibitive reconstruction time. On the other hand, deep multi-view stereo methods can quickly reconstruct scene geometry via direct network inference. Point-NeRF combines the advantages of these two approaches by using neural 3D point clouds, with associated neural features, to model a radiance field. Point-NeRF can be rendered efficiently by aggregating neural point features near scene surfaces, in a ray marching-based rendering pipeline. Moreover, Point-NeRF can be initialized via direct inference of a pre-trained deep network to produce a neural point cloud; this point cloud can be finetuned to surpass the visual quality of NeRF with 30× faster training time. Point-NeRF can be combined with other 3D re-construction methods and handles the errors and outliers in such methods via a novel pruning and growing mechanism.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/055e87ce418a83d6fd555b73aea0d838385dfa85

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields (2023, background): “Point NeRF [45] uses neural 3D points for representing and rendering a continuous radiance volume.”
TensoRF: Tensorial Radiance Fields (2022, methodology): “Other methods design generalizable network modules trained across scenes to achieve image-dependent radiance field rendering [48,41,8] and fast reconstruction [7,43].”
PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics (2023, methodology): “Building upon the achievements of NeRF, further studies have focused on enhancing reconstruction quality and improving training speeds [1, 7, 20, 26, 40, 46].”
Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (2023, neutral): “Another group trades space for speed by incorporating explicit and localized neural representations [8, 15, 17, 27, 36, 60, 67, 75, 82, 83, 96, 101].”
PGSR: Planar-Based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction (2024, positive): “To address this challenge, various following studies aim to reduce dependency on MLP layers by decomposing scene information into separable structures, such as points [52] and voxels [18], [53], [54].”
NeRF: Representing scenes as neural radiance ﬁelds for view synthesis (neutral): “Point-nerf [ 79 ] introduced the notion of point-based NeRF representations as easier to train.”
MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures (2022, background): “This embedding can either be a dense voxel grid [23,43], a sparse voxel grid [41], a low-rank decomposition of a voxel grid [9], a point-based representation [50], or a multi-resolution hash map [34].”
DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization (2024, background): “This range is set of [5 , 17] for LLFF and Blender, and a larger [17 , 51] for DTU since the objects are smaller but occupy a large proportion of the image.”
Mip-Splatting: Alias-Free 3D Gaussian Splatting (2023, background): “Further advancements have been made to improve the training and rendering of NeRF with advanced scene representations [4, 6, 11, 18, 19, 24, 32, 46, 51].”
Compact 3D Gaussian Representation for Radiance Field (2023, methodology): “Point-NeRF [45] employed points to represent a radiance field with volumetric rendering.”
DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision (2023, neutral): “Additionally, it in-a spires waves of innovative developments such as NeRF variants [3, 4, 8, 27, 32, 37] and 3D Gaussian Splatting (3DGS) [15], significantly enhancing experiences in virtual reality, augmented reality, and advanced simulations.”
GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting (2024, methodology): “…a radiance field as a coordinate-based MLP, this area has expanded with various models, including voxel-based [34, 59, 72], factorization-based [7, 9–11], hash grid-based [42], and point-(or primitive-) based [24, 30, 35, 67] representations among others [4–6 , 74 Feed-forward Reconstruction.”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, positive): “Depth/Point Cloud Supervision DS-NeRF (2021)[42] (2021), NerfingMVS (2021) [43] (2021), Urban Radiance Field (2021) [44], PointNeRF (2022) [45]”
Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering (2023, methodology): “Consistent with existing methods [22, 53], we use the sparse point cloud from COLMAP [39] as our initial input.”
Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians (2024, background): “Various spatial structures have been explored in previous NeRF-based representations, including dense voxel grids [20], [22], sparse voxel grids [17], [21], point clouds [40], multiple compact low-rank tensor components [23], [41], [42], and multi-resolution hash tables [24].”
DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes (2023, positive): “21.94 0.781 Point-NeRF [20] 21.54 0.793 NSG [11] 22.89 0.836 Mip-NeRF360 [2] 23.27 0.836 PNF [6] 23.06 0.839 SUDS [14] 23.30 0.844 DNMP [8] 23.41 0.846 3DGS [4] 22.93 0.847 Ours-S 25.18 0.862 Ours-L 25.62 0.868 In contrast, although our method is not specifically designed for monocular scenarios,…”
SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes (2023, neutral): “4.3.”
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (2023, background): “Each Gaussian is characterized by a covariance matrix Σ and a center point X , which is referred to as the mean value of the Gaussian: For differentiable optimization, the covariance matrix Σ can be decomposed into a scaling matrix S and a rotation matrix R : the computation of gradient flow during…”