Neural Point-Based Graphics

By Kara-Ali Aliev, Dmitry Ulyanov, Victor S. Lempitsky. European Conference on Computer Vision, 2019

496 citations recorded by the source corpus.

We present a new point-based approach for modeling the appearance of real scenes. The approach uses a raw point cloud as the geometric representation of a scene, and augments each point with a learnable neural descriptor that encodes local geometry and appearance. A deep rendering network is learned in parallel with the descriptors, so that new views of the scene can be obtained by passing the rasterizations of a point cloud from new viewpoints through this network. The input rasterizations use the learned descriptors as point pseudo-colors. We show that the proposed approach can be used for modeling complex scenes and obtaining their photorealistic views, while avoiding explicit surface estimation and meshing. In particular, compelling results are obtained for scene scanned using hand-held commodity RGB-D sensors as well as standard RGB cameras even in the presence of objects that are challenging for standard mesh-based modeling.

View Paper: https://www.semanticscholar.org/paper/eb516fbfe0d49d2e903ff4605ec63cd7983d059a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

2D Gaussian Splatting for Geometrically Accurate Radiance Fields (2024, unknown): “Differentiable point-based rendering [Aliev et al. 2020; Insafutdinov and Dosovitskiy 2018; Rückert et al. 2022; Wiles et al. 2020; Yifan et al. 2019]has been explored extensively due to its efficiency and flexibility in representing intricate structures.”
Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline (2021, background): “…point-cloud analysis problems like segmentation (Ladick ` y et al., 2010; Tighe & Lazeb-nik, 2010; Riemenschneider et al., 2014; Qin et al., 2018; Dai & Nießner, 2018; Kalogerakis et al., 2017; Tatarchenko et al., 2018), reconstruction (Pittaluga et al., 2019) and rendering (Aliev et al., 2019).”
Compact 3D Gaussian Representation for Radiance Field (2023, methodology): “The points were represented by neural features and rendered with CNNs [1, 20, 27].”
Neural Sparse Voxel Fields (2020, methodology): “Other methods (Thies et al., 2019; Kim et al., 2018; Liu et al., 2019a, 2020; Meshry et al., 2019; Martin Brualla et al., 2018; Aliev et al., 2019) ﬁrst render a given geometry with explicit or neural textures into coarse RGB images or feature maps which are then translated into high-quality images.”
NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections (2020, methodology): “Several approaches employ image translation networks [12] to re-render content more realistically using as input traditional reconstruction results [21], learned latent textures [37], point clouds [2], voxels [31], or plane sweep volumes [8, 9].”
Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans (2020, unknown): “In these works, deep neural networks are employed to learn scene representations from 2D images with differentiable renderers, such as voxels [52, 33], point clouds [58, 1], textured meshes [56, 30, 29], multi-plane images [61, 15], and implicit functions [53, 31, 43, 40].”
Plenoxels: Radiance Fields without Neural Networks (2021, methodology): “Neural representations were first used to model occupancy [6, 23, 26] and signed distance to an object’s surface [33, 50], and perform novel view synthesis from 3D point clouds [2, 18, 42, 57].”
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds (2021, methodology): “By introducing differentiable 3D projection and using trainable layers that operate in the 3D and 2D feature space, several recent methods [4, 18, 42, 43, 56, 62] are able to model the geometry and appearance of 3D scenes from 2D images.”
Advances in Neural Rendering (2021, unknown): “One can use a deep neural networks to reconstruct the signal directly in the image space [ASK ∗ 20b].”
A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose (2021, background): “Recent neural scene representations learn low-dimensional nonlinear representations of meshes [31, 66], point clouds [2, 41, 71], sphere sets [27], and dense volumetric grids [32, 58].”
SynSin: End-to-End View Synthesis From a Single Image (2019, methodology): “While [1] also renders point clouds of features, they only back-propagate to the feature vectors, not the 3D positions.”
UniSim: A Neural Closed-Loop Sensor Simulator (2023, background): “Some works perform geometry reconstruction and then warp and aggregate pixel-features from the input images into new camera views, which are then processed by learning-based modules [1, 54, 58, 59].”
Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering (2023, background): “[1 , 24] augmented points with neural features and rendered using 2D CNNs.”
Neural Scene Graphs for Dynamic Scenes (2020, methodology): “Existing methods have proposed to learn features on discrete geometric primitives, such as points [2, 26], multi-planes [9, 16, 20, 33, 43], meshes [6, 37] or higher-order primitives like voxel grids [15, 29, 41], or implicitly represent features [14, 19, 23, 25, 40, 21] using functions F : R 3 → R…”
Point-NeRF: Point-based Neural Radiance Fields (2022, methodology): “We run NPBG [2] using the same point cloud generated by our MVSNet-based network.”
Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior (2023, methodology): “To avoid noisy colors and bleeding artifacts [2], we adopt a multi-scale deferred rendering scheme.”
Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering (2021, unknown): “In these works, deep neural networks are employed to learn novel view synthesis from 2D images in various approaches, such as voxels [25, 15], point clouds [1, 20], textured meshes [27, 13, 11], multi-plane images [27, 31, 6], and implicit functions [26, 14, 18].”
GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting (2024, background): “Early efforts adopt CNNs to estimate neural points [2,66,71] or multi-plane images (MPIs) [33,39,76], achieving rendering via point splatting or alpha compositing.”