Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction

By Chen-Hsuan Lin, Chen Kong, Simon Lucey. 2017

462 citations recorded by the source corpus.

Conventional methods of 3D object generative modeling learn volumetric predictions using deep networks with 3D convolutional operations, which are direct analogies to classical 2D ones. However, these methods are computationally wasteful in attempt to predict 3D shapes, where information is rich only on the surfaces. In this paper, we propose a novel 3D generative modeling framework to efficiently generate object shapes in the form of dense point clouds. We use 2D convolutional operations to predict the 3D structure from multiple viewpoints and jointly apply geometric reasoning with 2D projection optimization. We introduce the pseudo-renderer, a differentiable module to approximate the true rendering operation, to synthesize novel depth maps for optimization. Experimental results for single-image 3D object reconstruction tasks show that we outperforms state-of-the-art methods in terms of shape similarity and prediction density.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Implicit Fields for Generative Shape Modeling (2018, background): “There have been a variety of 3D shape representations for deep learning of shapes, such as voxel grids [9, 14, 28, 45, 46], octrees [18, 35, 40, 43, 44], multi-view images [29, 39], point clouds [1, 12, 13, 32, 33, 47, 48], geometry images [37, 38], deformable mesh/patches [16, 38, 42, 47], and part”
What Do Single-View 3D Reconstruction Networks Learn? (2019, methodology): “Multiple methods [27, 30, 43, 45] produced multiview depth maps that are fused together into an output point cloud.”
Mesh R-CNN (2019, background): “Other approaches predict novel 3D shapes as sets of 3D points [8, 34], patches [15, 70], or geometric primitives [9, 64, 67]; others use deep networks to model signed distance functions [42].”
Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering (2023, background): “To alleviate the discontinuity issue, differentiable point-based rendering has been extensively studied to model objects geometry [16, 20, 27, 48, 55].”
Learning to Infer Implicit Surfaces without 3D Supervision (2019, background): “Progress has also been made in reconstructing point clouds from single images using point feature learning [14, 15, 16, 17, 3].”
PGSR: Planar-Based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction (2024, unknown): “Numerous pioneering efforts have leveraged pure deep neural networks to predict surface models directly from single or multiple image conditions using point clouds [43], [44], voxels [45], [46], and triangular meshes [7], [47] or implicit ﬁelds [48], [49] in end-to-end manner.”
PaMIR: Parametric Model-Conditioned Implicit Representation for Image-Based Human Reconstruction (2020, background): “Explicit representations like point clouds [37], [38], voxel grids [39], [40], triangular meshes [41], [42] have recently been explored to replicate the success of deep learning techniques on 2D images.”
Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers (2018, methodology): “The final fusion of views has been addressed by merging them into a point cloud and pruning outliers using the predicted silhouettes [25], registering views and solving an optimization problem [18], as well as learning a differentiable depth map renderer to produce consistent projections [17].”
Deep Stereo Using Adaptive Thin Volume Representation With Uncertainty Awareness (2019, background): “Researchers have also tried to reconstruct point clouds [22, 13, 35, 52, 36, 2], however the high dimensionality of a point cloud often results in noisy outliers since a point cloud does not efficiently encode connectivity between points.”
Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstruction (2020, methodology): “Explicit shape models can be classiﬁed by the type of 3D shape representation that they use: voxel, point cloud or (parametric) mesh [1, 4–6, 10, 11, 13, 15, 20, 24, 25, 28, 29].”
What's Behind the Couch? Directed Ray Distance Functions (DRDF) for 3D Scene Reconstruction (2021, background): “Most work on invisible surfaces focuses on single objects with voxels [9, 18, 21], point-clouds [14, 32], CAD models [26] and meshes [19, 20].”
GRNet: Gridding Residual Network for Dense Point Cloud Completion (2020, background): “Some recent works [19,20,21,22,23] attempt to solve the unorderness while preserving fine-grained details by projecting the 3D point cloud to an image, which is then supervised by the corresponding ground truth masks.”
Articulation-Aware Canonical Surface Mapping (2020, background): “There are many existing approaches which tackle more general forms of 3D prediction, ranging from volumetric prediction [5, 10] to point cloud inference [8, 20].”
Advances in Neural Rendering (2021, methodology): “Essentially, this approach amounts to blending the discrete samples with some local deterministic blurring kernels [LKL18, ID18, RROG18], such as EWA splat-ting [ZPVBG01, ZPVBG02], which is a spatially-variant reconstruction kernel that is designed to minimize aliasing.”
P2P-NET: Bidirectional Point Displacement Net for Shape Transform (methodology): “Multiple point clouds from different views have also been constructed as intermediate shape representations for the purpose of generating 3D shapes from 2D images and/or sketches [Lin et al. 2018; Lun et al. 2017].”
Convolutional Occupancy Networks (2020, background): “Point Clouds: An alternative output representation for 3D reconstruction is 3D point clouds which have been used in [9,21,34,49].”
Geometric Capsule Autoencoders for 3D Point Clouds (2019, background): “The idea of using multiple views to come up with a coherent understanding of an object has been studied extensively [20, 17, 18, 13, 4, 5, 23, 24, 6, 9].”
MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion (2024, methodology): “Commonly used output representations include point clouds (Guo et al., 2020; Lin et al., 2018), meshes (Gkioxari et al., 2019; Wang et al., 2018), voxel (Sitzmann et al., 2019; Choy et al., 2016; Tulsiani et al., 2017), implicit representation (Wang et al., 2021a; Peng et al., 2020; Chen & Zhang,…”
DUSt3R: Geometric 3D Vision Made Easy (2023, unknown): “Similarly, view-wise modeling is a common theme in monocular 3D reconstruction works [57, 113, 127, 141] and in view synthesis works [151].”
PF-Net: Point Fractal Network for 3D Point Cloud Completion (2020, methodology): “Though L-GAN [1] is capable of conducting shape completion tasks to some extent, its architecture is not primarily built to do shape completion task, and thus the performance is not considered ideal.”