A Point Set Generation Network for 3D Object Reconstruction from a Single Image

By Haoqiang Fan, Hao Su, Leonidas J. Guibas. Computer Vision and Pattern Recognition, 2016

2,684 citations recorded by the source corpus.

Generation of 3D data by deep neural network has been attracting increasing attention in the research community. The majority of extant works resort to regular representations such as volumetric grids or collection of images, however, these representations obscure the natural invariance of 3D shapes under geometric transformations, and also suffer from a number of other issues. In this paper we address the problem of 3D reconstruction from a single image, generating a straight-forward form of output – point cloud coordinates. Along with this problem arises a unique and interesting issue, that the groundtruth shape for an input image may be ambiguous. Driven by this unorthordox output form and the inherent ambiguity in groundtruth, we design architecture, loss function and learning paradigm that are novel and effective. Our final solution is a conditional shape sampler, capable of predicting multiple plausible 3D point clouds from an input image. In experiments not only can our system outperform state-of-the-art methods on single image based 3D reconstruction benchmarks, but it also shows strong performance for 3D shape completion and promising ability in making multiple plausible predictions.

Subfields: Aerospace Engineering, Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://arxiv.org/abs/1612.00603

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, background): “Set representations are commonly used in tasks across many data modalities ranging from point cloud prediction [29, 30], classifying multiple objects in an image [31], or generation of molecules with desired properties [32, 33].”
Multiview Compressive Coding for 3D Reconstruction (2023, background): “One line of work trains models that predict 3D geometry via CAD [16, 68], meshes [30,75], voxels [15,72] or point clouds [12,36] supervision.”
Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, methodology): “Existing representations can be broadly categorized into three categories: voxel-based representations [4,19,43,58,64,69,75] , point-based representations [1,17] and mesh representations [34,57,70], see Fig.”
RealPoint3D: Point Cloud Generation from a Single Image with Complex Background (2018, methodology): “Dataset: We use the ObjectNet3D dataset and follow the training and testing procedure used in (Fan, Su, and Guibas 2017) and (Tatarchenko, Dosovitskiy, and Brox 2017).”
Flexible Neural Representation for Physics Prediction (2018, methodology): “To enable a pixel-based end-to-end trainable version of the HRN for use in key computer vision applications, it will be critical to combine our work with adaptations of existing methods (e.g. [54, 23, 15]) for inferring initial (non-hierarchical) scene graphs from LIDAR/RGBD/RGB image or video data.”
Dynamic Graph CNN for Learning on Point Clouds (2018, neutral): “For point clouds, multiple generative architectures have been proposed [Fan et al. 2017; Li et al. 2018b; Yang et al. 2018].”
Deep Level Sets: Implicit Surface Representations for 3D Shape Inference (2019, background): “A majority of existing 3D representation learning approaches are based on voxel occupancy [3,12,28,29] but a considerable amount of attention has also been put on point clouds [10,27] and explicit shape parameterisation [21].”
Learning to Reconstruct Shapes from Unseen Classes (2018, background): “Apart from voxels, some researchers have also studied reconstructing objects in point clouds [Fan et al., 2017] or octave trees [Riegler et al., 2017, Tatarchenko et al., 2017].”
Learning Implicit Fields for Generative Shape Modeling (2018, background): “There have been a variety of 3D shape representations for deep learning of shapes, such as voxel grids [9, 14, 28, 45, 46], octrees [18, 35, 40, 43, 44], multi-view images [29, 39], point clouds [1, 12, 13, 32, 33, 47, 48], geometry images [37, 38], deformable mesh/patches [16, 38, 42, 47], and part”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, methodology): “While most of these approaches encode objects in voxels from vision, there have also been attempts to reconstruct objects in point clouds [11,15] or octave trees [40,49,39], or using tactile signals [53].”
RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials (2018, background): “In contrast, recent learning-based solutions to motion estimation [10, 15, 24], stereo matching [20, 21, 42] and 3D reconstruction [5, 6, 9, 16, 37] have demonstrated impressive results by learning feature representations that are much more robust to local viewpoint and lighting changes.”
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, background): “attempts to reconstruct objects in point clouds [12] or octave trees [44, 57].”
Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images (2018, methodology): “[9] (PSG) producing point cloud.”
Learning Category-Specific Mesh Reconstruction from Image Collections (2018, methodology): “Relying on ground-truth 3D supervision (using synthetic data), some approaches have examined learning voxel [4,8,39,33], point cloud [7] or octree [10,26] prediction.”
A Papier-Mache Approach to Learning 3D Surface Generation (2018, neutral): “To perform a quantitative comparison against PointSet-Gen [8], we evaluated the Chamfer distance between generated points and points from the original mesh for both PointSetGen and our method with 25 learned parameterizations.”
Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018, methodology): “Our surface accuracy results can be seen in table 3 compared to [44, 6, 14, 3].”
Learning a Multi-View Stereo Machine (2017, background): “Approaches which predict shape from a single image have been proposed in form of direct depth map regression [46, 31, 10], generating multiple depth maps from novel viewpoints [51], producing voxel occupancies [5, 16], geometry images [49] and point clouds [11].”
Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs (2017, methodology): “Networks producing point clouds have been applied to object generation [10] and semantic segmentation [26].”
Neural 3D Mesh Renderer (2017, methodology): “While some works reconstruct 3D structures via depth prediction [4, 25], others directly predict 3D shapes [2, 5, 30, 31, 34].”