Point Cloud GAN

By Chun-Liang Li, Manzil Zaheer, Yang Zhang, Barnab'as P'oczos, Ruslan Salakhutdinov. 2018

231 citations recorded by the source corpus.

Generative Adversarial Networks (GAN) can achieve promising performance on learning complex data distributions on different types of data. In this paper, we first show a straightforward extension of existing GAN algorithm is not applicable to point clouds, because the constraint required for discriminators is undefined for set data. We propose a two fold modification to GAN algorithm for learning to generate point clouds (PC-GAN). First, we combine ideas from hierarchical Bayesian modeling and implicit generative models by learning a hierarchical and interpretable sampling process. A key component of our method is that we train a posterior inference network for the hidden variables. Second, instead of using only state-of-the-art Wasserstein GAN objective, we propose a sandwiching objective, which results in a tighter Wasserstein distance estimate than the commonly used dual form. Thereby, PC-GAN defines a generic framework that can incorporate many existing GAN algorithms. We validate our claims on ModelNet40 benchmark dataset. Using the distance between generated point clouds and true meshes as metric, we find that PC-GAN trained by the sandwiching objective achieves better results on test data than the existing methods. Moreover, as a byproduct, PC- GAN learns versatile latent representations of point clouds, which can achieve competitive performance with other unsupervised learning algorithms on object recognition task. Lastly, we also provide studies on generating unseen classes of objects and transforming image to point cloud, which demonstrates the compelling generalization capability and potentials of PC-GAN.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Dynamic Graph CNN for Learning on Point Clouds (2018, neutral): “For point clouds, multiple generative architectures have been proposed [Fan et al. 2017; Li et al. 2018b; Yang et al. 2018].”
Learning Gradient Fields for Shape Generation (2020, neutral): “005 .”
LION: Latent Point Diffusion Models for 3D Shape Generation (2022, background): “Both IM-GAN [7], which models shapes as neural fields, and l-GAN [2] train GANs over latent variables that encode the shapes, similar to other works [3], while r-GAN [2] generates point clouds directly.”
CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation (2021, background): “Recently, there has been tremendous progress in 3D shape generating in different data formats such as point cloud [3,34,59], voxel [58], implicit representation [8,37,44] and mesh [39].”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “As such, an increasing number of papers, at least one in 2017 [72], more than 12 in 2018 [21], [21], [22], [73], [74], [75], [76], [77], [78], [79], [80], [81], [82], and a few others in 2019 [81], explored their usage for deep learning-based reconstruction.”
3D Shape Generation and Completion through Point-Voxel Diffusion (2021, background): “GAN-based models and flow-based models [21, 1, 46, 16, 17] also adopt a probabilistic view but separate shape-level distribution from point-level distribution.”
Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures (2022, background): “The most conventional supervision type is applied directly with 3D shapes, through different representations such as implicit functions [10, 20, 33], meshes [19, 50] or point clouds [27, 49].”
Yes, we GAN: Applying Adversarial Techniques for Autonomous Driving (2019, background): “Most of GAN approaches are not applicable to 3D point clouds, however Point Cloud GAN (PC-GAN) [33] proposed a two fold modiﬁcation to GAN algorithm for learning to generate point clouds.”
MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers (2023, background): “Early works involved GANs for synthesizing point locations [30, 54, 59] and latent shape codes [1].”
Global-Local Bidirectional Reasoning for Unsupervised Representation Learning of 3D Point Clouds (2020, methodology): “PointNet [26] pioneers this line of works and designed a deep network that can handle unordered and unstructured 3D points by independently learning on each point and fusing point features with max pooling.”
PF-Net: Point Fractal Network for 3D Point Cloud Completion (2020, background): “Thus, algorithms which can directly conduct shape completion task on unordered point have been significantly developed [15, 10, 28, 33].”
PointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows (2019, methodology): “We compare our method with three existing generative models for point clouds: raw-GAN [1], latent-GAN [1], and PC-GAN [29], using their official implementations that are either publicly available or obtained by contacting the authors.”