Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

By Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, Josh Tenenbaum. 2016

2,154 citations recorded by the source corpus.

We study the problem of 3D object generation. We propose a novel framework, namely 3D Generative Adversarial Network (3D-GAN), which generates 3D objects from a probabilistic space by leveraging recent advances in volumetric convolutional networks and generative adversarial nets. The benefits of our model are three-fold: first, the use of an adversarial criterion, instead of traditional heuristic criteria, enables the generator to capture object structure implicitly and to synthesize high-quality 3D objects; second, the generator establishes a mapping from a low-dimensional probabilistic space to the space of 3D objects, so that we can sample objects without a reference image or CAD models, and explore the 3D object manifold; third, the adversarial discriminator provides a powerful 3D shape descriptor which, learned without supervision, has wide applications in 3D object recognition. Experiments demonstrate that our method generates high-quality 3D objects, and our unsupervisedly learned features achieve impressive performance on 3D object recognition, comparable with those of supervised learning methods.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Learning Descriptor Networks for 3D Shape Synthesis and Analysis (2018, methodology): “It can be considered an alternative to 3D-GAN [28] for 3D shape generation.”
Gradient descent GAN optimization is locally stable (2017, methodology): “…deployed for tasks such as photo super-resolution, object generation, video prediction, language modeling, vocal synthesis, and semi-supervised learning, amongst many others [Ledig et al., 2016, Wu et al., 2016, Mathieu et al., 2015, Nguyen et al., 2016, Denton et al., 2015, Im et al., 2016].”
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, methodology): “Recent methods adopt the same idea for conditional image generation applications, such as text2image [36], image inpainting [34], and future prediction [32], as well as to other domains like videos [50] and 3D models [53].”
Towards Principled Methods for Training Generative Adversarial Networks (2017, background): “(Denton et al., 2015; Radford et al., 2015; Salimans et al., 2016; Lamb et al., 2016; Wu et al., 2016) However, they still remain remarkably difﬁcult to train, with most current papers dedicated to heuristically ﬁnding stable architectures.”
Mode Regularized Generative Adversarial Networks (2016, background): “l on various tasks, such as image generation, image super-resolution, 3D object generation, and video prediction (Radford et al., 2015; Ledig et al., 2016; Sønderby et al., 2016; Nguyen et al., 2016; Wu et al., 2016; Mathieu et al., 2015). The objective is to train a parametrized function (the gener”
Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, background): “Voxel Representations: Due to their simplicity, voxels are the most commonly used representation for discriminative [45,55,63] and generative [9,23,58,64,75,77] 3D tasks.”
Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View (2019, background): “face frontal view generation [50], generating new human poses [51], photos to emojis [52], photograph editing [53]–[55], face aging [56], [57], photo blending [58], super resolution [59]–[61], photo inpainting [62]–[64], video prediction [65], and 3D object generation [66], [67]; among”
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation (2019, methodology): “[54] trains a GAN to generate objects in a voxel representation, while the recent work of Hamu et al.”
Learning to Reconstruct Shapes from Unseen Classes (2018, background): “…progress in recent years, mostly with voxel representations [Choy et al., 2016, Girdhar et al., 2016, Häne et al., 2017, Kar et al., 2015, Novotny et al., 2017, Rezende et al., 2016, Tatarchenko et al., 2016, Tulsiani et al., 2017, Wu et al., 2016, 2017, 2018, Zhu et al., 2018, Yan et al., 2016].”
Learning Implicit Fields for Generative Shape Modeling (2018, methodology): “We also compared our results with two state-of-the-art methods, 3DGAN [45] and the generative model for point clouds in [1] (PC-GAN).”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, methodology): “Our model characterizes this naturalness loss through adversarial learning, a research topic that has received immense attention in recent years and is still rapidly growing [14,37,57].”
RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials (2018, background): “Learning-based 3D Reconstruction: The development of large 3D shape databases [4, 38] has fostered the development of learning based solutions [3, 5, 8, 37] to 3D reconstruction.”
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, methodology): “We compare our model, with and without the pose estimation branch, with the state-ofthe-art systems, including 3D-VAE-GAN [62], 3D-R2N2 [9],”
Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018, methodology): “For 3D generation, these methods typically use an autoencoder network, with a decoder composed of 3D deconvolutional layers [3, 46].”
MarrNet: 3D Shape Reconstruction via 2.5D Sketches (2017, methodology): “These approaches usually regress a 3D shape from a single RGB image directly [Tulsiani et al., 2017, Choy et al., 2016, Wu et al., 2016b].”
Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs (2017, methodology): “Applications include single- and multi-view 3D object reconstruction trained in supervised [14, 5, 7, 17] and unsupervised [37, 12, 19] ways, probabilistic generative modeling of 3D shapes [36, 35, 29], semantic segmentation [2, 4] and shape deformation [39].”
3D Shape Induction from 2D Views of Multiple Objects (2016, methodology): “…multiple categories are combined; (ii) On the task of generating 3D shapes, PrGANs perform well in comparison to GANs trained directly on 3D data [33]; (iii) The learned 3D representation can be used for unsupervised estimation of 3D shape and view-point given a novel 2D shape, and for…”
Shape Completion Using 3D-Encoder-Predictor CNNs and Shape Synthesis (2016, background): “Recently deep learning also explore models for generative tasks; for instance, with generative adversarial networks (GANs) [10, 19, 18, 44].”