3D Shape Induction from 2D Views of Multiple Objects

By Matheus Gadelha, Subhransu Maji, Rui Wang. 2016

311 citations recorded by the source corpus.

In this paper we investigate the problem of inducing a distribution over three-dimensional structures given two-dimensional views of multiple objects taken from unknown viewpoints. Our approach called "projective generative adversarial networks" (PrGANs) trains a deep generative model of 3D shapes whose projections match the distributions of the input 2D views. The addition of a projection module allows us to infer the underlying 3D shape distribution without using any 3D, viewpoint information, or annotation during the learning phase. We show that our approach produces 3D shapes of comparable quality to GANs trained on 3D data for a number of shape categories including chairs, airplanes, and cars. Experiments also show that the disentangled representation of 2D shapes into geometry and viewpoint leads to a good generative model of 2D shapes. The key advantage is that our model allows us to predict 3D, viewpoint, and generate novel views from an input image in a completely unsupervised manner.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, background): “Existing representations can be broadly categorized into three categories: voxel-based representations [4,19,43,58,64,69,75] , point-based representations [1,17] and mesh representations [34,57,70], see Fig.”
Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View (2019, background): “face frontal view generation [50], generating new human poses [51], photos to emojis [52], photograph editing [53]–[55], face aging [56], [57], photo blending [58], super resolution [59]–[61], photo inpainting [62]–[64], video prediction [65], and 3D object generation [66], [67]; among”
Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs (2017, methodology): “Applications include single- and multi-view 3D object reconstruction trained in supervised [14, 5, 7, 17] and unsupervised [37, 12, 19] ways, probabilistic generative modeling of 3D shapes [36, 35, 29], semantic segmentation [2, 4] and shape deformation [39].”
Unsupervised Learning of Depth and Ego-Motion from Video (2017, background): “, [13, 41, 46, 52]) has also shown success in learning 3D volumetric representations from 2D observations based on similar principles of projective geometry.”
Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild (2019, unknown): “[13] 2DS Prior on 3DV, pred.”
Rethinking Reprojection: Closing the Loop for Pose-Aware Shape Reconstruction from a Single Image (2017, background): “Reprojection Loss as Supervision The role of reprojection loss in related tasks has been explored in various occasions, including constraining 3D reconstruction with multi-view annotations [29], weakly-supervised shape optimization [9, 22], or structure from silhouettes [5].”
Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction (2017, background): “Similar applications include the use of an additional encoded pose embedding to learn shape deformations [33] and using adversarial training to learn more realistic shape generation [28, 5].”
A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, background): “3D image synthesis: 3D-GAN [76], Pr-GAN [77], 3D-CGAN [78]”
3D Neural Field Generation Using Triplane Diffusion (2022, unknown): “For example, concurrently developed single-latent approaches [2,14] generate a global latent for conditioning the neural field, relying on a 3D decoder to transform the scene representation from 1D to 3D without directly performing 3D diffusion.”
Yes, we GAN: Applying Adversarial Techniques for Autonomous Driving (2019, background): “Pix2Pix [3], SRGAN [23], CycleGAN [4], DiscoGAN [24], StarGAN [25], UNIT [26], Pix2PixHD [27], Bicycle-GAN [28], MUNIT [29], Augmented GAN [30] 3D Synthesis 3D-GAN[31], PrGAN [32], PC-GAN [33] Video Synthesis video-to-video [34], TGAN [35], [36]”
OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation (2023, background): “Early approaches [27, 35, 54, 81, 93] extend 2D generation frameworks to 3D voxels with a high computational cost when generating high-resolution contents.”
Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction (2021, background): “Volumetric represe-nations are projected to 2D via differentiable raymarching [25, 15, 41, 43, 39, 55] or in a similar fashion via sphere tracing for signed distance ﬁelds [45, 74].”
Generative Adversarial Networks (GANs) (2020, methodology): “Several researchers proposed to tackle the training instability problem by finding better distance measures [29] or regularizers [28][45].”
Tangent Convolutions for Dense Prediction in 3D (2018, methodology): “In a more controlled setting with fixed camera poses, multi-view methods are successfully used for shape segmentation [21], shape recognition [49, 40], and shape synthesis [14, 30, 22, 50].”
Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics (2019, background): “They can broadly be categorized into voxel-based [10, 14, 21, 53, 54, 59, 74], point cloud-based [1, 20], mesh-based [25, 29, 34, 71], and continuous representations [13, 22, 28, 38, 39, 46, 56, 72].”
Unsupervised Learning of 3D Object Categories from Videos in the Wild (2021, unknown): “Generative models trained in the wild were proposed in [9, 38].”
Visual Object Networks: Image Generation with Disentangled 3D Representations (2018, methodology): “Popular representations include voxels [Wu et al., 2016], point clouds [Gadelha et al., 2017b, Achlioptas et al., 2018], and octave trees [Tatarchenko et al., 2017].”
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, methodology): “Previously GAN-based approached have explored different methods to generate 3D models in the form of point clouds (Wu et al., 2016; Gadelha et al., 2017; Smith & Meger, 2017), triangle meshes (Gao et al., 2022; Pavllo et al., 2020; Chen et al., 2019; Luo et al., 2021) or volumes (Chan et al., 2022;…”