Improved Adversarial Systems for 3D Object Generation and Reconstruction

By Edward James Smith, David Meger. 2017

187 citations recorded by the source corpus.

This paper describes a new approach for training generative adversarial networks (GAN) to understand the detailed 3D shape of objects. While GANs have been used in this domain previously, they are notoriously hard to train, especially for the complex joint data distribution over 3D objects of many categories and orientations. Our method extends previous work by employing the Wasserstein distance normalized with gradient penalization as a training objective. This enables improved generation from the joint object shape distribution. Our system can also reconstruct 3D shape from 2D images and perform shape completion from occluded 2.5D range scans. We achieve notable quantitative improvements in comparison to existing baselines

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018, methodology): “Implementation For 3D object reconstruction, we first trained a standard autoencoder, similar to prior work [3, 41], to produce objects at 32(3) resolution.”
Transferring GANs: generating images from limited data (2018, methodology): “GANs have further been widely applied in applications, including super-resolution [26], 3D object generation and reconstruction [40], human pose estimation [28], and age estimation [47].”
3D Object Dense Reconstruction from a Single Depth View (2018, background): “Recently, [69] [70] [71] [21] applied generative networks for 3D structure generation.”
OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation (2023, background): “Early approaches [20,26,37,52,62] extend 2D generation frameworks to 3D voxels with a high computational cost.”
PU-GAN: A Point Cloud Upsampling Adversarial Network (2019, background): “Most of them focus on generating 3D objects either from a probabilistic space [32] or from 2D images [36, 11, 27].”
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, methodology): “…GAN-based approached have explored different methods to generate 3D models in the form of point clouds (Wu et al., 2016; Gadelha et al., 2017; Smith & Meger, 2017), triangle meshes (Gao et al., 2022; Pavllo et al., 2020; Chen et al., 2019; Luo et al., 2021) or volumes (Chan et al., 2022;…”
PCN: Point Completion Network (2018, methodology): “While most existing learning-based methods [9, 16, 44, 47, 50, 54, 56, 59] represents shapes using voxels, which are convenient for convolutional neural networks, our method uses point clouds, which preserve complete geometric information about the shapes while being memory efﬁcient.”
SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation (2022, background): “For instance, 3D generative models have been explored for point clouds [2, 21], voxel grids [20, 42, 47], meshes [51], signed distance functions (SDFs) [9, 11, 25], etc.”
An Acceleration Framework for High Resolution Image Synthesis (2019, background): “2018), high resolution image generation [2, 6, 7, 14], 3D generation and reconstruction [15, 13, 16].”
Magic3D: High-Resolution Text-to-3D Content Creation (2022, background): “There is a large body of work on 3D generative modeling, exploring different types of 3D representations such as 3D voxel grids [7, 12, 20, 40, 45], pointclouds [1, 21, 24, 46, 47, 49], meshes [9, 48], implicit [6, 22], or octree [15] representations.”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “Wu et al.’s work [3] has led to several extensions [7], [8], [17], [27], [40].”
Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer (2019, unknown): “Adverserial 3D Object Generation: Generation of 3D shapes through deep learning has been approached using a Generative Adverserial Network (GAN) [5] in a plethora of work [39, 1, 37, 30].”
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images (2022, methodology): “This initial feature grid is up-sampled to a feature grid F ′ tex ∈ R 256 × 256 × 96 that is conditioned on w 1 and w 2 .”