Weakly Supervised 3D Reconstruction with Adversarial Constraint

By JunYoung Gwak, Christopher Bongsoo Choy, Manmohan Chandraker, Animesh Garg, Silvio Savarese. 2017

124 citations recorded by the source corpus.

Supervised 3D reconstruction has witnessed a significant progress through the use of deep neural networks. However, this increase in performance requires large scale annotations of 2D/3D data. In this paper, we explore inexpensive 2D supervision as an alternative for expensive 3D CAD annotation. Specifically, we use foreground masks as weak supervision through a raytrace pooling layer that enables perspective projection and backpropagation. Additionally, since the 3D reconstruction from masks is an ill posed problem, we propose to constrain the 3D reconstruction to the manifold of unlabeled realistic 3D shapes that match mask observations. We demonstrate that learning a log-barrier solution to this constrained optimization problem resembles the GAN objective, enabling the use of existing tools for training GANs. We evaluate and analyze the manifold constrained reconstruction on various datasets for single and multi-view reconstruction of both synthetic and real images.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, methodology): “[17] explored adversarial nets for single-image 3D reconstruction and chose to use GANs to model 2D projections instead of 3D shapes.”
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, background): “Chairs cover the significant part of Pix3D, because they are common, highly diverse, and well-studied by recent literature [11, 59, 20].”
Learning Category-Specific Mesh Reconstruction from Image Collections (2018, background): “Reducing the supervision to multi-view masks [34,21,30,9] or depth images [30] has been explored for voxel prediction, but the requirement of multiple views per instance is still restrictive.”
3D Object Dense Reconstruction from a Single Depth View (2018, background): “For general and complex object reconstruction from a single RGB image, recent works [52] [53] [54] aim to infer 3D shapes using multiple RGB images for weak supervision.”
Self-supervised Learning of 3D Objects from Natural Images (2019, background): “Learning object naturalness from data [10, 44] would be a promising direction.”
Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation (2019, unknown): “[12] learns shapes from multi-view silhouettes”
PU-GAN: A Point Cloud Upsampling Adversarial Network (2019, background): “Most of them focus on generating 3D objects either from a probabilistic space [32] or from 2D images [36, 11, 27].”
Geometric Capsule Autoencoders for 3D Point Clouds (2019, background): “The idea of using multiple views to come up with a coherent understanding of an object has been studied extensively [20, 17, 18, 13, 4, 5, 23, 24, 6, 9].”