Unsupervised Learning of 3D Structure from Images

By Danilo Jimenez Rezende, S. M. Ali Eslami, Shakir Mohamed, Peter W. Battaglia, Max Jaderberg, Nicolas Manfred Otto Heess. 2016

410 citations recorded by the source corpus.

A key goal of computer vision is to recover the underlying 3D structure from 2D observations of the world. In this paper we learn strong deep generative models of 3D structures, and recover these structures from 3D and 2D images via probabilistic inference. We demonstrate high-quality samples and report log-likelihoods on several datasets, including ShapeNet [2], and establish the first benchmarks in the literature. We also show how these models and their inference networks can be trained end-to-end from 2D images. This demonstrates for the first time the feasibility of learning to infer 3D representations of the world in a purely unsupervised manner.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, background): “Voxel Representations: Due to their simplicity, voxels are the most commonly used representation for discriminative [45,55,63] and generative [9,23,58,64,75,77] 3D tasks.”
A Point Set Generation Network for 3D Object Reconstruction from a Single Image (2016, methodology): “The use of synthesized data has been adopted in a number of existing works [5, 16].”
Causal Effect Inference with Deep Latent-Variable Models (2017, neutral): “Since their recent introduction, VAEs have been shown to be remarkably successful in capturing latent structure across a wide-range of previously difficult problems, such as modeling images [19], volumes [24], time-series [10] and fairness [34].”
OctNet: Learning Deep 3D Representations at High Resolutions (2016, background): “Recently, generative models [36] and auto-encoders [5, 39] have demonstrated impressive performance in learning low-dimensional object representations from collections of low-resolution (32(3)) 3D shapes.”
Deep Level Sets: Implicit Surface Representations for 3D Shape Inference (2019, methodology): “Convolutional neural networks [9,12,16,22,28] and generative adversarial models (GANs) [3, 24] have been successfully applied to 3D reconstruction problems by using either volumetric or point cloud representations.”
Learning to Reconstruct Shapes from Unseen Classes (2018, background): “…progress in recent years, mostly with voxel representations [Choy et al., 2016, Girdhar et al., 2016, Häne et al., 2017, Kar et al., 2015, Novotny et al., 2017, Rezende et al., 2016, Tatarchenko et al., 2016, Tulsiani et al., 2017, Wu et al., 2016, 2017, 2018, Zhu et al., 2018, Yan et al., 2016].”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, methodology): “With the development of large-scale shape repositories like ShapeNet [5] and methods like deep convolutional networks, researchers have built more scalable and eﬃcient models in recent years [7,13,18,27,36,38,48,52,56,57,62].”
RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials (2018, background): “[26] propose a neural projection layer and a black box renderer for supervising the learning process.”
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, background): “Using deep convolutional networks, researchers have made significant progress in recent years [9, 17, 21, 29, 41, 43, 56, 59, 60, 62, 66, 52, 61].”
Learning Category-Specific Mesh Reconstruction from Image Collections (2018, background): “Reducing the supervision to multi-view masks [34,21,30,9] or depth images [30] has been explored for voxel prediction, but the requirement of multiple views per instance is still restrictive.”
MarrNet: 3D Shape Reconstruction via 2.5D Sketches (2017, background): “There have been works attempting to enforce the consistency between estimated 3D shape and 2D representations in a neural network [Yan et al., 2016, Rezende et al., 2016, Wu et al., 2016a, Tulsiani et al., 2017].”
Learning a Multi-View Stereo Machine (2017, background): “A line of recent works, complementary to ours, has proposed to incorporate ideas from multi-view geometry in a learning framework to train single view prediction systems [14, 56, 53, 42, 61] using multiple views as supervisory signal.”
Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency (2017, background): “We note that some concurrent approaches [25, 38] have also been proposed to specifically address the case of learning object reconstruction from foreground masks.”
OctNetFusion: Learning Depth Fusion from Data (2017, background): “Generalization on Unseen Categories: Most existing approaches that leverage deep learning for 3D reconstruction train a model specific for a particular object class [4, 8, 17, 41, 35] which typically does not generalize well to other views=10 views=20”
Hierarchical Surface Prediction for 3D Object Reconstruction (2017, background): “ground truth shapes, CNNs can also be trained to predict 3D shapes using weaker supervision [27], [35], [34] for example image silhouettes.”
Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs (2017, methodology): “Applications include single- and multi-view 3D object reconstruction trained in supervised [14, 5, 7, 17] and unsupervised [37, 12, 19] ways, probabilistic generative modeling of 3D shapes [36, 35, 29], semantic segmentation [2, 4] and shape deformation [39].”
3D Shape Induction from 2D Views of Multiple Objects (2016, background): “Rezende et al . in [24] show results for 3D shape completion for simple shapes when views are provided, but require the viewpoints to be known and the generative models are trained on 3D data.”
Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling (2016, background): “, 2015] for testing, following Sharma et al. [2016], Maturana and Scherer [2015], Qi et al.”
Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision (2016, background): “[16] introduced a general framework to learn 3D structures from 2D observations with 3D-2D projection mechanism.”