Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision

By Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, Honglak Lee. 2016

607 citations recorded by the source corpus.

Understanding the 3D world is a fundamental problem in computer vision. However, learning a good representation of 3D objects is still an open problem due to the high dimensionality of the data and many factors of variation involved. In this work, we investigate the task of single-view 3D object reconstruction from a learning agent's perspective. We formulate the learning process as an interaction between 3D and 2D representations and propose an encoder-decoder network with a novel projection loss defined by the projective transformation. More importantly, the projection loss enables the unsupervised learning using 2D observation without explicit 3D supervision. We demonstrate the ability of the model in generating 3D volume from a single 2D image with three sets of experiments: (1) learning from single-class objects; (2) learning from multi-class objects and (3) testing on novel object classes. Results show superior performance and better generalization ability for 3D object reconstruction when the projection loss is involved.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In 3D Vision & Pose

Citations in the corpus

Multiview Compressive Coding for 3D Reconstruction (2023, background): “Some [66, 77] explore category-agnostic models, but focus on synthetic datasets.”
Deep Level Sets: Implicit Surface Representations for 3D Shape Inference (2019, background): “To tackle this problem, different CNNs-based approaches have been introduced which require only weak supervision [28, 34], and they are able to handle more shape variations.”
Learning to Reconstruct Shapes from Unseen Classes (2018, background): “…progress in recent years, mostly with voxel representations [Choy et al., 2016, Girdhar et al., 2016, Häne et al., 2017, Kar et al., 2015, Novotny et al., 2017, Rezende et al., 2016, Tatarchenko et al., 2016, Tulsiani et al., 2017, Wu et al., 2016, 2017, 2018, Zhu et al., 2018, Yan et al., 2016].”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, methodology): “With the development of large-scale shape repositories like ShapeNet [5] and methods like deep convolutional networks, researchers have built more scalable and eﬃcient models in recent years [7,13,18,27,36,38,48,52,56,57,62].”
RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials (2018, background): “Yan et al. [39] and Gwak et al. [9] use 2D silhouettes as supervision for 3D reconstruction from a single image.”
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, background): “Using deep convolutional networks, researchers have made significant progress in recent years [9, 17, 21, 29, 41, 43, 56, 59, 60, 62, 66, 52, 61].”
Learning Category-Specific Mesh Reconstruction from Image Collections (2018, background): “Reducing the supervision to multi-view masks [36,22,32,9] or depth images [32] has been explored for voxel prediction, but the requirement of multiple views per instance is still restrictive.”
MarrNet: 3D Shape Reconstruction via 2.5D Sketches (2017, background): “There have been works attempting to enforce the consistency between estimated 3D shape and 2D representations in a neural network [Yan et al., 2016, Rezende et al., 2016, Wu et al., 2016a, Tulsiani et al., 2017].”
Learning a Multi-View Stereo Machine (2017, background): “A line of recent works, complementary to ours, has proposed to incorporate ideas from multi-view geometry in a learning framework to train single view prediction systems [14, 56, 53, 42, 61] using multiple views as supervisory signal.”
Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency (2017, background): “We note that some concurrent approaches [25, 38] have also been proposed to specifically address the case of learning object reconstruction from foreground masks.”
Hierarchical Surface Prediction for 3D Object Reconstruction (2017, background): “Recently, 3D prediction approaches which directly learn a function to map an input image to the output geometry have emerged [6], [13], [35].”
Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs (2017, methodology): “Applications include single- and multi-view 3D object reconstruction trained in supervised [14, 5, 7, 17] and unsupervised [37, 12, 19] ways, probabilistic generative modeling of 3D shapes [36, 35, 29], semantic segmentation [2, 4] and shape deformation [39].”
3D Shape Induction from 2D Views of Multiple Objects (2016, background): “in [36] learn a mapping from an image to 3D using multiple projections of the 3D shape from known viewpoints.”
The limits and potentials of deep learning for robotics (2018, background): “Similarly, there is a lot of ongoing work on estimating the 3D structure of objects from a single or multiple views without having depth information available [20], [40], [131], [137].”
Unsupervised Learning of Depth and Ego-Motion from Video (2017, background): “, [13, 41, 46, 52]) has also shown success in learning 3D volumetric representations from 2D observations based on similar principles of projective geometry.”
Neural 3D Mesh Renderer (2017, methodology): “[36] demonstrated that single-image 3D reconstruction can be realized without 3D training data.”
Synthesizing 3D Shapes via Modeling Multi-view Depth Maps and Silhouettes with Deep Generative Networks (2017, background): “View based representations have demonstrated strong potential for both shape recognition and synthesis [23, 18, 5, 25, 16, 13, 29].”
Geometric Capsule Autoencoders for 3D Point Clouds (2019, background): “The idea of using multiple views to come up with a coherent understanding of an object has been studied extensively [20, 17, 18, 13, 4, 5, 23, 24, 6, 9].”
3D-Aware Scene Manipulation via Inverse Graphics (2018, methodology): “We also use a reprojection loss to ensure the 2D rendering of the predicted shape ﬁts its silhouette S [Yan et al., 2016b, Rezende et al., 2016, Wu et al., 2016a, 2017b].”