Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images Using a View-Based Representation
By Sai Rajeswar, Fahim Mannan, Florian Golemo, Jérôme Parent-Lévesque, David Vázquez, Derek Nowrouzezahrai, Aaron C. Courville. 2020
11 citations recorded by the source corpus.
We infer and generate three-dimensional (3D) scene information from a single input image and without supervision. This problem is under-explored, with most prior work relying on supervision from, e.g., 3D ground-truth, multiple images of a scene, image silhouettes or key-points. We propose Pix2Shape, an approach to solve this problem with four component: (i) an encoder that infers the latent 3D representation from an image, (ii) a decoder that generates an explicit 2.5D surfel-based reconstruction of a scene—from the latent code—(iii) a differentiable renderer that synthesizes a 2D image from the surfel representation, and (iv) a critic network trained to discriminate between images generated by the decoder-renderer and those from a training distribution. Pix2Shape can generate complex 3D scenes that scale with the view-dependent on-screen resolution, unlike representations that capture world-space resolution, i.e., voxels or meshes. We show that Pix2Shape learns a consistent scene representation in its encoded latent space, and that the decoder can then be applied to this latent representation in order to synthesize the scene from a novel viewpoint. We evaluate Pix2Shape with experiments on the ShapeNet dataset as well as on a novel benchmark we developed – called 3D-IQTT—to evaluate models based on their ability to enable 3d spatial reasoning. Qualitative and quantitative evaluation demonstrate Pix2Shape’s ability to solve scene reconstruction, generation and understanding tasks.
Subfields: Computational Mechanics, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
- Jiajun Wu
- Josh Tenenbaum
- Gordon Wetzstein
- Eric Chan
- Petr Kellnhofer
- M. Monteiro
- Daniel L. K. Yamins
- Li Fei-Fei
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Josh Tenenbaum
- Jiajun Wu
- Li Fei-Fei
- Andrea Vedaldi
- Jitendra Malik
- Trevor Darrell
- Ming-Hsuan Yang
Citations in the corpus
- Bowtie Networks: Generative Modeling for Joint Few-Shot Recognition and Novel-View Synthesis (2020, unknown): “5D surfel based reconstruction [23].”
- pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis (2020, background): “Various 3D representations have been considered for the task, including voxels [18, 62], meshes [23, 6, 14, 17, 40], point clouds [59], a depth map [65], and implicit functions [50, 61].”
- Learning Physical Graph Representations from Visual Scenes (2020, methodology): “A few unsupervised models such as Pix2Shape [39], GQN [11], and neural rendering methods [47] have been proposed.”