Single Image 3D Interpreter Network

By Jiajun Wu, Tianfan Xue, Joseph J. Lim, Yuandong Tian, Josh Tenenbaum, Antonio Torralba, William T. Freeman. 2016

291 citations recorded by the source corpus.

Understanding 3D object structure from a single image is an important but difficult task in computer vision, mostly due to the lack of 3D object annotations in real images. Previous work tackles this problem by either solving an optimization task given 2D keypoint positions, or training on synthetic data with ground truth 3D information. In this work, we propose 3D INterpreter Network (3D-INN), an end-to-end framework which sequentially estimates 2D keypoint heatmaps and 3D object structure, trained on both real 2D-annotated images and synthetic 3D data. This is made possible mainly by two technical innovations. First, we propose a Projection Layer, which projects estimated 3D structure to 2D space, so that 3D-INN can be trained to predict 3D structural parameters supervised by 2D annotations on real images. Second, heatmaps of keypoints serve as an intermediate representation connecting real and synthetic data, enabling 3D-INN to benefit from the variation and abundance of synthetic 3D objects, without suffering from the difference between the statistics of real and synthesized images due to imperfect rendering. The network achieves state-of-the-art performance on both 2D keypoint estimation and 3D structure recovery. We also show that the recovered 3D information can be used in other vision applications, such as 3D rendering and image retrieval.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Flexible Neural Representation for Physics Prediction (2018, methodology): “To enable a pixel-based end-to-end trainable version of the HRN for use in key computer vision applications, it will be critical to combine our work with adaptations of existing methods (e.g. [54, 23, 15]) for inferring initial (non-hierarchical) scene graphs from LIDAR/RGBD/RGB image or video data.”
OctNet: Learning Deep 3D Representations at High Resolutions (2016, background): “While 2D convolutional networks have proven very successful in extracting information from images [11, 13, 18, 35, 41, 42, 46, 48, 49], there exists comparably little work on processing three-dimensional data.”
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, background): “Using deep convolutional networks, researchers have made significant progress in recent years [9, 17, 21, 29, 41, 43, 56, 59, 60, 62, 66, 52, 61].”
End-to-End Recovery of Human Shape and Pose (2017, methodology): “Even methods that are based on a reprojection loss [33, 47, 50] require paired 2D-to-3D training data.”
MarrNet: 3D Shape Reconstruction via 2.5D Sketches (2017, methodology): “These approaches usually regress a 3D shape from a single RGB image directly [Tulsiani et al., 2017, Choy et al., 2016, Wu et al., 2016b].”
Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency (2017, methodology): “[36], using similar annotations, learned a system to predict sparse 3D by inferring parameters of a shape skeleton.”
OctNetFusion: Learning Depth Fusion from Data (2017, background): “By utilizing large datasets of 3D models and high capacity 3D convolutional neural networks (3D CNNs), our approach learns to smooth out sensor noise, deals with outliers and completes missing 3D geometry.”
Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling (2016, background): “…3D shapes and synthesized images, Su et al. [2015a], Qi et al. [2016] focused on learning discriminative representations for 3D object recognition, Wu et al. [2016], Xiang et al. [2015], Choy et al. [2016] discussed 3D object reconstruction from in-the-wild images, possibly with a recurrent…”
Learning a Predictable and Generative Vector Representation for Objects (2016, neutral): “Related works in 3D prediction include [17, 38, 3].”
Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision (2016, methodology): “lutional networks, Jaderberg et al. [6] introduced a differentiable sampling layer that directly incorporates geometric transformations into representation learning. Concurrent to our work, Wu et al. [21] proposed a 3D-2D projection layer that enables the learning of 3D object structures using 2D ke”
Synthesizing 3D Shapes via Modeling Multi-view Depth Maps and Silhouettes with Deep Generative Networks (2017, neutral): “Different approaches for modeling 3D shapes fall into three key categories: parts-based [8, 10, 1, 26], voxel-based [19, 27, 6, 28], and view based methods [18, 23].”
Neural Scene De-rendering (2017, background): “Our work is closely related to research on learning an interpretable representation with a neural network [13, 21, 35, 4, 33].”
Compositional Human Pose Regression (2017, background): “[48] propose 3D interpreter network that sequentially estimates 2D keypoint heat maps and 3D object structure.”
SurfNet: Generating 3D Shape Surfaces Using Deep Residual Networks (2017, background): “A 3D interpreter network is developed in [35] which estimates the 3D skeleton of a shape.”
IM2CAD (2016, background): “Another closely related line of research is 3D object and pose recognition of chairs and other objects [3, 20, 36, 23, 18, 43, 4, 44].”
KETO: Learning Keypoint Representations for Tool Manipulation (2019, background): “Most keypoint representations are either defined as low-level features in detection and matching [32], [47], [2] or used to describe object parts on a category level [24], [56].”
Learning 3D Human Dynamics From Video (2018, methodology): “In order to train these modules from 2D pose annotations, we employ the reprojection loss [51] and the adversarial prior proposed in [27] to constrain the output pose to lie in the space of possible human poses.”
StarMap for Category-Agnostic Keypoint and Viewpoint Estimation (2018, methodology): “…represents object keypoints by a N × D vector ( N for number of keypoints and D for dimensions), or multi-channel heatmaps [34,24], which associate each channel with one speciﬁc keypoint on a speciﬁc object category, e.g., 16-channel heatmaps for human [34,24], 10-channel heatmaps for chair [42].”
Using Locally Corresponding CAD Models for Dense 3D Reconstructions from a Single Image (2017, background): “[18] trained a deep network to infer 3D points from 2D landmarks.”