Category-specific object reconstruction from a single image

By Abhishek Kar, Shubham Tulsiani, João Carreira, Jitendra Malik. 2014

295 citations recorded by the source corpus.

Object reconstruction from a single image - in the wild - is a problem where we can make progress and get meaningful results today. This is the main message of this paper, which introduces an automated pipeline with pixels as inputs and 3D surfaces of various rigid categories as outputs in images of realistic scenes. At the core of our approach are deformable 3D models that can be learned from 2D annotations available in existing object detection datasets, that can be driven by noisy automatic object segmentations and which we complement with a bottom-up module for recovering high-frequency shape details. We perform a comprehensive quantitative analysis and ablation study of our approach using the recently introduced PASCAL 3D+ dataset and show very encouraging automatic reconstructions on PASCAL VOC.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Geology

In 3D Vision & Pose

Citations in the corpus

A Point Set Generation Network for 3D Object Reconstruction from a Single Image (2016, neutral): “Coarse correspondences in an image collection can also be used for rough 3D shape estimation [14, 3].”
Learning to Reconstruct Shapes from Unseen Classes (2018, background): “…to the significant progress in recent years, mostly with voxel representations [Choy et al., 2016, Girdhar et al., 2016, Häne et al., 2017, Kar et al., 2015, Novotny et al., 2017, Rezende et al., 2016, Tatarchenko et al., 2016, Tulsiani et al., 2017, Wu et al., 2016, 2017, 2018, Zhu et…”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, methodology): “With the development of largescale shape repositories like ShapeNet [5] and methods like deep convolutional networks, researchers have built more scalable and efficient models in recent years [7,13,18,27,36,38,48,52,56,57,62].”
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, background): “Using deep convolutional networks, researchers have made significant progress in recent years [9, 17, 21, 29, 41, 43, 56, 59, 60, 62, 66, 52, 61].”
Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images (2018, background): “To avoid this problem, Kar et al.[16] learns a 3D deformable model for each object category and capture the shape variations in different images.”
Learning Category-Specific Mesh Reconstruction from Image Collections (2018, methodology): “We compare to previous deformable model fittingbased [15] and volumetric prediction [30] approaches that use similar image collection supervision.”
MarrNet: 3D Shape Reconstruction via 2.5D Sketches (2017, methodology): “…like ShapeNet [Chang et al., 2015], researchers developed models encoding shape prior for this task [Girdhar et al., 2016, Choy et al., 2016, Tulsiani et al., 2017, Wu et al., 2016b, Kar et al., 2015, Kanazawa et al., 2016, Soltani et al., 2017], with extension to scenes [Song et al., 2017].”
Learning a Multi-View Stereo Machine (2017, methodology): “Learning has been used for multi-view reconstruction in the form of shape priors for objects [2, 9, 58, 20, 27, 52], or semantic class specific surface priors for scenes [22, 17, 45].”
Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling (2016, methodology): “…can be used for 3D shape classiﬁcation [Su et al., 2015a, Sharma et al., 2016, Maturana and Scherer, 2015], 3D shape retrieval [Shi et al., 2015, Su et al., 2015a], and single image 3D reconstruction [Kar et al., 2015, Bansal et al., 2016, Girdhar et al., 2016], mostly with full supervision.”
3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction (2016, methodology): “[32] took ground-truth object segmentation masks and keypoint labels as additional inputs for both training and testing.”
Learning a Predictable and Generative Vector Representation for Objects (2016, methodology): “(1) In the first stage, we train the autoencoder part of the network independently.”
Single Image 3D Interpreter Network (2016, methodology): “Depth-/mesh-based models can recover detailed 3D object structure from a single image, either by adapting existing 3D models from a database [3,19,6,8,20,21,22,23,24], or by inferring from its detected 2D silhouette [5,7,4].”
Multi-view 3D Models from Single Images with a Convolutional Network (2015, background): “[18] extended this concept to object categories with more variation, such as cars and chairs, and combined it with shape-from-shading to retrieve also the high frequency components of the shape.”
Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images Using a View-Based Representation (2020, background): “…such asmultiple views of the same object (Yan et al. 2016; Tulsiani et al. 2017; Li et al. 2019), 2.5D supervision (Wu et al. 2016a, 2017), key-point (Kar et al. 2014; Novotný et al. 2019) and silhouette annotations (Wiles and Zisserman 2017; Henderson and Ferrari 2018; Chen et al. 2019).”
KETO: Learning Keypoint Representations for Tool Manipulation (2019, background): “Most keypoint representations are either defined as low-level features in detection and matching [32], [47], [2] or used to describe object parts on a category level [24], [56].”