Rethinking Reprojection: Closing the Loop for Pose-Aware Shape Reconstruction from a Single Image

By Rui Zhu, Hamed Kiani Galoogahi, Chaoyang Wang, Simon Lucey. 2017

91 citations recorded by the source corpus.

An emerging problem in computer vision is the reconstruction of 3D shape and pose of an object from a single image. Hitherto, the problem has been addressed through the application of canonical deep learning methods to regress from the image directly to the 3D shape and pose labels. These approaches, however, are problematic from two perspectives. First, they are minimizing the error between 3D shapes and pose labels - with little thought about the nature of this “label error” when reprojecting the shape back onto the image. Second, they rely on the onerous and ill-posed task of hand labeling natural images with respect to 3D shape and pose. In this paper we define the new task of pose-aware shape reconstruction from a single image, and we advocate that cheaper 2D annotations of objects silhouettes in natural images can be utilized. We design architectures of pose-aware shape reconstruction which reproject the predicted shape back on to the image using the predicted pose. Our evaluation on several object categories demonstrates the superiority of our method for predicting pose-aware 3D shapes from natural images.

Subfields: Aerospace Engineering, Computational Mechanics, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computational Mechanics

In Geology

Citations in the corpus

Learning Category-Specific Mesh Reconstruction from Image Collections (2018, methodology): “Relying on ground-truth 3D supervision (using synthetic data), some approaches have examined learning voxel [4,8,41,35], point cloud [7] or octree [10,28] prediction.”
The limits and potentials of deep learning for robotics (2018, background): “Similarly, there is a lot of ongoing work on estimating the 3D structure of objects from a single or multiple views without having depth information available [20], [40], [131], [137].”
Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction (2019, methodology): “When considering reconstructing 3D shape, voxel-based representation has received most attention [52, 56, 44, 48] due to its simplicity of regular structure and compatibility with convolutional neural network.”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “Also, the binary cross-entropy loss used in [4], [24] can be thought of as an approximation of the one derived using ray consistency.”
Differentiable surface splatting for point-based geometry processing (2019, methodology): “Numerous recent works employed DR for learning based 3D vision tasks, such as single view image reconstruction [Pontes et al. 2017; Vogels et al. 2018; Yan et al. 2016; Zhu et al. 2017], face reconstruction [Richardson et al.”
Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers (2018, background): “[34] showed that optimizing projections of the predicted shape benefits the reconstruction.”
DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction (2019, methodology): “Over the last few years, there have been extensive studies on learning based single-view 3D reconstruction using various 3D representations including voxels [28, 6, 31, 25, 20, 26, 29], octrees [8, 19, 22], points [5], and primitives [32, 13].”
BodyNet: Volumetric Inference of 3D Human Body Shapes (2018, unknown): “Similar losses have been employed for rigid objects by [54, 55] in the absence of 3D labels and by [21] as additional regularization.”
Image2Mesh: A Learning Framework for Single Image 3D Reconstruction (2017, methodology): “ ShapeNet [2]) has sparked a considerable interest in methods using deep learning to solve tasks related to geometry and 3D reconstruction. Several papers have proposed a 3D volumetric representation [3,4,10,11,20,24,27,30,33,35,38] so they can feed deep neural networks applying 3D convolutions, poo”