Multi-view 3D Models from Single Images with a Convolutional Network

By Maxim Tatarchenko, Alexey Dosovitskiy, Thomas Brox. 2015

388 citations recorded by the source corpus.

We present a convolutional network capable of inferring a 3D representation of a previously unseen object given a single image of this object. Concretely, the network can predict an RGB image and a depth map of the object as seen from an arbitrary view. Several of these depth maps fused together give a full point cloud of the object. The point cloud can in turn be transformed into a surface mesh. The network is trained on renderings of synthetic 3D models of cars and chairs. It successfully deals with objects on cluttered background and generates reasonable predictions for real images of cars.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

OctNet: Learning Deep 3D Representations at High Resolutions (2016, background): “While 2D convolutional networks have proven very successful in extracting information from images [11, 13, 18, 35, 41, 42, 46, 48, 49], there exists comparably little work on processing three-dimensional data.”
Learning to Reconstruct Shapes from Unseen Classes (2018, background): “…progress in recent years, mostly with voxel representations [Choy et al., 2016, Girdhar et al., 2016, Häne et al., 2017, Kar et al., 2015, Novotny et al., 2017, Rezende et al., 2016, Tatarchenko et al., 2016, Tulsiani et al., 2017, Wu et al., 2016, 2017, 2018, Zhu et al., 2018, Yan et al., 2016].”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, methodology): “D models. With the development of largescale shape repositories like ShapeNet [5] and methods like deep convolutional networks, researchers have built more scalable and ecient models in recent years [7,13,18,27,36,38,48,52,56,57,62]. While most of these approaches encode 4 J. Wu et al. objects in vo”
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, background): “Using deep convolutional networks, researchers have made significant progress in recent years [9, 17, 21, 29, 41, 43, 56, 59, 60, 62, 66, 52, 61].”
SurfaceNet: An End-to-End 3D Neural Network for Multiview Stereopsis (2017, background): “While the 3D reconstruction of an object from a single or multiple views using convolutional neural networks has been studied in [26, 5], the approaches focus on a very coarse reconstruction without geometric details when only one or very few images are available.”
Learning a Multi-View Stereo Machine (2017, methodology): “This geometric procedure of lifting features from 2D maps into 3D space is in contrast with recent learning based approaches [5, 51] which either reshape ﬂattened feature maps into 3D grids for subsequent processing or inject pose into the system using fully connected layers.”
Hierarchical Surface Prediction for 3D Object Reconstruction (2017, background): “[32] propose to predict a collection of RGBD (color and depth) images from a single color image and fuse them in a post-processing step.”
Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs (2017, background): “[32] predict RGB images and depth maps for multiple views of an object, and fuse those into a single 3D model.”
3D Shape Induction from 2D Views of Multiple Objects (2016, background): “More recently, convolutional networks have been trained to generate views of 3D objects given their attributes and camera parameters [8], to generate 3D shape given a 2D view of the object [29], and to generate novel views of an object [36].”
Unsupervised Learning of Depth and Ego-Motion from Video (2017, background): “, [45]), warping-based methods are forced to learn intermediate predictions of geometry and/or correspondence.”
Neural 3D Mesh Renderer (2017, background): “Multi-view RGB(D) images, which represent a 3D scene through a set of images, are used for recognition [20, 27] and view synthesis [29].”
Gibson Env: Real-World Perception for Embodied Agents (2018, methodology): “via an encoderdecoder like architecture that directly renders pixels [30, 52, 88] or predicts a flow map for pixels [99].”
Learning 3D Object Categories by Looking Around Them (2017, methodology): “Alternatively, CAD models can be used to train a network to directly predict the 3D shape of an object [10, 41, 38, 7].”
Video Frame Interpolation via Adaptive Separable Convolution (2017, background): “Some of these methods render unseen views from input images for objects like faces and chairs, instead of complex real-world scenes [13, 26, 49, 59].”
Pixels, Voxels, and Views: A Study of Shape Representations for Single View 3D Object Shape Prediction (2018, methodology): “[13, 7]), but more recent works have proposed surface-based representations [18].”
Neural RGB-D Surface Reconstruction (2021, background): “Large synthetic shape datasets [4] enable modeling object categories, where generative models can be used for single view reconstruction [53, 58].”
Photographic Image Synthesis with Cascaded Refinement Networks (2017, background): “Researchers have also studied image inpainting [32], superresolution [3, 17, 19], novel view synthesis [10, 45, 51], and interactive image manipulation [52].”