Synthesizing 3D Shapes via Modeling Multi-view Depth Maps and Silhouettes with Deep Generative Networks

By Amir Arsalan Soltani, Haibin Huang, Jiajun Wu, Tejas D. Kulkarni, Josh Tenenbaum. 2017

207 citations recorded by the source corpus.

We study the problem of learning generative models of 3D shapes. Voxels or 3D parts have been widely used as the underlying representations to build complex 3D shapes, however, voxel-based representations suffer from high memory requirements, and parts-based models require a large collection of cached or richly parametrized parts. We take an alternative approach: learning a generative model over multi-view depth maps or their corresponding silhouettes, and using a deterministic rendering function to produce 3D shapes from these images. A multi-view representation of shapes enables generation of 3D models with fine details, as 2D depth maps and silhouettes can be modeled at a much higher resolution than 3D voxels. Moreover, our approach naturally brings the ability to recover the underlying 3D representation from depth maps of one or a few viewpoints. Experiments show that our framework can generate 3D shapes with variations and details. We also demonstrate that our model has out-of-sample generalization power for real-world tasks with occluded objects.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

In 3D Vision & Pose

Citations in the corpus

Learning to Reconstruct Shapes from Unseen Classes (2018, methodology): “Another way of representing 3D shapes is to use a set of multi-view depth images [Soltani et al., 2017, Shin et al., 2018, Jayaraman et al., 2018].”
Learning Implicit Fields for Generative Shape Modeling (2018, background): “f visual quality. 2. Related work There have been a variety of 3D shape representations for deep learning of shapes, such as voxel grids [10,15,30, 48,49], octrees [19,38,43,46,47], multi-view images [4, 31,42], point clouds [1,13,14,35,36,50,51], geometry images [40,41], deformable mesh/patches [17”
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, background): “Using deep convolutional networks, researchers have made significant progress in recent years [9, 17, 21, 29, 41, 43, 56, 59, 60, 62, 66, 52, 61].”
Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018, neutral): “Multi-View Representation Our work connects to multi-view representations which capture the characteristics of a 3D object from multiple viewpoints in 2D [17, 26, 43, 32, 12, 39, 34], such as decomposing image silhouettes [23, 42], Light Field Descriptors [2], and 2D panoramic mapping [38].”
MarrNet: 3D Shape Reconstruction via 2.5D Sketches (2017, methodology): “…like ShapeNet [Chang et al., 2015], researchers developed models encoding shape prior for this task [Girdhar et al., 2016, Choy et al., 2016, Tulsiani et al., 2017, Wu et al., 2016b, Kar et al., 2015, Kanazawa et al., 2016, Soltani et al., 2017], with extension to scenes [Song et al., 2017].”
HyperCube: Implicit Field Representations of Voxelized 3D Models (2021, background): “3D objects can be represented using different approaches including voxel grids [4, 5, 6, 7], octrees [8, 9, 10, 11], multi-view images [12, 13, 14], point clouds [15, 16, 17, 18, 19], geometry images [20, 21], deformable meshes [5, 21, 22, 19], and part-based structural graphs [23, 24].”
Pixels, Voxels, and Views: A Study of Shape Representations for Single View 3D Object Shape Prediction (2018, background): “Our work is related to recent studies [18, 9, 23, 24, 22, 14, 10] that generate multiview projections of 3D objects.”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “This architecture has been used to learn continuous latent spaces for volumetric [17], [18], depth-based [19], surfacebased [20], and point-based [21], [22] 3D reconstruction.”
3D-Aware Scene Manipulation via Inverse Graphics (2018, background): “Many recent papers have explored the problem of 3D reconstruction from a single color image, depth map, or silhouette [Choy et al., 2016, Kar et al., 2015, Tatarchenko et al., 2016, Tulsiani et al., 2017, Wu et al., 2017b, 2016b, Yan et al., 2016b, Soltani et al., 2017].”
What Do Single-View 3D Reconstruction Networks Learn? (2019, methodology): “Multiple methods [27, 30, 43, 45] produced multiview depth maps that are fused together into an output point cloud.”
Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers (2018, methodology): “The final fusion of views has been addressed by merging them into a point cloud and pruning outliers using the predicted silhouettes [25], registering views and solving an optimization problem [18], as well as learning a differentiable depth map renderer to produce consistent projections [17].”
Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images Using a View-Based Representation (2020, background): “…has been studied extensively in the computer vision andgraphics communities (Saxena et al. 2009; Chaudhuri et al. 2011; Kalogerakis et al. 2012; Chang et al. 2015; Rezende et al. 2016; Soltani et al. 2017; Kulkarni et al. 2015; Tulsiani et al. 2016; Huang et al. 2019; Jiang et al. 2019).”
3D Object Reconstruction from a Single Depth View with Adversarial Learning (2017, background): “Recently, [12] [42] [57] [54] [8] [6] [43] [49] [31] leverage deep neural nets to learn a 3D shape from multiple images.”
Learning Object-Centric Representations of Multi-Object Scenes from Multiple Views (2021, background): “In addition, several works have sought explicit 3D representations either in the latent space [21] or output space [25, 2].”
General-Purpose Deep Point Cloud Feature Extractor (2018, unknown): “[2] - 82.”
Multiresolution Tree Networks for 3D Point Cloud Processing (2018, methodology): “These include multiview representations that render a shape from a collection of views [31,39,37] or voxel-based representations [44,19,30,4,43] that discretize point occupancy onto a 3D grid.”
3D Shape Perception from Monocular Vision, Touch, and Shape Priors (2018, unknown): “With large-scale shape repositories like ShapeNet [12], researchers have made signiﬁcant progress on data-driven approaches for shape synthesis, completion, and reconstruction [13], [14], [15], [16], [17].”
3D Object Dense Reconstruction from a Single Depth View (2018, background): “Recently, [69] [70] [71] [21] applied generative networks for 3D structure generation.”
Multi-chart generative surface modeling (2018, background): “In a different variation, [30] learn a model that takes a depth map or a silhouette of a shape and generates multiview depth maps and corresponding silhouettes.”