Learning Shape Priors for Single-View 3D Completion and Reconstruction

By Jiajun Wu, Chengkai Zhang, Xiuming Zhang, Zhoutong Zhang, William T. Freeman, Josh Tenenbaum. 2018

201 citations recorded by the source corpus.

The problem of single-view 3D shape completion or reconstruction is challenging, because among the many possible shapes that explain an observation, most are implausible and do not correspond to natural objects. Recent research in the field has tackled this problem by exploiting the expressiveness of deep convolutional networks. In fact, there is another level of ambiguity that is often overlooked: among plausible shapes, there are still multiple shapes that fit the 2D image equally well; i.e., the ground truth shape is non-deterministic given a single-view input. Existing fully supervised approaches fail to address this issue, and often produce blurry mean shapes with smooth surfaces but no fine details. In this paper, we propose ShapeHD, pushing the limit of single-view shape completion and reconstruction by integrating deep generative models with adversarially learned shape priors. The learned priors serve as a regularizer, penalizing the model only if its output is unrealistic, not if it deviates from the ground truth. Our design thus overcomes both levels of ambiguity aforementioned. Experiments demonstrate that ShapeHD outperforms state of the art by a large margin in both shape completion and shape reconstruction on multiple real datasets.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, background): “While recent works [74, 76, 79] have applied 3D convolutional neural networks to resolutions up to 128 3 , this is only possible with shallow architectures and small batch sizes, which leads to slow training.”
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation (2019, methodology): “While a few recent shape completion methods were presented [24, 55], we could not find the code to run the comparisons, and their underlying 3D representation is voxel grid which we extensively compare against.”
Learning Implicit Fields for Generative Shape Modeling (2018, background): “There have been a variety of 3D shape representations for deep learning of shapes, such as voxel grids [9, 14, 28, 45, 46], octrees [18, 35, 40, 43, 44], multi-view images [29, 39], point clouds [1, 12, 13, 32, 33, 47, 48], geometry images [37, 38], deformable mesh/patches [16, 38, 42, 47], and part”
Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer (2019, unknown): “Adverserial 3D Object Generation: Generation of 3D shapes through deep learning has been approached using a Generative Adverserial Network (GAN) [5] in a plethora of work [39, 1, 37, 30].”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “To address this issue, Smith and Meger [40] and later Wu et al. [91] used as a training objective the Wasserstein distance normalized with the gradient penalization.”
AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation (2022, unknown): “Several approaches have shown impressive single-view reconstruction results using voxels [10, 14, 31, 38, 39], point clouds [13, 19, 41], meshes [36, 37], and most recently implicit representations of 3D surfaces like SDFs [17,43], UDFs [9] and CSPs [35].”
Deep Stereo Using Adaptive Thin Volume Representation With Uncertainty Awareness (2019, background): “Volumetric based approaches partition the space into a regular 3D volume with millions of small voxels [23, 27, 54, 55, 60, 40], and the network predicts if a voxel is on the surface or not.”
Self-supervised Learning of 3D Objects from Natural Images (2019, background): “Learning object naturalness from data [10, 44] would be a promising direction.”
Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image (2023, background): “Reconstructing various objects from a single image has been well studied [1, 54, 56].”
Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-Shot Metric Depth and Surface Normal Estimation (2024, unknown): “3D reconstruction from a single image: The reconstruction of diverse objects from a singular image has been extensively investigated in prior research [51], [52], [53].”
BSP-Net: Generating Compact Meshes via Binary Space Partitioning (2019, background): “Early approaches generalized 2D convolutions to 3D [6, 14, 24, 48, 49], and employed volumetric grids to represent shapes in terms of coarse occupancy functions, where a voxel evaluates to zero if it is outside and one otherwise.”
Few-Shot Generalization for Single-Image 3D Reconstruction via Priors (2019, methodology): “One approach to using priors is to use an adversary to enforce realistic reconstruction [28, 12].”
Learning to Recover 3D Scene Shape from a Single Image (2020, background): “A number of works have addressed reconstructing different types of objects from a single image [2, 39, 42], such as humans [30, 31], cars, planes, tables, etc.”
TopNet: Structural Point Cloud Decoder (2019, unknown): “5D sketches [35], point clouds [23], etc.”