Shape Completion Using 3D-Encoder-Predictor CNNs and Shape Synthesis

By Angela Dai, Charles Ruizhongtai Qi, Matthias Nießner. 2016

690 citations recorded by the source corpus.

We introduce a data-driven approach to complete partial 3D shapes through a combination of volumetric deep neural networks and 3D shape synthesis. From a partially-scanned input shape, our method first infers a low-resolution – but complete – output. To this end, we introduce a 3D-Encoder-Predictor Network (3D-EPN) which is composed of 3D convolutional layers. The network is trained to predict and fill in missing data, and operates on an implicit surface representation that encodes both known and unknown space. This allows us to predict global structure in unknown areas at high accuracy. We then correlate these intermediary results with 3D geometry from a shape database at test time. In a final pass, we propose a patch-based 3D shape synthesis method that imposes the 3D geometry from these retrieved shapes as constraints on the coarsely-completed mesh. This synthesis process enables us to reconstruct fine-scale detail and generate high-resolution output while respecting the global mesh structure obtained by the 3D-EPN. Although our 3D-EPN outperforms state-of-the-art completion method, the main contribution in our work lies in the combination of a data-driven shape predictor and analytic 3D shape synthesis. In our results, we show extensive evaluations on a newly-introduced shape completion benchmark for both real-world and synthetic data.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

In 3D Vision & Pose

Citations in the corpus

3D Instance Segmentation of MVS Buildings (2021, methodology): “Volumetric methods ease this issue by using a 3-D grid representation, which is notoriously expensive in terms of computational efficiency and memory consumption [4], [12], [40]–[43].”
Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, background): “For achieving sub-voxel precision, several works [12,42,60] have proposed to predict truncated signed distance ﬁelds (TSDF) [11] where each point in a 3D grid stores the truncated signed distance to the closest 3D surface point.”
ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes (2017, neutral): “When training data is synthetic and test is performed on real data, there is also a significant discrepancy of test performance, as data characteristics, such as noise and occlusions patterns, are inherently different.”
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation (2019, methodology): “shape completion networks of [16, 58] require preparing training data of partial shapes.”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, methodology): “We compare with the state of the art: 3D-EPN [8].”
MarrNet: 3D Shape Reconstruction via 2.5D Sketches (2017, methodology): “This idea is also widely used in 3D shape completion from depths or silhouettes [Firman et al., 2016, Rock et al., 2015, Dai et al., 2017].”
OctNetFusion: Learning Depth Fusion from Data (2017, background): “[11] consider the problem of high-resolution 3D shape completion.”
Deformable Shape Completion with Graph Convolutional Autoencoders (2017, methodology): “Additionally, voxel representations are often memory intensive and suffer from poor resolution [61], although recent models have been proposed to address these issues: implicit surface representation [12], sparse octree networks [57, 44], encoderdecoder CNN for patch-level geometry refinement [19], ”
Convolutional Occupancy Networks (2020, background): “S P S R [1 8 ]”
Multi-content GAN for Few-Shot Font Style Transfer (2017, positive): “Similarly, 3D models can be completed from a partial 3D shape [5, 28].”
Shape Inpainting Using 3D Generative Adversarial Network and Recurrent Convolutional Networks (2017, methodology): “[8] introduced a 3D-EncoderPredictor Network to predict and fill missing data for 3D distance field and proposed a 3D synthesis procedure to obtain high-resolution objects.”
Scan2CAD: Learning CAD Model Alignment in RGB-D Scans (2018, background): “Learning-based approaches for object and scene completion [9, 38, 10] cannot reliably recover sharp edges or planar surfaces, resulting in quality far from artist-modeled 3D content.”
VoxFormer: Sparse Voxel Transformer for Camera-Based 3D Semantic Scene Completion (2023, background): “3D object shape completion is an active research topic that estimates the complete geometry from a partial shape in the format of point [23–25], voxels [26–28], and distance fields [29], etc.”
Implicit Geometric Regularization for Learning Shapes (2020, background): “In (Dai et al., 2017), the RGVF models a signed distance function to a shape, where an encoder-decoder network is trained for the task of shape completion.”
Unsupervised 3D Shape Completion through GAN Inversion (2021, background): “Most existing approaches are trained in a fully-supervised manner with partial shapes of a particular form [10, 15, 7, 23, 32, 37], and paired complete shapes.”
3D Shape Perception from Monocular Vision, Touch, and Shape Priors (2018, neutral): “[8] obtained very impressive results on 3D shape completion from partial depth scans by levering 3D convolutional networks and nonparametric patch-based shape synthesis methods.”
EC-Net: an Edge-aware Point set Consolidation Network (2018, methodology): “Other works converted the input triangular meshes or point clouds to regular voxel grids [24,25,26,27,28,29] for CNN to process.”
LRM: Large Reconstruction Model for Single Image to 3D (2023, neutral): “Many previous works apply an encoder-decoder framework to model the image-to-3D data distribution (Choy et al., 2016; Yan et al., 2016; Dai et al., 2017; Xu et al., 2019; Wu et al., 2020; M¨uller et al., 2022; Sajjadi et al., 2022; Goel et al., 2023), where a compact latent code is trained to carry…”
Morphing and Sampling Network for Dense Point Cloud Completion (2019, methodology): “Existing learning-based methods represent 3D shapes as volumetric grids (Dai, Ruizhongtai Qi, and Nießner 2017; Han et al. 2017; Stutz and Geiger 2018) or view-based projection (Park et al. 2017) and then leverage 3D/2D convolution operations.”
Atlas: End-to-End 3D Scene Reconstruction from Posed Images (2020, methodology): “Following [14], we log-transform the predicted and target values before applying the `1 loss, and only backpropagate loss for voxels that were observed in the ground truth (i.”