High-Resolution Shape Completion Using Deep Neural Networks for Global Structure and Local Geometry Inference

By Xiaoguang Han, Z. Li, Haibin Huang, Evangelos Kalogerakis, Yizhou Yu. 2017

324 citations recorded by the source corpus.

We propose a data-driven method for recovering missing parts of 3D shapes. Our method is based on a new deep learning architecture consisting of two sub-networks: a global structure inference network and a local geometry refinement network. The global structure inference network incorporates a long short-term memorized context fusion module (LSTM-CF) that infers the global structure of the shape based on multi-view depth information provided as part of the input. It also includes a 3D fully convolutional (3DFCN) module that further enriches the global structure representation according to volumetric information in the input. Under the guidance of the global structure network, the local geometry refinement network takes as input local 3D patches around missing regions, and progressively produces a high-resolution, complete surface through a volumetric encoder-decoder architecture. Our method jointly trains the global structure inference and local geometry refinement networks in an end-to-end manner. We perform qualitative and quantitative evaluations on six object categories, demonstrating that our method outperforms existing state-of-the-art work on shape completion.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

A Papier-Mache Approach to Learning 3D Surface Generation (2018, neutral): “Methods for generating a voxel grid have been demonstrated with various inputs, namely one or several images [6, 9], full 3D objects in the form of voxel grids [9, 33], and 3D objects with missing shape parts [33, 11].”
Deformable Shape Completion with Graph Convolutional Autoencoders (2017, background): “…suffer from poor resolution [61], although recent models have been proposed to address these issues: implicit surface representation [13], sparse octree networks [57, 42], encoder-decoder CNN for patch-level geometry reﬁnement [19], and a long-term recurrent CNN for upsampling coarse shapes [58].”
3D Object Dense Reconstruction from a Single Depth View (2018, methodology): “In terms of the generality, Varley et al. [2] and Han et al. [23] are inferior because [2] uses a single fully connected layers, instead of 3D ConvNets, for shape generation which is unlikely to be general for various shapes, and [23] applies LSTMs for shape blocks generation which is inefficient an”
SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation (2017, background): “Convolutional neural networks generalize well to 3D by performing convolution on voxels for certain tasks such as object classification [34, 48, 27, 51, 41, 29, 30], shape reconstruction [49, 17, 8] of simple objects, and 3D object detection as mentioned in Section 2.”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “[45] require 3D boundary detection while Cao et al.”
EC-Net: an Edge-aware Point set Consolidation Network (2018, methodology): “Other works converted the input triangular meshes or point clouds to regular voxel grids [24,25,26,27,28,29] for CNN to process.”
PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers (2021, methodology): “Traditional methods for 3D shape completion tasks often adopt voxel grids or distance fields to describe 3D objects [6, 15, 31].”
PCN: Point Completion Network (2018, methodology): “While most existing learning-based methods [9, 16, 43, 46, 49, 53, 55, 58] represents shapes using voxels, which are convenient for convolutional neural networks, our method uses point clouds, which preserve complete geometric information about the shapes while being memory efficient.”
Morphing and Sampling Network for Dense Point Cloud Completion (2019, methodology): “Existing learning-based methods represent 3D shapes as volumetric grids (Dai, Ruizhongtai Qi, and Nießner 2017; Han et al. 2017; Stutz and Geiger 2018) or view-based projection (Park et al.”
PF-Net: Point Fractal Network for 3D Point Cloud Completion (2020, background): “Thus, algorithms which can directly conduct shape completion task on unordered point have been significantly developed [15, 10, 28, 33].”
Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion (2020, methodology): “Mean-while, coarse-to-ﬁne strategies (e.g., LSTM based model) are used in (Dai et al. 2018; Han et al. 2017) to recover missing parts of 3D shapes.”
TopNet: Structural Point Cloud Decoder (2019, background): “These works have shown great success in the tasks of 3D Reconstruction [5, 11], shape completion [6, 14, 36, 19], and shape generation from embeddings [3].”
GRNet: Gridding Residual Network for Dense Point Cloud Completion (2020, methodology): “Early works [2,3,33] usually apply 3D convolutional neural networks (CNNs) build upon the volumetric representation of 3D point clouds.”
ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans (2017, background): “Several other works have followed, using 3D convolutional neural networks (CNNs) for object classification [18, 27] or completion [7, 9].”
3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation (2018, background): “In order to address the memory and compute requirements, hierarchical 3D CNNs have been proposed to more eﬃciently represent and process 3D volumes [17,18,19,20,21,16].”
3D-PhysNet: Learning the Intuitive Physics of Non-Rigid Object Deformations (2018, methodology): “Generative networks have been applied with success to the problem of reconstructing 3D objects from partial views or synthesizing 3D objects [Yang et al., 2017; Han et al., 2017].”
SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis (2018, methodology): “Similar distance-field representations are used in recent works on 3D shape recovery [51, 18].”