Shape Inpainting Using 3D Generative Adversarial Network and Recurrent Convolutional Networks

By Weiyue Wang, Qiangui Huang, Suya You, Chao Yang, Ulrich Neumann. 2017

186 citations recorded by the source corpus.

Recent advances in convolutional neural networks have shown promising results in 3D shape completion. But due to GPU memory limitations, these methods can only produce low-resolution outputs. To inpaint 3D models with semantic plausibility and contextual details, we introduce a hybrid framework that combines a 3D Encoder-Decoder Generative Adversarial Network (3D-ED-GAN) and a Longterm Recurrent Convolutional Network (LRCN). The 3DED- GAN is a 3D convolutional neural network trained with a generative adversarial paradigm to fill missing 3D data in low-resolution. LRCN adopts a recurrent neural network architecture to minimize GPU memory usage and incorporates an Encoder-Decoder pair into a Long Shortterm Memory Network. By handling the 3D model as a sequence of 2D slices, LRCN transforms a coarse 3D shape into a more complete and higher resolution volume. While 3D-ED-GAN captures global contextual structure of the 3D shape, LRCN localizes the fine-grained details. Experimental results on both real-world and synthetic data show reconstructions from corrupted models result in complete and high-resolution 3D objects.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Lifelong Generative Modeling (2017, background): “It has wide ranging applications from image denoising [4, 5], inpainting [6, 7], 5”
Deformable Shape Completion with Graph Convolutional Autoencoders (2017, neutral): “Additionally, voxel representations are often memory intensive and suffer from poor resolution [61], although recent models have been proposed to address these issues: implicit surface representation [12], sparse octree networks [57, 44], encoderdecoder CNN for patch-level geometry refinement [19], ”
Contextual-Based Image Inpainting: Infer, Match, and Translate (2017, background): “Using deep neural network for image inpainting has also been studied in [35, 27, 34, 33, 14].”
PU-GAN: A Point Cloud Upsampling Adversarial Network (2019, methodology): “[30] introduced a 3D-ED-GAN for shape completion given a corrupted 3D scan as input.”
3D Object Dense Reconstruction from a Single Depth View (2018, background): “Recent works [11] [64] [23] [65] can infer higher resolution 3D shapes.”
SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation (2017, unknown): “Convolutional neural networks generalize well to 3D by performing convolution on voxels for certain tasks such as object classiﬁcation [32, 46, 27, 49, 39], shape reconstruction [47, 17, 8] of simple objects, and 3D object detection as mentioned in Section 2.1.”
Evolutionary Generative Adversarial Networks (2018, methodology): “GANs have recently been successfully applied to image generation [4, 22, 9, 37], image editing [13, 31, 40, 32], video prediction[29, 8, 30], and many other tasks [38, 28, 17].”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “Wang et al. [28] treat the reconstructed coarse voxel grid as a sequence of images (or slices).”
DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction (2019, methodology): “To represent 3D shapes, many of these methods utilize either voxels [2–9] or point clouds [10] due to ease of encoding them in a neural network.”
PF-Net: Point Fractal Network for 3D Point Cloud Completion (2020, methodology): “For voxel grid-based algorithm, architectures such as 3D-RecGAN [30], 3D-Encoder-Predictor Networks (3D-EPN) [5], and hybrid framework of 3D-ED-GAN and LRCN [27] have been developed to realize the goal of repairing incomplete input data.”
Recurrent Slice Networks for 3D Segmentation of Point Clouds (2018, background): “As a compromise, many researchers transform point clouds into alternative data formats such as voxels [31, 33, 16, 22, 32] and multi-view renderings [22, 30, 19].”
Depth-aware CNN for RGB-D Segmentation (2018, background): “An alternative is to leverage 3D networks [4,11,12] to handle geometry.”
Visual Object Networks: Image Generation with Disentangled 3D Representations (2018, background): “Recent work also explored 3D shape completion from partial scans with deep generative models [Dai et al., 2017, Wang et al., 2017, Wu et al., 2018], including generalization to unseen object categories [Zhang et al., 2018].”