3D Object Reconstruction from a Single Depth View with Adversarial Learning

By Bo Yang, Hongkai Wen, Sen Wang, R. Clark, Andrew Markham, Agathoniki Trigoni. 2017

208 citations recorded by the source corpus.

In this paper, we propose a novel 3D-RecGAN approach, which reconstructs the complete 3D structure of a given object from a single arbitrary depth view using generative adversarial networks. Unlike the existing work which typically requires multiple views of the same object or class labels to recover the full 3D geometry, the proposed 3D-RecGAN only takes the voxel grid representation of a depth view of the object as input, and is able to generate the complete 3D occupancy grid by filling in the occluded/missing regions. The key idea is to combine the generative capabilities of autoencoders and the conditional Generative Adversarial Networks (GAN) framework, to infer accurate and fine-grained 3D structures of objects in high-dimensional voxel space. Extensive experiments on large synthetic datasets show that the proposed 3D-RecGAN significantly outperforms the state of the art in single view 3D object reconstruction, and is able to reconstruct unseen types of objects. Our code and data are available at: https://github.com/Yang7879/3D-RecGAN.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

PF-Net: Point Fractal Network for 3D Point Cloud Completion (2020, methodology): “For voxel gridbased algorithm, architectures such as 3D-RecGAN [16], 3D-Encoder-Predictor Networks (3D-EPN) [17], and hybrid framework of 3D-ED-GAN and LRCN [18] have been developed to realize the goal of repairing incomplete input data.”
PU-GAN: A Point Cloud Upsampling Adversarial Network (2019, background): “Most of them focus on generating 3D objects either from a probabilistic space [32] or from 2D images [36, 11, 27].”
3D Object Dense Reconstruction from a Single Depth View (2018, background): “A preliminary version of this work has been published in ICCV 2017 workshops [24].”
BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, background): “Object reconstruction Most existing work on reconstructing 3D objects from RGB [21, 46, 53, 75, 78] and RGBD [45, 55, 82] data does so in isolation, without the human involvement or the interaction.”
3D-PhysNet: Learning the Intuitive Physics of Non-Rigid Object Deformations (2018, methodology): “In the cascaded configuration we use 3D-RecGAN [Yang et al., 2017] for shape reconstruction from rotated partial viewpoints.”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “…see the recent survey of Laga [153], 3D shape completion [26], [28], [45], [103], [156], [160], [161], 3D reconstruction from depth images [103], which can be seen as a 3D fusion and completion problem, 3D reconstruction and modelling from hand-drawn 2D sketches [162], [163], novel view…”
Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer (2019, unknown): “Adverserial 3D Object Generation: Generation of 3D shapes through deep learning has been approached using a Generative Adverserial Network (GAN) [5] in a plethora of work [39, 1, 37, 30].”
An Acceleration Framework for High Resolution Image Synthesis (2019, background): “2018), high resolution image generation [2, 6, 7, 14], 3D generation and reconstruction [15, 13, 16].”
Implicit Geometric Regularization for Learning Shapes (2020, background): “One option is to use an indicator function (Choy et al., 2016; Girdhar et al., 2016; Wu et al., 2016; Yan et al., 2016; Tulsiani et al., 2017; Yang et al., 2017).”