Deformable Shape Completion with Graph Convolutional Autoencoders

By Or Litany, A. Bronstein, Michael M. Bronstein, Ameesh Makadia. 2017

241 citations recorded by the source corpus.

The availability of affordable and portable depth sensors has made scanning objects and people simpler than ever. However, dealing with occlusions and missing parts is still a significant challenge. The problem of reconstructing a (possibly non-rigidly moving) 3D object from a single or multiple partial scans has received increasing attention in recent years. In this work, we propose a novel learning-based method for the completion of partial shapes. Unlike the majority of existing approaches, our method focuses on objects that can undergo non-rigid deformations. The core of our method is a variational autoencoder with graph convolutional operations that learns a latent space for complete realistic shapes. At inference, we optimize to find the representation in this latent space that best fits the generated shape to the known partial input. The completed shape exhibits a realistic appearance on the unknown part. We show promising results towards the completion of synthetic and real scans of human body and face meshes exhibiting different styles of articulation and partiality.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Graph Learning, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Graph Learning

In 3D Vision & Pose

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “…applied to a wide variety of graph-structured data including social networks (Hamilton et al., 2017), citation networks (Kipf and Welling, 2017), 3D surfaces (Litany et al., 2018), knowledge base completion tasks (Schlichtkrull et al., 2018), and bio-chemical modeling (Atwood and Towsley, 2016).”
Generating 3D faces using Convolutional Mesh Autoencoders (2018, background): “Litani et al.[29] use graph convolutions for shape completion.”
Dynamic Graph CNN for Learning on Point Clouds (2018, methodology): “p work proposed different local charting techniques using anisotropic diffusion [Boscaini et al. 2016] or Gaussian mixture models [Monti et al. 2017a; Veličković et al. 2017]. In [Halimi et al. 2018; Litany et al. 2017b], a differentiable functional map [Ovsjanikov et al. 2012] layer was incorporate”
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation (2019, background): “Recent 3D vision works [5, 2, 31] often adopt a variational auto-encoder (VAE) learning scheme, in which bottleneck features are perturbed with Gaussian noise to encourage smooth and complete latent spaces.”
Learning Gradient Fields for Shape Generation (2020, neutral): “Common representations emerged for deep generative 3D modeling include voxel-based [23,63], mesh-based [3,48,20,27,35,57], and assembly-based techniques [34,41].”
Dual-Primal Graph Convolutional Networks (2018, methodology): “Such methods, known under the name geometric deep learning [6], have been applied in a variety of applications, from computer graphics, vision [27,5,24,38], and medicine [33,43] to chemistry [12,13] and high energy physics [17].”
Physically Realizable Adversarial Examples for LiDAR Object Detection (2020, background): “Furthermore, meshes have previously demonstrated high-fidelity shape generation results on faces and human bodies [3, 22].”
SAL: Sign Agnostic Learning of Shapes From Raw Data (2019, methodology): “[25] use variational auto-encoder to learn body shape embeddings of a template mesh.”
MeshCNN: a network with an edge (2019, methodology): “Subsequently, feature computation is performed on top of the edges themselves, while leveraging mesh decimation techniques, such as edge collapse, in a manner that adheres to shape geometry and topology.”
PCN: Point Completion Network (2018, background): “One recent work [26] also explores deformable meshes as the shape representation.”
TopNet: Structural Point Cloud Decoder (2019, background): “Although moderate success has been achieved with performing shape completion on regular representations of 3D objects such as distance fields, meshes, and voxel grids [3, 6, 22, 28, 16, 14], these methods fall short due to inefficiency of such representations.”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “The closest topics include depth reconstruction from RGB images, which has been recently addressed using deep learning techniques, see the recent survey of Laga [153], 3D shape completion [26], [28], [45], [103], [156], [160], [161], 3D reconstruction from depth images [103], which can be seen as a ”
ShapeFormer: Transformer-based Shape Completion via Sparse Representation (2022, methodology): “While meshes are more data-efficient, due to the difficulty of handling mesh topology, mesh-based methods have to either use shape template [41, 57, 68], limiting to a single topology, or produce self-intersecting meshes [30].”
Learning to Dress 3D People in Generative Clothing (2019, background): “There is an extensive recent literature on methods and applications of graph convolutions [11, 25, 32, 41, 50].”
Deep Surface Normal Estimation With Hierarchical RGB-D Fusion (2019, background): “, semantic segmentation [27, 5, 23], object detection [11, 20], 3D reconstruction [15, 18, 12], etc.”
Graph convolutional networks: a comprehensive review (2019, background): “propose to combine graph convolutional networks with variational auto-encoder for the shape completion task [94].”
Survey on Emotional Body Gesture Recognition (2018, background): “Richer, more realistic variants can be defined for example as a collection of connected cylinders or spheroids or 3D meshes [37], [38].”