Generating 3D faces using Convolutional Mesh Autoencoders

By Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, Michael J. Black. 2018

644 citations recorded by the source corpus.

Learned 3D representations of human faces are useful for computer vision problems such as 3D face tracking and reconstruction from images, as well as graphics applications such as character generation and animation. Traditional models learn a latent representation of a face using linear subspaces or higher-order tensor generalizations. Due to this linearity, they can not capture extreme deformations and non-linear expressions. To address this, we introduce a versatile model that learns a non-linear representation of a face using spectral convolutions on a mesh surface. We introduce mesh sampling operations that enable a hierarchical mesh representation that captures non-linear variations in shape and expression at multiple scales within the model. In a variational setting, our model samples diverse realistic 3D faces from a multivariate Gaussian distribution. Our training data consists of 20,466 meshes of extreme expressions captured over 12 different subjects. Despite limited training data, our trained model outperforms state-of-the-art face models with 50% lower reconstruction error, while using 75% fewer parameters. We show that, replacing the expression space of an existing state-of-the-art face model with our model, achieves a lower reconstruction error. Our data, model and code are available at http://coma.is.tue.mpg.de/.

Subfields: Generative Models, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

In 3D Vision & Pose

Citations in the corpus

Fast Graph Representation Learning with PyTorch Geometric (2019, methodology): “In addition, we provide embedded datasets like MNIST superpixels (Monti et al., 2017), FAUST (Bogo et al., 2014), ModelNet10/40 (Wu et al., 2015), ShapeNet (Chang et al., 2015), COMA (Ranjan et al., 2018), and the PCPNet dataset from Guerrero et al. (2018).”
Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, background): “Moreover, they are only able to generate meshes with simple topology [70], require a reference template from the same object class [33, 41, 57] or cannot guarantee closed surfaces [26].”
Dynamic Graph CNN for Learning on Point Clouds (2018, methodology): “Litany et al. [2017a] introduced the intrinsic VAE for meshes and applied it to shape completion; a similar architecture was used by Ranjan et al. [2018] for 3D face synthesis.”
3D Morphable Face Models—Past, Present, and Future (2019, neutral): “ars 1,519 individuals 3dMD 4DFAB [Cheng et al. 2018] triangle mesh (60k-75k vertices), UV texture map face, neck and ears 180 individuals ×4k-16k frames of dynamic sequences Dimensional Imaging CoMA [Ranjan et al. 2018] triangle mesh (80k-140k vertices), texture images (avg resolution 3, 700 ×3, 200”
Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (2023, background): “Contrarily, from the perspective of computer vision, the task of 3D reconstruction from an unposed image - which encompasses the creation of geometry and textures - remains an unresolved, ill-posed problem, despite decades of exploration and development [88, 59, 69, 25].”
MeshCNN: a network with an edge (2019, background): “In [Ranjan et al. 2018], a fixed pooling operation has been proposed for meshes autoencoders.”
Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild (2019, methodology): “[43] 3D scans 3DMM Face [62] 3DV, I Prior on 3DV, predict from I ShapeNet, Ikea [1] 3DP Prior on 3DP ShapeNet [44] 3DM Prior on 3DM Face”
RetinaFace: Single-stage Dense Face Localisation in the Wild (2019, methodology): “We directly employ the mesh decoder (mesh convolution and mesh up-sampling) from [70, 40], which is a graph convolution method based on fast localised spectral ﬁltering [10].”
Semantic Graph Convolutional Networks for 3D Human Pose Regression (2019, background): “We assume that graph data share the same topological structure, such as human skeletons [10, 26, 61, 67], 3D morphable models [33, 45, 72] and citation networks [50].”
Neural Unsigned Distance Fields for Implicit Function Learning (2020, methodology): “Mesh based methods deform a template [77, 63, 58] but are limited to a single topology or predict vertices and faces directly [29, 18], but do not guarantee surface continuity.”
End-to-End Human Pose and Mesh Reconstruction with Transformers (2020, methodology): “However, due to memory constraints of current hardware, our transformer processes a coarse mesh: (1) We use a coarse template mesh (431 vertices) for positional encoding, and transformer outputs a coarse mesh; (2) We use learnable Multi-Layer Perceptrons (MLPs) to upsample the coarse mesh to the ori”
Learning to Reconstruct People in Clothing From a Single RGB Camera (2019, methodology): “In order to predict per-vertex offsets from the latent code l, we use a four-step graph convolutional network with Chebyshev filters and mesh upsampling layers similar to [59].”
Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose (2020, background): “Ranjan et al. [51] also proposed a GraphCNN-based VAE, which learns a latent space of the human face meshes in a hierarchical manner.”
3D Hand Shape and Pose Estimation From a Single RGB Image (2019, methodology): “Different from these methods, we propose to estimate 3D mesh vertices using Graph CNNs in order to learn nonlinear hand shape variations and better utilize the relationship among vertices in the mesh topology.”
PolyGen: An Autoregressive Generative Model of 3D Meshes (2020, background): “Such works primarily use parameterized deformable meshes (Groueix et al., 2018; Ranjan et al., 2018), or form meshes through a collection of mesh patches.”
Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion (2020, background): “Meshes for rigid objects: Most mesh-based methods predict shape as a deformation from a template [72, 56] and hence are limited to a single topology.”
Learning to Dress 3D People in Generative Clothing (2019, positive): “[41] learn a convolutional autoencoder using graph convolutions [11] with mesh down- and up-sampling layers [13].”
Mesh Graphormer (2021, background): “Among the previous studies, the Graph Convolutional Neural Network (GCNN) [17, 5, 20] is one of the most popular options as it is able to model the local interactions between neighboring vertices based on a given adjacency matrix [32, 5, 20, 38].”
Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild (2020, methodology): “Spiral refers to the proposed approach, Spectral follows the CoMA decoder [43] implemented as in Kulon et al.”