Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion

By Mihir Sahasrabudhe, Zhixin Shu, Edward Bartrum, Riza Alp Güler, Dimitris Samaras, Iasonas Kokkinos. 2019

43 citations recorded by the source corpus.

In this work we introduce Lifting Autoencoders, a generative 3D surface-based model of object categories. We bring together ideas from non-rigid structure from motion, image formation, and morphable models to learn a controllable, geometric model of 3D categories in an entirely unsupervised manner from an unstructured set of images. We exploit the 3D geometric nature of our model and use normal information to disentangle appearance into illumination, shading, and albedo. We further use weak supervision to disentangle the non-rigid shape variability of human faces into identity and expression. We combine the 3D representation with a differentiable renderer to generate RGB images and append an adversarially trained refinement network to obtain sharp, photorealistic image reconstruction results. The learned generative model can be controlled in terms of interpretable geometry and appearance factors, allowing us to perform photorealistic image manipulation of identity, expression, 3D pose, and illumination properties.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Generative Models, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Generative Models

In 3D Vision & Pose

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “One highly related direction is to model and estimate 3D hand joints [5,44,45,53,60,63,66,72,78,79,80,85,3] and mesh [26,27,51,55,75,83,46], object pose [14,28,29,35,43,57,64,71,76,81], or both [6,17,18,19,69].”
Reconstructing Hand-Object Interactions in the Wild (2020, background): “On the other hand, some approaches [24, 28, 53, 57, 45, 39] assume known 3D object model and focus on 6DOF object pose prediction.”
Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild (2019, methodology): “Our method produces much higher quality reconstructions than both methods, with fine details of the facial expression, whereas [45] recovers 3D shapes poorly and [52] generates unnatural shapes.”
Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs (2020, background): “For example, Sahasrabudhe et al. (2019) adopts an autoencoder with a small bottleneck embedding to constrain the deformation ﬁeld, which tend to produce blur and imprecise results.”