Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance

By Zhixin Shu, Mihir Sahasrabudhe, Riza Alp Güler, Dimitris Samaras, Nikos Paragios, Iasonas Kokkinos. 2018

211 citations recorded by the source corpus.

In this work we introduce Deforming Autoencoders, a generative model for images that disentangles shape from appearance in an unsupervised manner. As in the deformable template paradigm, shape is represented as a deformation between a canonical coordinate system (`template') and an observed image, while appearance is modeled in `canonical', template, coordinates, thus discarding variability due to deformations. We introduce novel techniques that allow this approach to be deployed in the setting of autoencoders and show that this method can be used for unsupervised group-wise image alignment. We show experiments with expression morphing in humans, hands, and digits, face manipulation, such as shape and appearance interpolation, as well as unsupervised landmark localization. A more powerful form of unsupervised disentangling becomes possible in template coordinates, allowing us to successfully decompose face images into shading and albedo, and further manipulate face images.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Unsupervised Learning of Object Keypoints for Perception and Control (2019, background): “Other works similarly either require known transformations or output dense correspondences instead of discrete landmarks [35, 29, 33, 37].”
Few-Shot Adversarial Learning of Realistic Neural Talking Head Models (2019, background): “Both classical warping algorithms [5, 28] and warping fields synthesized using machine learning (including deep learning) [11, 29, 40] can be used for such purposes.”
Unsupervised Learning of Object Landmarks through Conditional Image Generation (2018, background): “[38] learn to factor a single object-category-specific image into an appearance template in a canonical coordinate system, and a deformation field which warps the template to reconstruct the input, as in an auto-encoder.”
A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, background): “al [137] and Sample-Level GAN (S-GAN) [138] for object tracking; Semi-Latent GAN (SLGAN) [139], Semantically Decomposing GAN (SD-GAN) [140], Wav2Pix [141], GAN-Fit [142], Deforming Autoencoders (DAE) [143], Dual Variational Generator (DVG) [144] and 3D Aided Duet GAN (3D-AD-GAN) [145] for face image”
Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild (2019, background): “DAE [48] learns to predict a deformation field through heavily constraining an autoencoder with a small bottleneck embedding and lift that to 3D in [45] — in post processing, they further decompose the reconstruction in albedo and shading, obtaining an output similar to ours.”
Contrastive Learning for Diverse Disentangled Foreground Generation (2022, background): “A large number of work try to disentangle object shape/structure from appearance [41,27,39,7,51].”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, background): “Face generation [19], [227], [228], [229], [230], [231], [232], [233] is relatively easy because the problem includes only one class of objects.”
Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion (2019, methodology): “Deforming Autoencoders, introduced in [58], and shown in Fig.”
BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images (2020, background): “This can be achieved with explicit 3D representations, like appearance ﬂow [58], occupancy voxel grids [43, 59], meshes, or shape templates [27, 46, 56], in conjunction with handcrafted differentiable renderers [8, 17, 31, 33].”
Neural volumes (2019, methodology): “In 2D, similar techniques have been used to learn warps that align images [Dai et al. 2017; Jaderberg et al. 2015; Shu et al. 2018].”
A survey on face data augmentation for the training of deep neural networks (2019, methodology): “[148] disentangled the texture and deformation of the input images by adopting the Intrinsic DAE (Deforming-Autoencoder) model [149].”
MixNMatch: Multifactor Disentanglement and Encoding for Conditional Image Generation (2019, methodology): “Moreover, unlike most unsupervised methods which can disentangle only two factors like shape and appearance [24, 35, 26], MixNMatch can disentangle four (background, object shape, pose, and texture).”