GANimation: Anatomically-aware Facial Animation from a Single Image

By Albert Pumarola, Antonio Agudo, Aleix M. Martinez, Alberto Sanfeliu, Francesc Moreno-Noguer. European Conference on Computer Vision, 2018

644 citations recorded by the source corpus.

Recent advances in Generative Adversarial Networks (GANs) have shown impressive results for task of facial expression synthesis. The most successful architecture is StarGAN [4], that conditions GANs’ generation process with images of a specific domain, namely a set of images of persons sharing the same expression. While effective, this approach can only generate a discrete number of expressions, determined by the content of the dataset. To address this limitation, in this paper, we introduce a novel GAN conditioning scheme based on Action Units (AU) annotations, which describes in a continuous manifold the anatomical facial movements defining a human expression. Our approach allows controlling the magnitude of activation of each AU and combine several of them. Additionally, we propose a fully unsupervised strategy to train the model, that only requires images annotated with their activated AUs, and exploit attention mechanisms that make our network robust to changing backgrounds and lighting conditions. Extensive evaluation show that our approach goes beyond competing conditional generators both in the capability to synthesize a much wider range of expressions ruled by anatomically feasible muscle movements, as in the capacity of dealing with images in the wild.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/80a322fc33c08c9f7a9d4e0b2f417fdb211c76cb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

3D Morphable Face Models—Past, Present, and Future (2019, positive): “Nowadays, many reenactment approaches are based on deep generative models [Kim et al. 2018a; Pumarola et al. 2018].”
Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning (2020, background): “GANs have seen heavy use in face image manipulation [34, 19, 49, 44, 36, 45, 53].”
Style Generator Inversion for Image Enhancement and Animation (2019, neutral): “Recently, success was achieved on the task of face reanimation, mostly by supervised methods which make use of ﬁducial points [2] or action units [24].”
AttentionFGAN: Infrared and Visible Image Fusion Using Attention-Based Generative Adversarial Networks (2021, background): “Therefore, attention mechanism has now played an important role in the applications of natural language processing [38], speech recognition [39] and computer vision [37], [40], [41].”
MixNMatch: Multifactor Disentanglement and Encoding for Conditional Image Generation (2019, background): “Those that do learn disentangled representations focus on specific categories like faces/humans [37, 31, 2, 32, 1, 27] or require clearly defined domains (e.”
ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection (2018, background): “Nowadays, we can generate entirely synthetic faces [9, 24], even at very high-resolutions [22], animate a subject’s face to make it express the desired emotions [5, 31], or modify facial expressions [39, 37, 23].”
A survey on face data augmentation for the training of deep neural networks (2019, background): “[20] and [64] proposed unsupervised strategies for visual attributes transfer and expression transfer.”
Thin-Plate Spline Motion Model for Image Animation (2022, methodology): “Specially, GANimation [22] uses the Facial Action Coding System (FACS) [10] to describe facial expressions.”
Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss (2019, background): “Pumarola et al. [26] generated facial expression conditioned on action units annotations.”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, methodology): “GANimation [130] employed a dual cGAN generator conditioned on emotion action units (AU) to transfer facial expressions.”
Realistic Speech-Driven Facial Animation with GANs (2019, methodology): “A similar approach is used in the GANimation model proposed in [28].”
RelGAN: Multi-Domain Image-to-Image Translation via Relative Attributes (2019, methodology): “Recent methods for facial attribute transfer [22, 15, 4, 7, 30, 23] formulate the problem as unpaired multi-domain image-to-image translation.”
Image-to-Image Translation: Methods and Applications (2021, background): “In addition, GANimation [172] is proposed to generate anatomically aware facial animation.”
Exploiting Visual Artifacts to Expose Deepfakes and Face Manipulations (2019, neutral): “[30] is able”
Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion (2019, background): “Several of these techniques have made it into some of the most compelling photorealistic, controllable generative models of object categories [32, 20].”