Every Smile is Unique: Landmark-Guided Diverse Smile Generation

By Wei Wang, Xavier Alameda-Pineda, Dan Xu, Elisa Ricci, Nicu Sebe. 2018

71 citations recorded by the source corpus.

Each smile is unique: one person surely smiles in different ways (e.g. closing/opening the eyes or mouth). Given one input image of a neutral face, can we generate multiple smile videos with distinctive characteristics? To tackle this one-to-many video generation problem, we propose a novel deep learning architecture named Conditional Multi-Mode Network (CMM-Net). To better encode the dynamics of facial expressions, CMM-Net explicitly exploits facial landmarks for generating smile sequences. Specifically, a variational auto-encoder is used to learn a facial landmark embedding. This single embedding is then exploited by a conditional recurrent network which generates a landmark embedding sequence conditioned on a specific expression (e.g. spontaneous smile). Next, the generated landmark embeddings are fed into a multi-mode recurrent landmark generator, producing a set of landmark sequences still associated to the given smile class but clearly distinct from each other. Finally, these landmark sequences are translated into face videos. Our experimental results demonstrate the effectiveness of our CMM-Net in generating realistic videos of multiple smile expressions.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Diversity in Faces (2019, methodology): “The gender annotations used two class labels (male and female) and the age group labeling used seven classes ([0-3],[4-12],[13-19],[20-30],[31-45],[46-60],[61-]), as well as a continuous age value to be consistent with the automatic prediction labels.”
Geometry-Contrastive GAN for Facial Expression Transfer. (2018, background): “Some researchers have attempted to incorporate continuous information such as geometry into cGANs [2, 13, 17, 22, 27].”
First Order Motion Model for Image Animation (2020, methodology): “[38], we use 1116 videos for training and 124 for evaluation.”
StoryGAN: A Sequential Conditional GAN for Story Visualization (2018, unknown): “Trajectory, skeleton or simple landmarks are used in existing works to help model the motion feature [12, 37, 33].”
Motion Representations for Articulated Animation (2021, methodology): “Further works used neural networks to get higher quality rendering [16, 38], sometimes requiring multiple images per identity [11, 23].”