paGAN

By Koki Nagano, Jaewoo Seo, Jun Xing, Lingyu Wei, Zimo Li, Shunsuke Saito, Aviral Agarwal, Jens Fursund, Hao Li. 2018

167 citations recorded by the source corpus.

With the rising interest in personalized VR and gaming experiences comes the need to create high quality 3D avatars that are both low-cost and variegated. Due to this, building dynamic avatars from a single unconstrained input image is becoming a popular application. While previous techniques that attempt this require multiple input images or rely on transferring dynamic facial appearance from a source actor, we are able to do so using only one 2D input image without any form of transfer from a source image. We achieve this using a new conditional Generative Adversarial Network design that allows fine-scale manipulation of any facial input image into a new expression while preserving its identity. Our photoreal avatar GAN (paGAN) can also synthesize the unseen mouth interior and control the eye-gaze direction of the output, as well as produce the final image from a novel viewpoint. The method is even capable of generating fully-controllable temporally stable video sequences, despite not using temporal information during training. After training, we can use our network to produce dynamic image-based avatars that are controllable on mobile devices in real time. To do this, we compute a fixed set of output images that correspond to key blendshapes, from which we extract textures in UV space. Using a subject's expression blendshapes at run-time, we can linearly blend these key textures together to achieve the desired appearance. Furthermore, we can use the mouth interior and eye textures produced by our network to synthesize on-the-fly avatar animations for those regions. Our work produces state-of-the-art quality image and video synthesis, and is the first to our knowledge that is able to generate a dynamically textured avatar with a mouth interior, all from a single image.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Rewriting a Deep Generative Model (2020, background): “Recently, thanks to rapid advances of deep generative models [26,45,31], learning-based image synthesis and editing methods have become widely-used tools in the community, enabling applications such as manipulating the semantics of an input scene [60,6,72,16], image colorization [83,34,52,85], photo”
Deferred neural rendering (2019, background): “We do not have to store multiple textures for different expressions like in paGAN [Nagano et al. 2018].”
Pixel-aligned Volumetric Avatars (2021, background): “Deep Appearance Models [12, 17] create a 3D morphable model using deep networks to create an extremely high-quality and driveable face model.”
FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping (2019, methodology): “[26] needs only one image to reenact the portrait within.”
Few-Shot Adversarial Learning of Realistic Neural Talking Head Models (2019, background): “A huge body of works is devoted to statistical modeling of the appearance of human faces [6], with remarkably good results obtained both with classical techniques [35] and, more recently, with deep learning [22, 25] (to name just a few).”
MaskGAN: Towards Diverse and Interactive Facial Image Manipulation (2019, background): “For 3D-based face manipulation, though 3D-based methods [1, 29, 6] achieve promising results on normal poses, they are often computationally expensive and their performance may degrade with large and extreme poses.”
Motion Representations for Articulated Animation (2021, background): “Such knowledge typically includes landmarks [4, 40, 24, 12], semantic segmentations [22], and parametric 3D models [11, 32, 9, 20, 18].”
FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals (2019, background): “Although the widespread use of GANs is positively impacting some technologies (i.e., personalized avatars [40], animations [44], and image inpainting [37]), there are also uses of GANs with malicious intent, which impacts the society by introducing inauthentic content (i.e., celebrity porn [3],…”
Artificial intelligence in the creative industries: a review (2020, methodology): “ion Technology CNN GAN RNN other Creation Content generation [16{18] [19{32] [25,28], TM [33], RL [34{37], for text, audio, video [38{42] BERT [43], VAE [44] and game [30] NEAT [45] Animation [46{50] [51,52] [53] VAE [52] AR/VR [54,55] Deepfakes [56] [57] VAE [58] Content and captions [59] [60{62] [”
Creating, Using, Misusing, and Detecting Deep Fakes (2022, background): “In this type of deep fake, the head movements and facial expressions of one person (the puppet master) are transferred, in real time, to another person (the puppet) (Justus Thies et al. 2016; Nagano et al. 2018; Kim et al. 2018; J. Thies et al. 2018; Siarohin et al. 2019).”
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (2020, methodology): “More recent 3D model-based methods are able to perform subject-agnostic face synthesis [19, 20, 55, 57].”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, background): “5 Deepfakes detection techniques The evolution of ML and the emergence of advanced artificial intelligence algorithms have increased the ease with which fake multimedia content is producing images, audio, or videos, and has improved the realism of manipulated information dramatically [8, 86, 130-132”
FaceScape: A Large-Scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction (2020, methodology): “Inspired by[33], we synthesize displacement map F for an arbitrary expression corresponding to speciﬁc blend-shape weight α , using a weighted linear combination of generated displacement maps ˆ F 0 in neutral expression and ˆ F i in other 19 key expressions: where M is the weight mask with the…”
The Creation and Detection of Deepfakes (2020, methodology): “, [83, 93, 103, 113, 114, 121, 182, 191]).”
Semantic photo manipulation with a generative image prior (2019, methodology): “l. 2018]. GANs have enabled several userguided image synthesis tasks such as generating images from user sketches [Isola et al. 2017; Sangkloy et al. 2017], creating face animation [Geng et al. 2018; Nagano et al. 2018], synthesizing photos from language description [Zhang et al. 2017a], inpainting ”