Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis

By Rui Huang, Shu Zhang, Tianyu Li, Ran He. 2017

675 citations recorded by the source corpus.

Photorealistic frontal view synthesis from a single face image has a wide range of applications in the field of face recognition. Although data-driven deep learning methods have been proposed to address this problem by seeking solutions from ample face data, this problem is still challenging because it is intrinsically ill-posed. This paper proposes a Two-Pathway Generative Adversarial Network (TP-GAN) for photorealistic frontal view synthesis by simultaneously perceiving global structures and local details. Four landmark located patch networks are proposed to attend to local textures in addition to the commonly used global encoderdecoder network. Except for the novel architecture, we make this ill-posed problem well constrained by introducing a combination of adversarial loss, symmetry loss and identity preserving loss. The combined loss function leverages both frontal face distribution and pre-trained discriminative deep face models to guide an identity preserving inference of frontal views from profiles. Different from previous deep learning methods that mainly rely on intermediate features for recognition, our method directly leverages the synthesized identity preserving image for downstream tasks like face recognition and attribution estimation. Experimental results demonstrate that our method not only presents compelling perceptual results but also outperforms state-of-theart results on large pose face recognition.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Demystifying MMD GANs (2018, background): “This class of models has recently led to many impressive examples of image generation (e.g. Huang et al., 2017a;b; Jin et al., 2017; Zhu et al., 2017).”
Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View (2019, background): “face frontal view generation [50], generating new human poses [51], photos to emojis [52], photograph editing [53]–[55], face aging [56], [57], photo blending [58], super resolution [59]–[61], photo inpainting [62]–[64], video prediction [65], and 3D object generation [66], [67]; among”
Recent Progress on Generative Adversarial Networks (GANs): A Survey (2019, background): “Huang et al. [56] proposed a Two-Pathway Generative Adversarial Network (TP-GAN) that can consider both global and local information like humans, and can be used to synthesize high-resolution frontal face images from a single side photo.”
Adversarial Scene Editing: Automatic Object Removal from Weak Supervision (2018, background): “Recent works have demonstrated altering facial attributes like hair color, orientation [3], gender [4] and expressions [5] and changing seasons in scenic photographs [6].”
Deep Facial Expression Recognition: A Survey (2018, background): “multaneously localize landmarks and convert facial poses using only frontal faces. Very recently, a series of GAN-based deep models were proposed for frontal view synthesis (e.g., FF-GAN [94], TP-GAN [95]) and DR-GAN [96]) and report promising performances. 3.2 Deep networks for feature learning Dee”
Eye In-painting with Exemplar Generative Adversarial Networks (2017, neutral): “An identity-preserving loss is proposed in [17], which uses a perceptual distance of two faces on the manifold of the DNN outlined in [43] as an objective to be minimized.”
An Introduction to Image Synthesis with Generative Adversarial Nets (2018, background): “Huang et. al propose Two-Pathway Generative Adversarial Network (TP-GAN) [90] that performs such task.”
CoReD: Generalizing Fake Media Detection with Continual Representation using Distillation (2021, methodology): “In particular GAN-based methods are also used to alter skin color [49], generate new viewpoints [24], and change the age of the face [4].”
FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces (2018, methodology): “Generative adversarial networks (GANs) are used to apply Face Aging [18], to generate new viewpoints [19], or to alter face attributes like skin color [20].”
Towards Real-World Blind Face Restoration with Generative Facial Prior (2021, methodology): “We draw inspiration from [32] and apply identity preserving loss in our model.”
Synthetic Data for Deep Learning (2019, background): “[264], who presented a TP-GAN (two-pathway GAN) architecture for frontal view synthesis: given a picture of a face, generate a frontal view picture.”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, background): “[101] TP-GAN § Deep Features § LFW 256x256 § Lack fine details § Lack semantic consistency Zhang et al.”
Deep Face Recognition: A Survey (2018, background): “CVAEGAN [11] combined a variational auto-encoder with a GAN for augmenting data, and took advantages of both statistic and pairwise feature matching to make the training process converge faster and more stably.”
Wasserstein CNN: Learning Invariant Features for NIR-VIS Face Recognition (2017, background): “progress has been made in associated research areas, such as near-infrared (NIR) versus VIS [4], sketch versus VIS [5], 2D versus 3D [6], and different resolutions [7] and poses [8].”