Towards Large-Pose Face Frontalization in the Wild

By Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, Manmohan Chandraker. IEEE International Conference on Computer Vision, 2017

334 citations recorded by the source corpus.

Despite recent advances in face recognition using deep learning, severe accuracy drops are observed for large pose variations in unconstrained environments. Learning pose-invariant features is one solution, but needs expensively labeled large-scale data and carefully designed feature learning algorithms. In this work, we focus on frontalizing faces in the wild under various head poses, including extreme profile view's. We propose a novel deep 3D Morphable Model (3DMM) conditioned Face Frontalization Generative Adversarial Network (GAN), termed as FF-GAN, to generate neutral head pose face images. Our framework differs from both traditional GANs and 3DMM based modeling. Incorporating 3DMM into the GAN structure provides shape and appearance priors for fast convergence with less training data, while also supporting end-to-end training. The 3DMM-conditioned GAN employs not only the discriminator and generator loss but also a new masked symmetry loss to retain visual quality under occlusions, besides an identity loss to recover high frequency information. Experiments on face recognition, landmark localization and 3D reconstruction consistently show the advantage of our frontalization method on faces in the wild datasets. 1

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/0db8e6eb861ed9a70305c1839eaef34f2c85bbaf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Xiaogang Wang
2. Luc Van Gool
3. Xiaoou Tang
4. Dacheng Tao
5. Ming-Hsuan Yang
6. Chen Change Loy
7. Ping Luo
8. Yi Yang

In Signal Processing

1. Xiaoou Tang
2. Xiaoming Liu
3. Xiaogang Wang
4. Weihong Deng
5. Ping Luo
6. Yujun Shen
7. Xi Yin
8. Mei Wang

Citations in the corpus

• Feature Transfer Learning for Deep Face Recognition with Under-Represented Data (2018, methodology): “ts than state-of-the-art methods [46]. 4.6. Qualitative Results We apply decoder Decin our framework for feature visualization. While skip link between encoder and decoder improves the visual quality [48], we do not apply it to encourage the rich features g to encode intra-class variance. Center vis”
• Generative Adversarial Networks for Face Generation: A Survey (2022, methodology): “Cross-pose GANs [75, 83, 88, 115, 149, 163] were first designed to generate either frontal or profile faces.”
• TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, methodology): “The position map, orientation map and regression map use a focal loss [108], cross-entropy loss, and L 1 loss respectively.”
• Deep Facial Expression Recognition: A Survey (2018, neutral): “, FF-GAN [69], TP-GAN [70]) and DR-GAN [71]).”
• Feature Transfer Learning for Face Recognition With Under-Represented Data (2019, methodology): “While skip link between encoder and decoder improves the visual quality [48], we do not apply it to encourage the rich features g to encode intra-class variance.”
• Interpreting the Latent Space of GANs for Semantic Face Editing (2019, methodology): “To achieve this goal, current methods required carefully designed loss functions [26, 10, 32], introduction of additional attribute labels or features [23, 36, 3, 34, 31], or special architectures [11, 30] to train new models.”
• Deep Face Recognition: A Survey (2018, methodology): “[197] incorporated 3DMM into the GAN structure to provide shape and appearance priors to guide the generator to frontalization.”
• A survey on face data augmentation for the training of deep neural networks (2019, methodology): “TP-GAN [47] and FFGAN [48] are two classic face frontalization methods based on GANs.”
• Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning (2020, methodology): “Many methods have been proposed to incorporate 3D prior knowledge into GAN for face image synthesis [51, 43, 24, 8, 12, 35, 13, 32].”
• FaceID-GAN: Learning a Symmetry Three-Player GAN for Identity-Preserving Face Synthesis (2018, methodology): “1), previous methods [33, 28] are typically built upon the original GAN [9], which is formulated as a two-player game, including a discriminator and a generator, denoted as D and G respectively.”
• Nonlinear 3D Face Morphable Model (2018, methodology): “3DMM has been widely applied in numerous areas, such as computer vision [4, 43], graphics [1], human behavioral analysis [2] and craniofacial surgery [34].”
• A Generative Model of Worldwide Facial Appearance (2019, methodology): “There are many ways [26, 33] to incorporate this metadata and to train conditional GANs.”
• InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs (2020, methodology): “To achieve this goal, existing methods require carefully designed loss functions [33], [34], [35], introduction of additional attribute labels [7], [36], [37], [38], [39], or special architectures [40], [41] to train new models.”
• A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, background): “ix2pix [34], PAN [112], CycleGAN [113], Disco-GAN [114], Dual-GAN [115], StarGAN [116], UNIT [117], MUNIT [118], DRIT [119], DRIT++ [120] Face frontal view generation: DRGAN [121], TPGAN [122], FFGAN [123], FTGAN [125] Audio Speech and audio synthesis: Rank -GAN [150], VAW 151] Music generation: C-R”