Effective face frontalization in unconstrained images

By Tal Hassner, Shai Harel, Eran Paz, Roee Enbar. Computer Vision and Pattern Recognition, 2014

358 citations recorded by the source corpus.

“Frontalization” is the process of synthesizing frontal facing views of faces appearing in single unconstrained photos. Recent reports have suggested that this process may substantially boost the performance of face recognition systems. This, by transforming the challenging problem of recognizing faces viewed from unconstrained viewpoints to the easier problem of recognizing faces in constrained, forward facing poses. Previous frontalization methods did this by attempting to approximate 3D facial shapes for each query image. We observe that 3D face shape estimation from unconstrained photos may be a harder problem than frontalization and can potentially introduce facial misalignments. Instead, we explore the simpler approach of using a single, unmodified, 3D surface as an approximation to the shape of all input faces. We show that this leads to a straightforward, efficient and easy to implement method for frontalization. More importantly, it produces aesthetic new frontal views and is surprisingly effective when used for face recognition and gender estimation.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/bb42568a9bc9c35f32c8248f931e953f9d086f76

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Multi-Task Convolutional Neural Network for Pose-Invariant Face Recognition (2017, methodology): “1) Pose-Invariant Face Recognition: According to [11], existing PIFR methods can be classified into four categories including: multi-view subspace learning [2], [28], pose-invariant feature extraction [6], [40], [42], face synthesis [19], [59], [65], and a hybrid approach of the above three [52], [5”
Automatic Analysis of Facial Actions: A Survey (2019, background): “receiving a lot of attention [74], [124], [148], and some of the novel methods might lead to improvements.”
Emotion Recognition in the Wild via Convolutional Neural Networks and Mapped Binary Patterns (2015, methodology): “Used with Support Vector Machines (SVM) classifiers, LBP code histograms have been key to the success of face recognition systems [41, 42, 14].”
Look at Boundary: A Boundary-Aware Face Alignment Algorithm (2018, background): “, face recognition [75], face verification [48, 49] and face frontalisation [21].”
A survey on face data augmentation for the training of deep neural networks (2019, background): “[46] rotated a single, unmodified 3D reference face model for face frontalization.”
Frankenstein: Learning Deep Face Representations Using Small Data (2016, neutral): “Top row: Type I hard boundary artefacts generated by our method (left) and 3D synthesis methods [12], [30], [56] (right).”
Disentangled Representation Learning GAN for Pose-Invariant Face Recognition (2017, methodology): “[11] use a mean 3D face model to generate a frontal face for any subject, which is proved to be effective and efficient.”
Eye In-painting with Exemplar Generative Adversarial Networks (2017, background): “Specifically within this body of work, the applications of GANs [13] to faces are numerous [15, 24, 40].”
Labeled Faces in the Wild: A Survey (2016, methodology): “, 2014 [97] Billion Faces, 2011 [95], DeepFace, 2014 [97], Effective face frontalization, 2014 [40].”
Regressing Robust and Discriminative 3D Morphable Models with a Very Deep Neural Network (2016, background): “It also explains why some recently proposed using coarse, simple 3D shape approximations only as proxies when rendering faces to new views rather than as face representations [13, 15, 25, 26, 39].”
Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation (2017, methodology): “• A reference facial model - an average neutral face that is used as an initialization of optical flow or shape from shading procedures [19, 26].”
Face Alignment in Full Pose Range: A 3D Total Solution (2015, methodology): “Among them RCPR is an occlusion-robust method with the potential to deal with self-occlusion and we train it with landmark visibility computed by 3D information [62].”
Age and gender classification using convolutional neural networks (2015, methodology): “Method Accuracy Best from [10] 77.8 ± 1.3 Best from [23] Table 2.”
Learning Pose-Aware Models for Pose-Invariant Face Recognition in the Wild (2019, methodology): “tion approach [3], [9], which corresponds to using the frontalized PAMout 0 on all the images.”
Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression (2019, background): “Face alignment plays an essential role in many face related applications such as face recognition [55, 43, 39, 70, 11], face frontalization [24, 60, 30] and 3D face reconstruction [16, 52, 37, 21].”
Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis (2017, methodology): “Besides, only the occluded part (right side) of I pred receives the symmetry (a) Proﬁle (b) Ours (c) [33] (d) [41] (e) [8] (f) [44] (g) [12] (h) Frontal loss, i.e. we explicitly pull the right side to be closer to the left.”
Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set (2019, background): “More recently, model ﬁtting using facial landmarks has gained much popularity with the growth of face alignment techniques [4, 58, 21, 3].”
Understanding and Comparing Deep Neural Networks for Age and Gender Classification (2017, methodology): “the face frontalization preprocessing from [17] was omitted and the model was completely trained from scratch, in order to demonstrate the feature learning capabilities of the neural network type classifier.”
On Face Segmentation, Face Swapping, and Face Perception (2017, methodology): “The ﬁrst, inspired by [14] uses a generic 3D face, making no attempt to ﬁt its shape to the face in the image aside from pose (viewpoint) alignment.”