Face Alignment in Full Pose Range: A 3D Total Solution

By Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, S. Li. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015

785 citations recorded by the source corpus.

Face alignment, which fits a face model to an image and extracts the semantic meanings of facial pixels, has been an important topic in the computer vision community. However, most algorithms are designed for faces in small to medium poses (yaw angle is smaller than 45 degree), which lack the ability to align faces in large poses up to 90 degree. The challenges are three-fold. First, the commonly used landmark face model assumes that all the landmarks are visible and is therefore not suitable for large poses. Second, the face appearance varies more drastically across large poses, from the frontal view to the profile view. Third, labelling landmarks in large poses is extremely challenging since the invisible landmarks have to be guessed. In this paper, we propose to tackle these three challenges in an new alignment framework termed 3D Dense Face Alignment (3DDFA), in which a dense 3D Morphable Model (3DMM) is fitted to the image via Cascaded Convolutional Neural Networks. We also utilize 3D information to synthesize face images in profile views to provide abundant samples for training. Experiments on the challenging AFLW database show that the proposed approach achieves significant improvements over the state-of-the-art methods.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/f79332fce3c3cc641c50d777454b1dd4debd3c5a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Multilinear Autoencoder for 3D Face Model Learning (2018, methodology): “Recent works use CNN frameworks to recover detailed 3D models from a single input photograph [30, 20, 21, 25].”
End-to-End 3D Face Reconstruction with Deep Neural Networks (2017, methodology): “Similar to [33], they use both an RGB image and the 3D rendering of an initial 3D face as input to the network.”
EMOCA: Emotion Driven Monocular Face Capture and Animation (2022, background): “The latter are trained fully supervised [15, 34, 42, 67, 87, 88, 98] or self-supervised with predicted 2D keypoints [19, 27, 52, 73, 76, 80, 82, 83, 96], 2D face contours [52], photometric constraints [19, 27, 30, 76, 80, 82, 83, 96], face recognition features [19, 27, 30, 76], multi-view constraint”
Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion (2019, methodology): “Hence, for the AFLW2000-3D dataset, we report localization accuracies only for 19 landmarks.”
Learning Detailed Face Reconstruction from a Single Image (2016, background): “For instance, in [47] the Projected Normalized Coordinate Code (PNCC) was introduced.”
Articulation-Aware Canonical Surface Mapping (2020, background): “Several previous approaches similarly pursue pixel to surface [2, 23, 24, 28, 42] or volume [35] mappings, but unlike our approach, crucially rely on direct supervision towards this end.”
A survey on face data augmentation for the training of deep neural networks (2019, methodology): “Meanwhile many works used 3D face models for face pose translation [6, 9, 34–37].”
Deep Single-Image Portrait Relighting (2019, methodology): “We use 3DDFA [34] (Code provided by [11]) since it outputs the shape parameters of a 3DMM, which can be used to generate portrait normal images at arbitrary resolution.”
3D Face Morphable Models "In-the-Wild" (2017, methodology): “3 using the 3D shape fits provided by [44].”
Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs (2020, methodology): “We use ResNet-50 [16] pre-trained on CelebAHQ for attribute manipulation, HopeNet [24] pre-trained on 300W-LP [35] for head pose manipulation, and MTCNN [31] to crop faces and detect landmarks for landmark editing.”
FaceID-GAN: Learning a Symmetry Three-Player GAN for Identity-Preserving Face Synthesis (2018, methodology): “Similar to [37], P is trained to minimize the weighted distance function”
Cross-Domain and Disentangled Face Manipulation With 3D Guidance (2021, methodology): “Specifically, we adopt the weighted parameter distance cost (WPDC) loss [54]”
Fake it till you make it: face analysis in the wild using synthetic data alone (2021, methodology): “[79] fit a 3DMM to face images, and warped them to augment the head pose.”
Towards Fast, Accurate and Stable 3D Dense Face Alignment (2020, methodology): “Face profiling [59] F (·), which is originally proposed to solving large-pose face alignment, is utilized to progressively increase the yaw angle ∆φ and pitch angle ∆γ of the face.”
GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction (2019, methodology): “For many years 3DMMs and its variants were the methods of choice for 3D face reconstruction [31, 42, 21].”
Learning to Regress 3D Face Shape and Expression From an Image Without 3D Supervision (2019, methodology): “Learning with 3D supervision: Deep learning methods are quickly replacing the optimization-based approaches [35, 39, 19, 16].”
Realistic Speech-Driven Facial Animation with GANs (2019, methodology): “To do this we use pose estimation software [18] based on the model proposed in [46] to select faces whose roll, pitch and yaw angles are smaller 10°.”
Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression (2017, methodology): “• Because of this fundamental difference, our method is also radically different in terms of the CNN architecture used: we used one that is able to make spatial predictions at a voxel level, as opposed to the networks of [28, 9] which holistically predict the 3DMM parameters.”