How Far are We from Solving the 2D & 3D Face Alignment Problem? (and a Dataset of 230,000 3D Facial Landmarks)
By Adrian Bulat, Georgios Tzimiropoulos. 2017
1,678 citations recorded by the source corpus.
This paper investigates how far a very deep neural network is from attaining close to saturating performance on existing 2D and 3D face alignment datasets. To this end, we make the following 5 contributions: (a) we construct, for the first time, a very strong baseline by combining a state-of-the-art architecture for landmark localization with a state-of-the-art residual block, train it on a very large yet synthetically expanded 2D facial landmark dataset and finally evaluate it on all other 2D facial landmark datasets. (b)We create a guided by 2D landmarks network which converts 2D landmark annotations to 3D and unifies all existing datasets, leading to the creation of LS3D-W, the largest and most challenging 3D facial landmark dataset to date (~230,000 images). (c) Following that, we train a neural network for 3D face alignment and evaluate it on the newly introduced LS3D-W. (d) We further look into the effect of all “traditional” factors affecting face alignment performance like large pose, initialization and resolution, and introduce a “new” one, namely the size of the network. (e) We show that both 2D and 3D face alignment networks achieve performance of remarkable accuracy which is probably close to saturating the datasets used. Training and testing code as well as the dataset can be downloaded from https://www.adrianbulat.com/face-alignment/
Subfields: Computer Vision and Pattern Recognition, 3D Vision & Pose, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Ming-Hsuan Yang
- Yi Yang
- Jitendra Malik
- Trevor Darrell
- Chen Change Loy
In 3D Vision & Pose
- Michael J. Black
- Christian Theobalt
- Jingdong Wang
- Stefanos P. Zafeiriou
- Xuanyi Dong
- Timo Bolkart
- Jian Zhao
- Bin Xiao
In Signal Processing
- Ping Luo
- Chen Change Loy
- Stefanos P. Zafeiriou
- Lior Wolf
- Xiaoou Tang
- Xiaogang Wang
- Ziwei Liu
- Georgios Tzimiropoulos
Citations in the corpus
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, neutral): “…outputs of VQ-SEG are m channels, representing the number of classes for all semantic segmentation groups m = m p + m h + m f + 1 , where m p , m h , m f are the number of categories for the panoptic segmentation [63], human segmentation [35], and face segmentation extracted with [5] respectively.”
- Deep High-Resolution Representation Learning for Visual Recognition (2019, methodology): “…process (e.g., VGGNet), with skipping connection over some mirrored layers to transform the pooling indices, e.g., SegNet [3] and DeconvNet [107], or copying the feature maps, e.g., U-Net [119] and Hourglass [8], [9], [27], [31], [68], [105], [134], [163], [165], encoder-decoder [112], and so on.”
- Res2Net: A New Multi-Scale Backbone Architecture (2019, background): “Multi-scale feature representations of CNNs are of great importance to a number of vision tasks including object detection [43], face analysis [4], [41], edge detection [37], semantic segmentation [6], salient object detection [34], [65], and skeleton detection [67], boosting the model performance o”
- High-Fidelity GAN Inversion for Image Attribute Editing (2021, background): “A naive alternative is to directly add the distortion map ∆̃ to X o in the image space with warping estimated by face landmarks (Bulat and Tzimiropoulos 2017; Guo et al. 2020).”
- Talking Face Generation by Adversarially Disentangled Audio-Visual Representation (2018, methodology): “2014) as the backbone for encoder Ep, and for encoder E v w, we modified a simple version of FAN (Bulat and Tzimiropoulos 2017).”
- What comprises a good talking-head video generation?: A Survey and Benchmark (2020, methodology): “Specifically, we extract the 3D facial landmarks from each video frame using a tool proposed in Bulat and Tzimiropoulos (2017). Furthermore, we register the extracted 3D landmark with a canonical 3D landmark by applying rotation and translation.”
- The face of art (2019, background): “Another approach to facial landmark detection takes the advantages of end-to-end training from deep CNN model to learn robust heatmap of facial landmarks [Bulat and Tzimiropoulos 2016, 2017; Yang et al. 2017].”
- Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set (2019, methodology): “To train our R-Net, we collected in-the-wild images from multiple sources such as CelebA [32], 300W-LP [57], I-JBA [30], LFW [24] and LS3D [8].”
- ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variation (2020, methodology): “For each input image sample, we first perform face and facial landmark detection using a state-of-the-art method [5].”
- Modeling Facial Geometry Using Compositional VAEs (2018, background): “However, none of the publicly available face shape datasets [6, 5] offer truly high-resolution models, which would not allow us to fully test descriptive power of our compositional model.”
- Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation (2021, methodology): “On the other hand, 3D tools [2, 19, 5, 29] However, their methods cannot render good quality under the “one-shot” condition.”
- Prior Guided GAN Based Semantic Inpainting (2020, methodology): “This calls for a refinement stage following the estimation by [7].”
- I M Avatar: Implicit Morphable Head Avatars from Videos (2021, methodology): “We calculate foreground masks with MODNet [24] and estimate the initial FLAME parameters using DECA [17], which are refined by fitting to 2D facial keypoints [5].”
- LaFIn: Generative Landmark Guided Face Inpainting (2019, methodology): “For PSNR and SSIM, higher values indicate better performance, while for FID, the lower the better. As the ground-truth landmarks are unavailable for the CelebA-HQ dataset, we apply the results by FAN [3] to perform as the reference information for training our landmark predictor. Result comparison. ”
- Artificial intelligence in the creative industries: a review (2020, methodology): “2016), and for converting 2D face images to 3D face reconstructions using CNN-based encoder-decoder architectures (Bulat and Tzimiropoulos 2017; Jackson et al.”
- DALL-EVAL: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models (2022, methodology): “For automated skin tone detection, we detect faces from images with FAN [8] and estimate illumination and facial albedo with TRUST [20].”
- RetinaFace: Single-stage Dense Face Localisation in the Wild (2019, methodology): “8(a) and 8(b), we give the CED curves of state-of-the-art methods [12, 70, 75, 25, 3] as well as RetinaFace.”
- Learning an animatable detailed 3D face model from in-the-wild images (2020, methodology): “We use FAN [Bulat and Tzimiropoulos 2017] to predict 68 2D landmarks k 𝑖 on each face.”