Interactive Facial Feature Localization

By Vuong Le, Jonathan Brandt, Zhe L. Lin, Lubomir D. Bourdev, Thomas S. Huang. 2012

1,008 citations recorded by the source corpus.

We address the problem of interactive facial feature localization from a single image. Our goal is to obtain an accurate segmentation of facial features on high-resolution images under a variety of pose, expression, and lighting conditions. Although there has been significant work in facial feature localization, we are addressing a new application area, namely to facilitate intelligent high-quality editing of portraits, that brings requirements not met by existing methods. We propose an improvement to the Active Shape Model that allows for greater independence among the facial components and improves on the appearance fitting step by introducing a Viterbi optimization process that operates along the facial contours. Despite the improvements, we do not expect perfect results in all cases. We therefore introduce an interaction model whereby a user can efficiently guide the algorithm towards a precise solution. We introduce the Helen Facial Feature Dataset consisting of annotated portrait images gathered from Flickr that are more diverse and challenging than currently existing datasets. We present experiments that compare our automatic method to published results, and also a quantitative evaluation of the effectiveness of our interactive method.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, neutral): “We report additional qualitative results on ADE20K dataset [61] and Helen Face dataset [30, 48].”
Deep High-Resolution Representation Learning for Visual Recognition (2019, methodology): “The dataset [121] is a combination of HELEN [80], LFPW [5], AFW [193], XM2VTS and IBUG datasets, where each face has 68 landmarks.”
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation (2020, background): “Additionally, we show multi-modal support for super-resolution via style-mixing on medium-level features and evaluate pSp on several image editing tasks, including image interpolation and local patch editing.”
Zoo-Tuning: Adaptive Transfer from a Zoo of Models (2021, methodology): “The 300W is a combination of HELEN (Le et al., 2012), LFPW (Belhumeur et al.”
FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors (2017, methodology): “Training a basic FSRNet on Helen dataset takes ∼6 hours on 1 Titan X GPU.”
One millisecond face alignment with an ensemble of regression trees (2014, methodology): “Database: Most of the experimental results reported are for the HELEN[12] face database which we found to be the most challenging publicly available dataset.”
Generative Face Completion (2017, methodology): “o learn the whole object in an holistic manner instead of a certain part only. 4.2. Face Parsing Since face images in the CelebA [15] dataset do not have segment labels, we use the Helen face dataset [13] to train a face parsing network for regularization. The Helen dataset consists of 2,330 images ”
Wavelet-SRNet: A Wavelet-Based CNN for Multi-scale Face Super Resolution (2017, methodology): “We evaluate Wavelet-SRNet quantitatively using average PSNR(dB) and SSIM on the two test sets of CelebA and Helen.”
MaskGAN: Towards Diverse and Interactive Facial Image Manipulation (2019, methodology): “Table 5 compares the dataset statistics of CelebAMask-HQ with Helen dataset [21].”
How Far are We from Solving the 2D & 3D Face Alignment Problem? (and a Dataset of 230,000 3D Facial Landmarks) (2017, methodology): “ignment, see for example [8,43,50,41]. Such methods are now considered to have largely “solved” the 2D face alignment problem for faces with controlled pose variation like the ones of LFPW [2], Helen [22] and 300-W [30]. We will keep the main result from these works, namely their performance on the ”
A survey on face detection in the wild: Past, present and future (2015, background): “Finally,otherdatabasesthatcanbeusedfortrainingfacedetectionalgorithms aretheLFPW[199],HELEN[200]andiBUGdatabases[197],sincefaciallandmarkannotationsareprovidedbythe databasecreators.”
Robust Face Landmark Estimation under Occlusion (2013, methodology): “Both [7] ( a) LFPW (b) HELEN (c) LFW Figure 5 .”
OpenFace: An open source facial behavior analysis toolkit (2016, methodology): “We train the CNN on the LFPW [11] and Helen [36] training sets with correct and randomly offset landmark locations.”
Style Aggregated Network for Facial Landmark Detection (2018, background): “This module is motivated by the recent advances on image-to-image translation [19, 71] and style-transfer [14, 15, 56].”
Coarse-to-Fine Auto-Encoder Networks (CFAN) for Real-Time Face Alignment (2014, methodology): “The images from LFPW training set [3], HELEN [18] and AFW [35] are used for training and the images in LFPW test set [3] are used for evaluation.”
The First Facial Landmark Tracking in-the-Wild Challenge: Benchmark and Results (2015, background): “This progress would not be feasible without the efforts made by the scientiﬁc community to design and develop both benchmarks with high-quality landmark annotations [16, 30, 28, 44, 46], as well as rigorous protocols for performance assessment.”
Face Alignment at 3000 FPS via Regressing Local Binary Features (2014, methodology): “Our training set consists of AFW, the training sets of LFPW, and the training sets of Helen, with 3148 images in total.”
Cross-Resolution Face Recognition via Prior-Aided Face Hallucination and Residual Knowledge Distillation (2019, methodology): “Testing Dataset We evaluate performance of RIM on face hallucination and cross-resolution face recognition over Helen [16], LFW [12], CALFW [33] and CPLFW [20] datasets, respectively.”
Mnemonic Descent Method: A Recurrent Process Applied for End-to-End Face Alignment (2016, background): “These annotations are provided for 3 existing in-the-wild datasets (LFPW [7], HELEN [30] and AFW [65]) which were originally annotated using different and incompatible markups.”
Interlinked Convolutional Neural Networks for Face Parsing (2015, methodology): “Experiments on a pixel-by-pixel [5] labeling version of the Helen dataset [8] demonstrate the effectiveness of iCNN compared with existing models.”