Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network

By Yao Feng, Fan Wu, Xiaohu Shao, Yanfeng Wang, Xi Zhou. European Conference on Computer Vision, 2018

746 citations recorded by the source corpus.

We propose a straightforward method that simultaneously reconstructs the 3D facial structure and provides dense alignment. To achieve this, we design a 2D representation called UV position map which records the 3D shape of a complete face in UV space, then train a simple Convolutional Neural Network to regress it from a single 2D image. We also integrate a weight mask into the loss function during training to improve the performance of the network. Our method does not rely on any prior face model, and can reconstruct full facial geometry along with semantic meaning. Meanwhile, our network is very light-weighted and spends only 9.8ms to process an image, which is extremely faster than previous works. Experiments on multiple challenging datasets show that our method surpasses other state-of-the-art methods on both reconstruction and alignment tasks by a large margin.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/cdb6b1a316995c42bcb90536d6fe17b81cd21b5f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Multi-Adversarial Discriminative Deep Domain Generalization for Face Presentation Attack Detection (2019, methodology): “We utilize the state-of-the-art dense face alignment network named PRNet [10] to estimate the depth map of real faces, which serves as the supervision for the real faces.”
Learning to Regress 3D Face Shape and Expression From an Image Without 3D Supervision (2019, methodology): “PRNet [9] requires a very tightly cropped face region to give good results and performs poorly when given the loosely cropped input image that comes with the benchmark database (see Sup.”
Searching Central Difference Convolutional Networks for Face Anti-Spoofing (2020, methodology): “Dense face alignment PRNet [18] is adopted to estimate the 3D shape of the living face and generate the facial depth map with size 32 × 32 .”
Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion (2019, neutral): “PRNet [17] (supervised) Y 4.”
Face Anti-Spoofing with Human Material Perception (2020, methodology): “Dense face alignment PRNet [56] is adopted to estimate facial 3D shapes and generate the facial depth maps with size 32 × 32.”
Learning an animatable detailed 3D face model from in-the-wild images (2020, methodology): “We compare DECA with publicly available methods, namely 3DDFA-V2 [Guo et al. 2020], Deng et al. [2019], RingNet [Sanyal et al. 2019], PRNet [Feng et al. 2018b], 3DMM-CNN [Tran et al. 2017] and Ex-treme3D [Tran et al. 2018].”
Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set (2019, methodology): “Many methods resort to synthetic data or using 3D shapes ﬁtted by traditional methods as surrogate labels [37, 57, 45, 31, 14, 18].”
Deepfacelab: Integrated, flexible and extensible face-swapping framework (2020, methodology): “…landmark extraction algorithms to solve this: (a) heatmap-based facial landmark algorithm 2DFAN [2] (for faces with standard pose) and (b) PRNet [6] with 3D face prior information (for faces with large Euler angle (yaw, pitch, roll), e.g., A face with a large yaw angle, means one side of the…”
Deep Spatial Gradient and Temporal Depth Learning for Face Anti-Spoofing (2020, methodology): “Dense face alignment method PRNet [17] is adopted to estimate the 3D shape of the living face and generate the facial depth map D G ∈ R 32 × 32 .”
CASIA-SURF: A Large-Scale Multi-Modal Benchmark for Face Anti-Spoofing (2019, methodology): “After face detection, we apply the PRNet [47] algorithm to perform 3D reconstruction and density alignment on the detected faces.”
3D Morphable Face Models—Past, Present, and Future (2019, neutral): “Currently, many approaches [Feng et al. 2018; Kim et al. 2018b; Klaudiny et al. 2017; McDonagh et al. 2016; Richardson et al. 2016; Sela et al. 2017; Yu et al. 2017] in the research community are trained on synthetic training data, since it is easy to acquire and comes by design with perfect…”
AdaFace: Quality Adaptive Margin for Face Recognition (2022, background): “[30] adopts 3D face reconstruction [7] to rotate faces and trains a facial attribute labeler to generate pseudo labels of training data.”
RetinaFace: Single-stage Dense Face Localisation in the Wild (2019, methodology): “Following [12, 70], we evaluate the accuracy of dense facial landmark localisation on the AFLW2000-3D dataset [75] considering (1) 68 landmarks with the 2D projection coordinates and (2) all landmarks with 3D coordinates.”
Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss (2019, background): “Feng et al. [10] observed that integrating a weighted mask into the loss function during training can improve the performance of the reconstruction network.”
CelebA-Spoof: Large-Scale Face Anti-Spoofing Dataset with Rich Annotations (2020, methodology): “In detail, refer to [16], the ground truth of the depth map of live image is generated by PRNet [11] and the ground truth of the reflection map of the spoof image is generated by the method in [38].”
EMOCA: Emotion Driven Monocular Face Capture and Animation (2022, methodology): “Model-free approaches directly regress 3D meshes [18,20,28,33,40,70,74,79,93,95,97] or voxels [38] from an image, or optimize a Signed Distance Function (SDF) [61] to fit a face image.”
Towards Fast, Accurate and Stable 3D Dense Face Alignment (2020, methodology): “Dense vertices regression methods directly regress the coordinates of all the 3D points (usually more than 20,000) through a fully convolutional network [26,17], achieving the state-of-the-art performance.”