On Face Segmentation, Face Swapping, and Face Perception

By Yuval Nirkin, Iacopo Masi, Anh Tuan Tran, Tal Hassner, Gérard G. Medioni. IEEE International Conference on Automatic Face & Gesture Recognition, 2017

298 citations recorded by the source corpus.

We show that even when face images are unconstrained and arbitrarily paired, face swapping between them is quite simple. To this end, we make the following contributions. (a) Instead of tailoring systems for face segmentation, as others previously proposed, we show that a standard fully convolutional network (FCN) can achieve remarkably fast and accurate segmentations, provided that it is trained on a rich enough example set. For this purpose, we describe novel data collection and generation routines which provide challenging segmented face examples. (b) We use our segmentations for robust face swapping under unprecedented conditions. (c) Unlike previous work, our swapping is robust enough to allow for extensive quantitative tests. To this end, we use the Labeled Faces in the Wild (LFW) benchmark and measure the effect of intra- and inter-subject face swapping on recognition. We show that our intra-subject swapped faces remain as recognizable as their sources, testifying to the effectiveness of our method. In line with established perceptual studies, we show that better face swapping produces less recognizable inter-subject results. This is the first time this effect was quantitatively demonstrated by machine vision systems.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/b723e9b114f1b1d495439b4db50af5723fe97120

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

DeepFakes: a New Threat to Face Recognition? Assessment and Detection (2018, methodology): “Then a facial mask was detected using a CNN-based face segmentation algorithm proposed in [4].”
HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation (2020, background): “overcame limited, scarce segmentation labels by utilizing motion from videos [31].”
Extreme 3D Face Reconstruction: Looking Past Occlusions (2017, methodology): “Speciﬁcally, we use the face segmentation method of [29] to determine occluded regions in the input image I .”
SimSwap: An Efficient Framework For High Fidelity Face Swapping (2020, methodology): “Nirkin et al. [27] utilized a 3D face dataset [29] to transfer the expression and posture and then used Poisson Blending [30] to merge the source face into the target image.”
Transferability and Hardness of Supervised Classification Tasks (2019, background): “It is therefore reasonable that CE could be extended to non-discrete labeled tasks, such as, for faces, 3D reconstruction [58], pose estimation [10, 11] or segmentation [46].”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, methodology): “Until recently, most of the research focused on advances in face-swapping technology, either using a reconstructed 3D morphable model (3DMM) [56, 66], or GANs based model [65, 67].”
Learning an animatable detailed 3D face model from in-the-wild images (2020, methodology): “Tran et al. [2018] gain robustness to occlusions by applying a face segmentation method [Nirkin et al. 2018] to determine occluded regions, and employ an example-based hole filling approach to deal with the occluded regions.”
Two-branch Recurrent Network for Isolating Deepfakes in Videos (2020, background): “Democratized artificial intelligence (AI) made it very easy to produce highly realistic face swaps with a few clicks, giving the ability to non-experts to synthesize content with “Hollywood-like” effects just by simply using off-theshelf applications [47].”
FSGAN: Subject Agnostic Face Swapping and Reenactment (2019, methodology): “Because Nirkin et al. [35] is an image-to-image face swapping method, to be fair in our comparison, for each frame in the target video we select the source frame with the most similar pose.”
The Creation and Detection of Deepfakes (2020, methodology): “Later, fully automated methods were proposed in [34, 80, 163] and [122] using methods such as warping and reconstructed 3D morphable face models.”
Live Face De-Identification in Video (2019, unknown): “Recent contributions have shown a great deal of robustness to the source image, as well as for the properties of the image, from which the target face is taken [19, 34].”
X2Face: A network for controlling face generation by using images, audio, and pose codes (2018, background): “An interesting related set of works consider how to frontalise a face in a still image using a generic reference face [14], transferring expressions of an actor to an avatar [35] and swapping one face with another [20, 24].”
Deepfacelab: Integrated, flexible and extensible face-swapping framework (2020, methodology): “[15] and Face2Face [20]) taken from FaceForensics++ dataset [19].”
Media Forensics and DeepFakes: An Overview (2020, background): “These include content generation [22], [23], image/video synthesis from semantic labels [24], [25] or sketches [26] or text [27], changes of style and attributes [22], [28], [29], domain translation [30], up to expression transfer [25], face swapping typical of deepfakes [31] and talking-head video ”
ManTra-Net: Manipulation Tracing Network for Detection and Localization of Image Forgeries With Anomalous Features (2019, background): “…ManTra-Net is sensitive to subtle manipulations, and robust to postprocessing disguising manipulations, and that it attains good generalizability to unseen data and unknown manipulation types, even for those latest DNN-based manipulations like face swapping [36] and deep image inpainting [32].”
FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping (2019, methodology): “[30], DeepFakes [1] and ours, are presented.”
Exposing Deep Fakes Using Inconsistent Head Poses (2018, background): “Thanks to the recent developments of machine learning, the technologies for manipulating and fabricating images and videos have reached a new level of sophistication [1, 2, 3, 4, 5, 6, 7].”
Extreme 3D Face Reconstruction: Seeing Through Occlusions (2017, methodology): “Specifically, we use an existing face segmentation method [32] to determine occluded regions in the input image I.”