Fast Face-Swap Using Convolutional Neural Networks

By Iryna Korshunova, Wenzhe Shi, Joni Dambre, Lucas Theis. IEEE International Conference on Computer Vision, 2016

454 citations recorded by the source corpus.

We consider the problem of face swapping in images, where an input identity is transformed into a target identity while preserving pose, facial expression and lighting. To perform this mapping, we use convolutional neural networks trained to capture the appearance of the target identity from an unstructured collection of his/her photographs. This approach is enabled by framing the face swapping problem in terms of style transfer, where the goal is to render an image in the style of another one. Building on recent advances in this area, we devise a new loss function that enables the network to produce highly photorealistic results. By combining neural networks with simple pre- and post-processing steps, we aim at making face swap work in real-time with no input from the user.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/fd57b5ae466b43f60f1539159891426b5c7f06bd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Editing in Style: Uncovering the Local Semantics of GANs (2020, unknown): “Previous efforts [18, 21, 22] describe methods for exchanging global properties between a pair of facial images.”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, methodology): “These approaches [56, 63] have the limitation of subject-specific or pair-specific training.”
Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics (2019, background): “Although in recent years there have been many sophisticated algorithms for generating realistic synthetic face videos [9, 13, 46, 51, 26, 47, 37, 20, 23, 10, 21, 50], most of these have not been in mainstream as open-source software tools that anyone can use.”
One Shot Face Swapping on Megapixels (2021, methodology): “[27] model different source identities separately, such as a CageNet for Nicolas Cage, or a SwiftNet for Taylor Swift.”
FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset (2021, methodology): “To generate deepfake videos, we use Faceswap [18], Faceswap GAN (FSGAN) [25], and DeepFaceLab [18] to perform face swaps, and use Wav2Lip for facial reenactment based on source audio.”
SimSwap: An Efficient Framework For High Fidelity Face Swapping (2020, background): “[18] trained a generator and was able to swap faces with one specific identity.”
Mask-Guided Portrait Editing With Conditional GANs (2019, methodology): “With the proposed structure that disentangles and re-combines facial instance embeddings with face masks, our method also enhances over face swapping methods [5, 26] by supporting explicit face and hair swapping.”
Neural Style Transfer: A Review (2017, methodology): “Similarly, for the area of Face Swapping, the idea of MOB-NST algorithm [48] can be directly applied to build a feed-forward Face-Swap algorithm [133].”
Deep Learning for Deepfakes Creation and Detection (2019, background): “The use of deep learning such as CNN and GAN has made swapped face images more challenging for forensics models as it can preserve pose, facial expression and lighting of the photographs [45].”
Exploring Temporal Coherence for More General Video Face Forgery Detection (2021, background): “With the development of deep generative models, especially Generative Adversarial Networks (GANs) [29, 40, 7, 38, 31].”
Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain (2021, background): “Since face manipulation is a classical research topic [44, 34, 33, 22, 18, 43] in computer vision, verifying its authenticity is not a new problem.”
DeepFakes: a New Threat to Face Recognition? Assessment and Detection (2018, methodology): “The ﬁrst approach that used a generative adversarial network to train a model between pre-selected two faces was proposed by Korshunova et al. in 2017 [3].”
The Creation and Detection of Deepfakes (2020, methodology): “In [88], the authors use a modied style transfer with CNN, where the content is xt and the style is the identity of xs .”
Live Face De-Identification in Video (2019, unknown): “Moreover, [23] is limited to stills and was not demonstrated on video.”
On Face Segmentation, Face Swapping, and Face Perception (2017, methodology): “Finally, deep learning was used to transfer faces [22], as if they were styles transfered between images.”
X2Face: A network for controlling face generation by using images, audio, and pose codes (2018, background): “An interesting related set of works consider how to frontalise a face in a still image using a generic reference face [14], transferring expressions of an actor to an avatar [35] and swapping one face with another [20, 24].”
End-to-End Reconstruction-Classification Learning for Face Forgery Detection (2022, background): “The recent years have witnessed the considerable progress of face forgery generation methods [2,4,11,18,20, 41, 42, 50, 53].”