FSGAN: Subject Agnostic Face Swapping and Reenactment

By Yuval Nirkin, Yosi Keller, Tal Hassner. IEEE International Conference on Computer Vision, 2019

729 citations recorded by the source corpus.

We present Face Swapping GAN (FSGAN) for face swapping and reenactment. Unlike previous work, FSGAN is subject agnostic and can be applied to pairs of faces without requiring training on those faces. To this end, we describe a number of technical contributions. We derive a novel recurrent neural network (RNN)-based approach for face reenactment which adjusts for both pose and expression variations and can be applied to a single image or a video sequence. For video sequences, we introduce continuous interpolation of the face views based on reenactment, Delaunay Triangulation, and barycentric coordinates. Occluded face regions are handled by a face completion network. Finally, we use a face blending network for seamless blending of the two faces while preserving target skin color and lighting conditions. This network uses a novel Poisson blending loss which combines Poisson optimization with perceptual loss. We compare our approach to existing state-of-the-art systems and show our results to be both qualitatively and quantitatively superior.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ee4dba65e8d833272d62acf468e7cc880842811c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Deepfake Video Detection Using Convolutional Vision Transformer (2021, methodology): “Face Swapping GAN (FSGAN) [43], and Region-Separative GAN (RSGAN) [39] are used for face swapping, face reenactment, attribute editing, and face part synthesis.”
M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection (2021, methodology): “Synthesis Approaches To guarantee the diversity of synthesized images, we use four facial manipulation methods, including two face swapping methods: FSGAN [42] and FaceShifter [31], and two facial reenactment methods: First-order-motion [50] and IcFace [56].”
Deep learning for deepfakes creation and detection: A survey (2019, background): “- Adjust to both pose and expression variations [51].”
FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping (2019, methodology): “We do not use the face landmark comparison as [12], since face landmarks involve identity information which should be inconsistent between the swapped face and the target face.”
Cross-Domain and Disentangled Face Manipulation With 3D Guidance (2021, background): “some frameworks [22], [23] focus on face swapping between unseen identities, while othermethods [24], [25], [26] achieve”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, methodology): “Real-time face reenactment approaches, such as FSGAN [67], perform both facial replacement and reenactment with occlusion handling.”
The Creation and Detection of Deepfakes (2020, methodology): “L1: xg, xt ′ L1: e I , Ƹ e L1: e , Ƹ e Lpixel−verif: xg, xt , xs LCE De, C : y, ො x Ladv 1: y1, Τ x xg Ladv 2: y2, Τ x xg Ladv I1: yI1, Τ x xg Ladv I2: yI2, Τ x xg 2÷ 2÷ [121] FSGAN: m: segmentation mask (face, hair, other), l: 3D facial landmarks, H: n passes through H while interpolating ls to lt”
SimSwap: An Efficient Framework For High Fidelity Face Swapping (2020, methodology): “Besides, FSGAN is very sensitive to input source image.”
Exploring Temporal Coherence for More General Video Face Forgery Detection (2021, background): “With the development of deep generative models, especially Generative Adversarial Networks (GANs) [29, 40, 7, 38, 31].”
Preserving integrity in online social networks (2020, background): “These advances have unleashed potential for harmful applications like face puppeteering [83, 136], speech manipulation [97], face transfer [115], full body manipulation [23], and other more general media manipulations.”
Multi-attentional Deepfake Detection (2021, background): “Benefiting from the great progress in generative models, deepfake techniques have achieved significant success recently and various face forgery methods [19, 41, 21, 31, 32, 44, 28, 38] have been proposed.”
CelebV-HQ: A Large-Scale Video Facial Attributes Dataset (2022, background): “CelebV-HQ provides the possibility of improving Video Generation/Editing, such as unconditional face generation [24,64,44,45,75,94,73], text-to-video generation [51,85,30,31,3], video facial attributes editing [13,34,70,76,88], face reenactment [87,71,95,83], and face swapping [49,101,22,60,89].”
The DeepFake Detection Challenge Dataset (2020, methodology): “For the DFDC, we generated FSGAN swap videos using generator checkpoints trained on the data described in [20], after performing brief fine-tuning on the reenactment generator for each source identity.”
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (2020, background): “On the other hand, subject-agnostic models [4, 9, 11, 20, 22, 24, 30, 56, 60, 68, 70, 81, 82, 86, 92, 93, 100] only need a single image of the target person, who is not seen during training, to synthesize arbitrary motions.”
Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection (2020, background): “Face manipulation methods in particular have raised considerable concerns due to their ability to alter a person’s identity [38, 30, 34], expression [52, 51, 44], or lip movements [50, 57, 56] to match the face in a given target video.”
FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset (2021, methodology): “To generate deepfake videos of this type, we employ four types of deepfake generation methods, Faceswap, DeepFaceLab, FSGAN for face-swapping, and Wav2Lip for audio-driven facial reenactment, see block (c) in Table 3.”
Motion Representations for Articulated Animation (2021, background): “Such knowledge typically includes landmarks [4, 40, 24, 12], semantic segmentations [22], and parametric 3D models [11, 32, 9, 20, 18].”
Face identity disentanglement via latent space mapping (2020, methodology): “A popular approach is performing face swap [Bao et al. 2018; deepfakes 2019; Li et al. 2019; Nirkin et al. 2019], in which the face in the target image is replaced by the one from the source image.”