FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping

By Lingzhi Li, Jianmin Bao, Hao Yang, Dong Chen, Fang Wen. arXiv.org, 2019

460 citations recorded by the source corpus.

In this work, we propose a novel two-stage framework, called FaceShifter, for high fidelity and occlusion aware face swapping. Unlike many existing face swapping works that leverage only limited information from the target image when synthesizing the swapped face, our framework, in its first stage, generates the swapped face in high-fidelity by exploiting and integrating the target attributes thoroughly and adaptively. We propose a novel attributes encoder for extracting multi-level target face attributes, and a new generator with carefully designed Adaptive Attentional Denormalization (AAD) layers to adaptively integrate the identity and the attributes for face synthesis. To address the challenging facial occlusions, we append a second stage consisting of a novel Heuristic Error Acknowledging Refinement Network (HEAR-Net). It is trained to recover anomaly regions in a self-supervised way without any manual annotations. Extensive experiments on wild faces demonstrate that our face swapping results are not only considerably more perceptually appealing, but also better identity preserving in comparison to other state-of-the-art methods.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/110dc626fcf90a49ec8c7c654533174e2705f734

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

The Creation and Detection of Deepfakes (2020, methodology): “Other methods may be slowed down due to their pre/post-processing steps, such as warping [60, 62, 185], UV mapping or segmentation prediction [23, 112, 123, 180], and the use of refinement networks [25, 60, 83, 93, 111, 153].”
Paint by Example: Exemplar-based Image Editing with Diffusion Models (2022, background): “These models enable various image manipulation tasks [3,22,30,52] previously unassailable, allowing image editing for general images with the guidance of text prompt.”
E2Style: Improve the Efficiency and Effectiveness of StyleGAN Inversion (2021, result): “Compared to traditional geometric transformation based FaceSwap [72] and learning-based FaceShifter [73], our results have higher identity consistency with the source image and better image quality.”
M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection (2021, methodology): “FaceShifter [33] consists of two networks for full pipeline: AEI-Net for face swapping, and HEAR-Net for occlusion handling.”
Deep learning for deepfakes creation and detection: A survey (2019, background): “- Can be applied to any new face pairs without requiring subject specific training [49].”
Disentangling in Latent Space by Harnessing a Pretrained Generator (2020, background): “A popular approach is performing face swap [46, 36, 13, 5], in which the face in the target image is replaced by the one from the source image.”
Face identity disentanglement via latent space mapping (2020, methodology): “This is in contrast to state-of-the-art identity manipulation methods, which manipulate facial features only [Bao et al. 2018; Gafni et al. 2019; Li et al. 2019; Nirkin et al. 2019].”
FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset (2021, background): “Later, FF++ added two more types of deepfake datasets, Deepfake Detection (DFD) [32] and FaceShifter [20] datasets.”
AttentionFGAN: Infrared and Visible Image Fusion Using Attention-Based Generative Adversarial Networks (2021, background): “Therefore, attention mechanism has now played an important role in the applications of natural language processing [38], speech recognition [39] and computer vision [37], [40], [41].”
SimSwap: An Efficient Framework For High Fidelity Face Swapping (2020, methodology): “The recently proposed method FaceShifter [20] was able to produce high fidelity face swapping results.”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, background): “Table 3 An overview of Face-swap based deepfake generation techniques Reference Technique Features Dataset Output Quality Limitations Faceswap [43] Encoder-decoder Facial landmarks Private 256×256 ▪ Blurry results due to lossy compression ▪ Lack of pose, facial expression, gaze direction, hairstyle,”