Two-branch Recurrent Network for Isolating Deepfakes in Videos

By Iacopo Masi, Aditya Killekar, Royston Marian Mascarenhas, Shenoy Pratik Gurudatt, Wael AbdAlmageed. European Conference on Computer Vision, 2020

489 citations recorded by the source corpus.

The current spike of hyper-realistic faces artificially generated using deepfakes calls for media forensics solutions that are tailored to video streams and work reliably with a low false alarm rate at the video level. We present a method for deepfake detection based on a two-branch network structure that isolates digitally manipulated faces by learning to amplify artifacts while suppressing the high-level face content. Unlike current methods that extract spatial frequencies as a preprocessing step, we propose a two-branch structure: one branch propagates the original information, while the other branch suppresses the face content yet amplifies multi-band frequencies using a Laplacian of Gaussian (LoG) as a bottleneck layer. To better isolate manipulated faces, we derive a novel cost function that, unlike regular classification, compresses the variability of natural faces and pushes away the unrealistic facial samples in the feature space. Our two novel components show promising results on the FaceForensics++, Celeb-DF, and Facebook's DFDC preview benchmarks, when compared to prior work. We then offer a full, detailed ablation study of our network architecture and cost function. Finally, although the bar is still high to get very remarkable figures at a very low false alarm rate, our study shows that we can achieve good video-level performance when cross-testing in terms of video-level AUC.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/004c7dbd5578865ac72cfa7b6ebc51c7fa7cda31

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Local Relation Learning for Face Forgery Detection (2021, unknown): “Two-branch RN (Masi et al. 2020) ampliﬁes multi-band frequencies using a Laplacian of Gaussian (LoG) as a bottleneck layer.”
Frequency-Aware Deepfake Detection: Improving Generalizability through Frequency Space Domain Learning (2024, background): “The work by Masi et al. (Masi et al. 2020) stands out as it meticulously investigates artifacts present in both the color space and the frequency domains, while F 3 -Net (Qian et al. 2020) suggests using the discrepancy of frequency statistics between real and forged images as a means to…”
Exploring Temporal Coherence for More General Video Face Forgery Detection (2021, methodology): “Without any specific design, current video face forgery detection methods [21, 5, 36] may rely more on spatial-related artifacts instead of the temporal incoherence for classification.”
Learning on Gradients: Generalized Artifacts Representation for GAN-Generated Images Detection (2023, background): “LOG [32] proposes a two-branch recurrent network to integrate information from the color domain and the frequency domain for detecting manipulated faces.”
The Creation and Detection of Deepfakes (2020, methodology): “To focus on the residuals, the authors of [108] use a two stream ED to encode the color image and a frequency enhanced version using “Laplacian of Gaussian layers” (LoG).”
Self-supervised Learning of Adversarial Example: Towards Good Generalizations for Deepfake Detection (2022, methodology): “Recent studies have made various attempts for deepfake detection and achieve remarkable success [26, 41, 3, 2, 32, 39, 14, 24, 30, 29, 19, 47, 23, 36, 5].”
End-to-End Reconstruction-Classification Learning for Face Forgery Detection (2022, background): “Many efforts have been made to improve the performance of face forgery detection [1,13,23, 29,30,40,45,46,60].”
Deepfake detection using deep learning methods: A systematic and comprehensive review (2023, background): “Deepfake, which refers to various face modification techniques, incorporates cutting-edge technologies, including computer vision and DL (Lv, Chen, et al., 2021; Masi et al., 2020).”
Interpretable and Trustworthy Deepfake Detection via Dynamic Prototypes (2021, methodology): “Recent work using multi-frame inputs [50, 40] and video architecture [48] have shown the competitive potential of leveraging temporal information.”
Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain (2021, methodology): “[28] leverage a Laplacian of Gaussian (LoG) to make frequency enhancement for purpose of suppressing the image content present in the low-level feature maps.”
Detecting Deepfakes with Self-Blended Images (2022, methodology): “We refer to four state-of-the-art methods, including: (1) Two-branch [47] proposes Laplacian of Gaus-”
One-Class Learning Towards Synthetic Voice Spoofing Detection (2020, background): “Recently, the one-class learning idea has been successfully introduced into image forgery detection [24]–[26].”
Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection (2021, methodology): “Although two-branch [29] is a video-based detection method, we still surpass it, especially in the c40 version.”
Multi-attentional Deepfake Detection (2021, methodology): “To mitigate such risks,many deepfake detection approaches [27, 34, 22, 33, 26, 45] have been proposed.”
UCF: Uncovering Common Features for Generalizable Deepfake Detection (2023, methodology): “We further evaluate our method against other state-of-the-art models, including the Two-stream [51], MesoNet [3], Headpose [45], FWA [22], VA-MLP [29], Capsule [31], SMIL [21], Two-branch [28], SPSL [25], MADD [48], SLADD [6], CFFs [46], and Schwarcz et al.”