FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals

By Umur Aybars Ciftci, Ilke Demir. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019

530 citations recorded by the source corpus.

The recent proliferation of fake portrait videos poses direct threats on society, law, and privacy [1]. Believing the fake video of a politician, distributing fake pornographic content of celebrities, fabricating impersonated fake videos as evidence in courts are just a few real world consequences of deep fakes. We present a novel approach to detect synthetic content in portrait videos, as a preventive solution for the emerging threat of deep fakes. In other words, we introduce a deep fake detector. We observe that detectors blindly utilizing deep learning are not effective in catching fake content, as generative models produce formidably realistic results. Our key assertion follows that biological signals hidden in portrait videos can be used as an implicit descriptor of authenticity, because they are neither spatially nor temporally preserved in fake content. To prove and exploit this assertion, we first engage several signal transformations for the pairwise separation problem, achieving 99.39% accuracy. Second, we utilize those findings to formulate a generalized classifier for fake content, by analyzing proposed signal transformations and corresponding feature sets. Third, we generate novel signal maps and employ a CNN to improve our traditional classifier for detecting synthetic content. Lastly, we release an "in the wild" dataset of fake portrait videos that we collected as a part of our evaluation process. We evaluate FakeCatcher on several datasets, resulting with 96%, 94.65%, 91.50%, and 91.07% accuracies, on Face Forensics [2], Face Forensics++ [3], CelebDF [4], and on our new Deep Fakes Dataset respectively. In addition, our approach produces a significantly superior detection rate against baselines, and does not depend on the source, generator, or properties of the fake content. We also analyze signals from various facial regions, under image distortions, with varying segment durations, from different generators, against unseen datasets, and under several dimensionality reduction techniques.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e4425f0b540acff10e0ecee1a8a362f584ff4d5c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Video Transformer for Deepfake Detection with Incremental Learning (2021, methodology): “A wide variety of detection systems [1, 3, 13, 14, 27, 30, 54] employ CNNs and recurrent networks to detect manipulated media.”
WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection (2020, background): “Ciftci et al. proposed the FakeCatcher [8] for deepfake detection.”
Deepfake Video Detection Using Convolutional Vision Transformer (2021, methodology): “Deepfakes can be used for creation of hyper-realistic Computer Generated Imagery (CGI), Virtual Reality (VR) [7], Augmented Re- ality (AR), Education, Animation, Arts, and Cinema [13].”
Media Forensics and DeepFakes: An Overview (2020, background): “In [163] and in [164] deepfakes are revealed by the lack of variations induced by heart beating, an idea already exploited in [160] for computer generated faces.”
The Creation and Detection of Deepfakes (2020, background): “For example, G can generate the biological signals monitored by Refs [30] and [96] by adding a discriminator that monitors these signals.”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, background): “The temporal artifacts involve detecting variation in a person’s behavior [146, 147], physiological signals [135, 148-150], coherence [151, 152], and video frame synchronization [153-156].”