ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection

By Davide Cozzolino, Justus Thies, Andreas Rössler, Christian Riess, Matthias Nießner, Luisa Verdoliva. 2018

321 citations recorded by the source corpus.

Distinguishing manipulated from real images is becoming increasingly difficult as new sophisticated image forgery approaches come out by the day. Naive classification approaches based on Convolutional Neural Networks (CNNs) show excellent performance in detecting image manipulations when they are trained on a specific forgery method. However, on examples from unseen manipulation approaches, their performance drops significantly. To address this limitation in transferability, we introduce Forensic-Transfer (FT). We devise a learning-based forensic detector which adapts well to new domains, i.e., novel manipulation methods and can handle scenarios where only a handful of fake examples are available during training. To this end, we learn a forensic embedding based on a novel autoencoder-based architecture that can be used to distinguish between real and fake imagery. The learned embedding acts as a form of anomaly detector; namely, an image manipulated from an unseen method will be detected as fake provided it maps sufficiently far away from the cluster of real images. Comparing to prior works, FT shows significant improvements in transferability, which we demonstrate in a series of experiments on cutting-edge benchmarks. For instance, on unseen examples, we achieve up to 85% in terms of accuracy, and with only a handful of seen examples, our performance already reaches around 95%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

CNN-Generated Images Are Surprisingly Easy to Spot… for Now (2019, background): “[14] found that forensics clas-siﬁers transferred poorly between models, often obtaining near-chance performance.”
Learning on Gradients: Generalized Artifacts Representation for GAN-Generated Images Detection (2023, background): “ForensicTransfer [9] presents a new autoencoder-based architecture to learn a discriminative feature representation in the latent space for detection well on new domains.”
Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection (2020, methodology): “To address this issue, ForensicTransfer [13] proposes an autoencoder-like structure to prevent the network from discarding frame information.”
Media Forensics and DeepFakes: An Overview (2020, background): “Focus on this problem is first found in [192], where an autoencoder-based architecture”
Detecting CNN-Generated Facial Images in Real-World Scenarios (2020, methodology): “For pre-processing techniques we use high-pass filters, co-occurrence matrices, and color transformations, since these have recently been demonstrated to work well in CNN-generated image detection [13, 33, 24].”
Detection of AI-Generated Synthetic Faces (2022, methodology): “In [14] a strategy based on few-shot learning”
Multi-task Learning for Detecting and Segmenting Manipulated Facial Images and Videos (2019, methodology): “For N samples, the reconstruction loss is The total loss is the weighted sum of the three activation losses: Unlike Cozzolino et al. [11], we set the three weights equal to each other (equal to 1).”
What makes fake images detectable? Understanding properties that generalize (2020, background): “Specialized architectures also help generalization: for example [8] uses an autoencoder with a bottleneck that encourages different embeddings for real and fake images.”
FaceForensics++: Learning to Detect Manipulated Facial Images (2019, methodology): “Our database is already used for this forensic transfer learning task, where knowledge of one source manipulation domain is transferred to another target domain, as shown by Cozzolino et al [17].”
DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection (2020, background): “popular Xception network and ForensicTransfer [79], which is an Autoencoder approach.”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, methodology): “Instead of focusing on a specific artifact, some approaches are data-driven, which detect manipulations by classification [157-177] or anomaly identification [178-182].”
Interpretable and Trustworthy Deepfake Detection via Dynamic Prototypes (2021, background): “With more advanced deepfake creations, recent works [13, 29, 15] have shown that the performance of current methods drops drastically on new types of facial manipulations.”
Face X-Ray for More General Face Forgery Detection (2019, background): “ForensicTransfer [11] proposes an autorencoder-based neural network to transfer knowledge between different but related manipulations.”
CoReD: Generalizing Fake Media Detection with Continual Representation using Distillation (2021, methodology): “[12] proposed Forensic Transfer, a weakly supervised method used for transfer learning to different domains.”
Attributing Fake Images to GANs: Learning and Analyzing GAN Fingerprints (2018, unknown): “Recent CNN-based methods [13, 22, 18, 11, 35, 67, 68, 7, 23, 26] learn deep features and further improve tampering detection performance on images or videos.”
Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection (2021, methodology): “Previous studies mostly use ﬁxed ﬁlter banks or hand-crafted methods from other ﬁelds to extract frequency information [9, 13, 41, 29].”
Two-branch Recurrent Network for Isolating Deepfakes in Videos (2020, methodology): “Later methods employed multi-task learning [44] with an encoder-decoder similar to [13] and capsule networks [45].”
Incremental learning for the detection and classification of GAN-generated images (2019, background): “The first solution requires to store very large datasets of images, which is not feasible as data increase continuously, while the latter does not provide satisfactory results unless specific solutions are devised [17].”
Towards Universal Fake Image Detectors that Generalize Across Generative Models (2023, methodology): “Earlier methods studied whether one can even learn a classifier that can detect other images from the same generative model [25, 34, 47], and later work found that such classifiers do not generalize to detecting fakes from other models [19, 53].”