Fighting Fake News: Image Splice Detection via Learned Self-Consistency

By Minyoung Huh, Andrew Liu, Andrew Owens, Alexei A. Efros. 2018

473 citations recorded by the source corpus.

Advances in photo editing and manipulation tools have made it significantly easier to create fake imagery. Learning to detect such manipulations, however, remains a challenging problem due to the lack of sufficient amounts of manipulated training data. In this paper, we propose a learning algorithm for detecting visual image manipulations that is trained only using a large dataset of real photographs. The algorithm uses the automatically recorded photo EXIF metadata as supervisory signal for training a model to determine whether an image is self-consistent — that is, whether its content could have been produced by a single imaging pipeline. We apply this self-consistency model to the task of detecting and localizing image splices. The proposed method obtains state-of-the-art performance on several image forensics benchmarks, despite never seeing any manipulated images at training. That said, it is merely a step in the long quest for a truly general purpose visual forensics tool.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

CNN-Generated Images Are Surprisingly Easy to Spot… for Now (2019, methodology): “Evaluation Following other recent forensics works [52, 18, 44], we evaluate our model’s performance on each dataset using average precision (AP), since it is a threshold-less, ranking-based score that is not sensitive to the base rate of the real and fake images in the dataset.”
Exploiting Visual Artifacts to Expose Deepfakes and Face Manipulations (2019, methodology): “Recent statistical methods form statistical fingerprints on the residuals of an image to detect manipulations [11, 6, 7], validate noise priors from metadata [14], or learn specific manipulation traces, such as recoloring [3] or recompression [28].”
Language-Driven Image Style Transfer (2021, methodology): “To mitigate this issue, we can apply techniques from image forensics [88, 89, 90] to detect the authenticity of an image.”
Noiseprint: A CNN-Based Camera Model Fingerprint (2018, methodology): “To save space, we use compact acronyms, for example EXIF-SC to mean EXIF self-consistency algorithm [57].”
ManTra-Net: Manipulation Tracing Network for Detection and Localization of Image Forgeries With Anomalous Features (2019, methodology): “Furthermore, methods that provide localization capabilities often rely on heavy, time-consuming pre-and/or post-processing, e.g ., patch extraction [53], expectation-maximization [19, 20], feature clustering [14, 11, 32, 28], segmentation [32, 28, 15], etc .”
DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection (2020, background): “In particular, we refer the reader to [45], [46].”
Steering Self-Supervised Feature Learning Beyond Local Pixel Statistics (2020, background): “Many works have considered the detection of image manipulations in the context of image forensics [22, 49, 59, 2].”
Predicting Video with VQVAE (2021, background): “As research in generative models progresses, it is essential that the ﬁeld of visual forensics [17]—progresses in tandem.”
Detecting Photoshopped Faces by Scripting Photoshop (2019, background): “Self-consistency, on the other hand, is designed to detect low-level differences in image characteristics.”
FakeSpotter: A Simple Baseline for Spotting AI-Synthesized Fake Faces (2019, methodology): “Huh et al. [13] observed that the image EXIF metadata can be used to differentiate fake faces.”
Media Forensics and DeepFakes: An Overview (2020, background): “A Siamese network is also trained in [154] to decide whether two image patches have similar metadata.”
Swapping Autoencoder for Deep Image Manipulation (2020, methodology): “Similar ideas for modeling co-occurences have been used for propagating a single texture in a supervised setting [74], self-supervised representation learning [33], and identifying image composites [29].”
GANprintR: Improved Fakes and Evaluation of the State of the Art in Face Manipulation Detection (2019, background): “Finally, we also include for completeness some relevant references to other recent studies focused on the detection of general GAN-based image manipulations, not facial ones: [12], [35], [36], [37], [38].”