MesoNet: a Compact Facial Video Forgery Detection Network

By Darius Afchar, Vincent Nozick, Junichi Yamagishi, Isao Echizen. 2018

1,740 citations recorded by the source corpus.

This paper presents a method to automatically and efficiently detect face tampering in videos, and particularly focuses on two recent techniques used to generate hyper-realistic forged videos: Deepfake and Face2Face. Traditional image forensics techniques are usually not well suited to videos due to the compression that strongly degrades the data. Thus, this paper follows a deep learning approach and presents two networks, both with a low number of layers to focus on the mesoscopic properties of images. We evaluate those fast networks on both an existing dataset and a dataset we have constituted from online videos. The tests demonstrate a very successful detection rate with more than 98% for Deepfake and 95% for Face2Face.

Subfields: Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

CoReD: Generalizing Fake Media Detection with Continual Representation using Distillation (2021, background): “[1] proposed MesoNet, which is a CNNbased state-of-the-art deepfake detection method.”
Deepfake detection using deep learning methods: A systematic and comprehensive review (2023, background): “Every one of these techniques seems to be an unquestionably valuable addition to the toolbox of a digital artist (Afchar et al., 2018; Y. Li, Zhang, et al., 2021).”
DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection (2020, methodology): “Celeb-DF [26] also provides a face forgery detection benchmark including seven methods [6, 12, 25, 28, 30, 41, 43] trained and tested on different datasets.”
FaceForensics++: Learning to Detect Manipulated Facial Images (2019, methodology): “While DeepFakes is also training one model per manipulation, it uses a fixed post-processing pipeline sim- ilar to the computer-based manipulation methods and thus has consistent artifacts.”
Detecting CNN-Generated Facial Images in Real-World Scenarios (2020, methodology): “Following these works, learning-based methods have been proposed, using CNNs to automatically learn features of real and generated images [1, 3, 7, 12, 13, 40].”
Exposing DeepFake Videos By Detecting Face Warping Artifacts (2018, methodology): “The negative examples can be generated by applying DeepFake algorithms as in [27], but it requires us to train and run the DeepFake algorithms, which is time-consuming and resource-demanding.”
End-to-End Reconstruction-Classification Learning for Face Forgery Detection (2022, methodology): “Early face forgery detection methods [1,8,23,31,35,60] typically follow the classic pipeline of learning convolutional neural networks (CNN) for image classification.”
Multi-task Learning for Detecting and Segmenting Manipulated Facial Images and Videos (2019, methodology): “Other approaches to detection include using a constrained convolutional layer [8], using a statistical pooling layer [25], using a two-stream network [31], using a lightweight CNN network [2], and using two cascaded convolutional layers at the bottom of a CNN [23].”
Recurrent Convolutional Strategies for Face Manipulation Detection in Videos (2019, methodology): “In the meantime, MesoNet [4] introduces two CNN based architectures for face manipulation detection: they take a mesoscopic approach to manipulation detection, which combines information from both low level (microscopic) and high level (macroscopic) features.”
What makes fake images detectable? Understanding properties that generalize (2020, methodology): “We train and evaluate full MesoInception4 [2], Resnet [13], and Xception [6] models on the same datasets that we use to train the truncated classifiers.”
Generalizing Face Forgery Detection with High-frequency Features (2021, background): “[6] provided two compact network architectures to capture the mesoscopic features.”
Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction (2020, background): “Moreover, the detection of manipulated videos is an active area of research [31, 12, 1, 24, 30, 13, 11, 2, 44] and we foresee significant improvements in this domain.”
Transformation-based Adversarial Video Prediction on Large-Scale Data (2020, background): “As the field progresses, methods that can detect generated images or videos automatically, such as [1, 57, 79], are therefore becoming increasingly important.”
Deepfake Video Detection through Optical Flow Based CNN (2019, background): “From another side, various studies have lastly focused on the problem to detect deepfake-like videos; most of them by analyzing possible inconsistencies within RGB frames of the video [9, 10, 1].”
Creating, Using, Misusing, and Detecting Deep Fakes (2022, background): “For example, a network designed to focus on mesoscopic (intermediate-scale) features (Afchar et al. 2018) rather than low-level (e.”
UCF: Uncovering Common Features for Generalizable Deepfake Detection (2023, background): “Self-made 0.548 0.497 MesoInception4 [3] Self-made 0.536 0.499 DSP-FWA [25] FF++ 0.646 0.646 VA-MLP [32] FF++ 0.550 - Multi-task [33] FF++ 0.543 - Headpose [52] UADFV 0.546 - Capsule [34] FF++ 0.575 0.575 SMIL [24] FF++ 0.563 0.563 Two-branch [ operate in the entire feature space, which inevitably…”
Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics (2019, methodology): “, [6, 17, 35, 36]) are data-driven, which directly employ various types of DNNs trained on real and DeepFake videos, not relying on any specific artifact.”
Watch Your Up-Convolution: CNN Based Generative Deep Neural Networks Are Failing to Reproduce Spectral Distributions (2020, background): “Most approaches reported in the literature, like [38, 3, 57], are themselves relying on CNNs and thus require large amounts of annotated training data.”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, methodology): “[163] MesoInceptio n-4 Deep features (DF) TPR= 81.”