Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection

By Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, Maja Pantic. Computer Vision and Pattern Recognition, 2020

624 citations recorded by the source corpus.

Although current deep learning-based face forgery detectors achieve impressive performance in constrained scenarios, they are vulnerable to samples created by unseen manipulation methods. Some recent works show improvements in generalisation but rely on cues that are easily corrupted by common post-processing operations such as compression. In this paper, we propose LipForensics, a detection approach capable of both generalising to novel manipulations and withstanding various distortions. LipForensics targets high-level semantic irregularities in mouth movements, which are common in many generated videos. It consists in first pretraining a spatio-temporal network to perform visual speech recognition (lipreading), thus learning rich internal representations related to natural mouth motion. A temporal network is subsequently finetuned on fixed mouth embeddings of real and forged data in order to detect fake videos based on mouth movements without overfitting to low-level, manipulation-specific artefacts. Extensive experiments show that this simple approach significantly surpasses the state-of-the-art in terms of generalisation to unseen manipulations and robustness to perturbations, as well as shed light on the factors responsible for its performance.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/cefef7a421d46a177a054698cc2d0bb4a788d00a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

End-to-End Reconstruction-Classification Learning for Face Forgery Detection (2022, background): “Considering the ubiquity of image processing on social media, we investigate the performance under several perturbations suggested by [15, 17], i.”
CelebV-HQ: A Large-Scale Video Facial Attributes Dataset (2022, methodology): “, attribute recognition [97,17,41], action recognition [81,37], emotion recognition [18,47] and forgery detection [50,27,104]).”
FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset (2021, methodology): “In particular, the detection performance of Face X-ray, F3Net, and LipForensics are 53.5%, 59.8%, and 50.4%, respectively.”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, methodology): “[115] 3D-ResNet18, multi-scale temporal convolutional network AUC=97.”
Frequency-Aware Deepfake Detection: Improving Generalizability through Frequency Space Domain Learning (2024, background): “Other image-based detection methods have developed specific artifact detection in distinct facial regions, such as eyes(Li et al. 2018) and lips (Haliassos et al. 2021).”
UCF: Uncovering Common Features for Generalizable Deepfake Detection (2023, methodology): “To date, deepfake detection can be broadly categorized into two types of tasks: image forgery detection [37, 23, 7] and video forgery detection [38, 16, 57].”