Recurrent Convolutional Strategies for Face Manipulation Detection in Videos

By Ekraam Sabir, Jiaxin Cheng, Ayush Jaiswal, Wael AbdAlmageed, Iacopo Masi, Premkumar Natarajan. CVPR Workshops, 2019

581 citations recorded by the source corpus.

The spread of misinformation through synthetically generated yet realistic images and videos has become a significant problem, calling for robust manipulation detection methods. Despite the predominant effort of detecting face manipulation in still images, less attention has been paid to the identification of tampered faces in videos by taking advantage of the temporal information present in the stream. Recurrent convolutional models are a class of deep learning models which have proven effective at exploiting the temporal information from image streams across domains. We thereby distill the best strategy for combining variations in these models along with domain specific face preprocessing techniques through extensive experimentation to obtain state-of-the-art performance on publicly available video-based facial manipulation benchmarks. Specifically, we attempt to detect Deepfake, Face2Face and FaceSwap tampered faces in video streams. Evaluation is performed on the recently introduced FaceForensics++ dataset, improving the previous state-of-the-art by up to 4.55% in accuracy.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4e910df41181637518563cf729d3bdaa166882c6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset (2021, methodology): “Most deepfake detectors [3, 4, 16, 23, 34] use FF++ and DFDC datasets to train their models to detect deepfakes.”
Deep learning for deepfakes creation and detection: A survey (2019, methodology): “After few steps of pre-processing such as aligning faces, extracting and scaling the bounding boxes of eye landmark points to create new sequences of frames, these cropped eye area sequences are distributed into longterm recurrent convolutional networks (LRCN) [115] for dynamic state prediction.”
Deepfake Video Detection through Optical Flow Based CNN (2019, methodology): “In [11], a recurrent convolutional strategy is used for face manipulation detection where a group of frames is evaluated as an ensemble.”
Video Transformer for Deepfake Detection with Incremental Learning (2021, methodology): “The recurrent networks can learn to detect the intra-frame discrepancies, such as flickering, blurry frames, and mouth movement [27, 47].”
Interpretable and Trustworthy Deepfake Detection via Dynamic Prototypes (2021, background): “Recent work using multi-frame inputs [50, 40] and video architecture [48] have shown the competitive potential of leveraging temporal information.”
Media Forensics and DeepFakes: An Overview (2020, background): “In [189], instead, a solution based on recurrent convolutional models has been proposed.”
Detecting Deepfakes with Self-Blended Images (2022, methodology): “Most previous detection methods [8,16,26,30,36,48,53, 64] perform well on the in-dataset scenario where they detect forgeries they learned in training; however, some studies [15,21,33,61] have found that the detection performance significantly drops in the cross-dataset scenario where fake Landmark ”
Learning Self-Consistency for Deepfake Detection (2020, methodology): “Recurrent neural networks [42] and various types of 3D ConvNets [6] are utilized to detect the manipulation artifacts across the video frames.”
Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues (2020, methodology): “Some studies [51,5] use videos as the input of the face forgery detection system.”
DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection (2020, methodology): “Finally, Sabir et al. [60] proposed a method to detect fake videos based on using the temporal information present in the stream.”
Predicting Video with VQVAE (2021, background): “There is already ongoing successful work in combating malicious applications of image and video generation—speciﬁcally the automatic detection of computer generated media [1, 44, 59].”
Two-branch Recurrent Network for Isolating Deepfakes in Videos (2020, methodology): “The basic network architecture Φ is derived by the recent work in [52] with major modification.”
Deep Learning for Deepfakes Creation and Detection (2019, methodology): “Based on the observation that temporal coherence is not enforced eﬀectively in the synthesis process of deepfakes, Sabir et al. [74] leveraged the use of spatio-temporal features of video streams to detect deepfakes.”
Emotions Don't Lie: An Audio-Visual Deepfake Detection Method using Affective Cues (2020, methodology): “3413570 23, 33, 49] and algorithms [9, 26, 36, 38, 42, 43, 49, 50, 57, 61, 64] for deepfake detection.”