Emotions Don't Lie: An Audio-Visual Deepfake Detection Method using Affective Cues

By Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, Dinesh Manocha. ACM Multimedia, 2020

363 citations recorded by the source corpus.

We present a learning-based method for detecting real and fake deepfake multimedia content. To maximize information for learning, we extract and analyze the similarity between the two audio and visual modalities from within the same video. Additionally, we extract and compare affective cues corresponding to perceived emotion from the two modalities within a video to infer whether the input video is "real" or "fake". We propose a deep learning network, inspired by the Siamese network architecture and the triplet loss. To validate our model, we report the AUC metric on two large-scale deepfake detection datasets, DeepFake-TIMIT Dataset and DFDC. We compare our approach with several SOTA deepfake detection methods and report per-video AUC of 84.4% on the DFDC and 96.6% on the DF-TIMIT datasets, respectively. To the best of our knowledge, ours is the first approach that simultaneously exploits audio and video modalities and also perceived emotions from the two modalities for deepfake detection.

View Paper: https://www.semanticscholar.org/paper/66970c48758b65a36d77c0fa9add1198d36ecd59

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deepfake detection using deep learning methods: A systematic and comprehensive review (2023, background): “In addition, Mittal et al. (2020) demonstrated a learning-based technique for distinguishing between genuine and phony deepfake multimedia content.”
FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset (2021, methodology): “Most deepfake detectors [3, 4, 16, 23, 34] use FF++ and DFDC datasets to train their models to detect deepfakes.”
Video Transformer for Deepfake Detection with Incremental Learning (2021, methodology): “propose a multi-modal deepfake detection method in [38].”
Exploring Temporal Coherence for More General Video Face Forgery Detection (2021, background): “Unlike previous works, they either directly apply 3D convolution network [15, 21] on video or detect a specific kind of incoherence, such as irregular eye blinking [33], lip motions [22] or emotion [37].”
Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection (2020, background): “datasets [46, 38, 30, 16], it is possible to train deep convolutional neural networks (CNNs) to detect fake videos [3, 25, 64, 4, 14, 45, 42, 36].”