Multi-task Learning for Detecting and Segmenting Manipulated Facial Images and Videos

By Huy Hoang Le Nguyen, Fuming Fang, Junichi Yamagishi, Isao Echizen. 2019 IEEE 10th International Conference on Biometrics Theory, Applications and Systems (BTAS), 2019

522 citations recorded by the source corpus.

Detecting manipulated images and videos is an important topic in digital media forensics. Most detection methods use binary classification to determine the probability of a query being manipulated. Another important topic is locating manipulated regions (i.e., performing segmentation), which are mostly created by three commonly used attacks: removal, copy-move, and splicing. We have designed a convolutional neural network that uses the multi-task learning approach to simultaneously detect manipulated images and videos and locate the manipulated regions for each query. Information gained by performing one task is shared with the other task and thereby enhance the performance of both tasks. A semi-supervised learning approach is used to improve the network’s generability. The network includes an encoder and a Y-shaped decoder. Activation of the encoded features is used for the binary classification. The output of one branch of the decoder is used for segmenting the manipulated regions while that of the other branch is used for reconstructing the input, which helps improve overall performance. Experiments using the FaceForensics and Face-Forensics++ databases demonstrated the networks effectiveness against facial reenactment attacks and face swapping attacks as well as its ability to deal with the mismatch condition for previously seen attacks. Moreover, fine-tuning using just a small amount of data enables the network to deal with unseen attacks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d44b403a81764457dce2ec098592e4981716bcb2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection (2020, methodology): “Celeb-DF [26] also provides a face forgery detection benchmark including seven methods [6, 12, 25, 28, 30, 41, 43] trained and tested on different datasets.”
Face X-Ray for More General Face Forgery Detection (2019, background): “Some recent related works [23, 13, 11, 29] have also noticed the generalization issue and tried to solve the problem to a certain degree.”
A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentials (2022, background): “To address this issue, many deepfake detection datasets (e.g.,[40, 44, 46, 21, 64, 19, 31]) and techniques (e.g., [87, 4, 80, 57, 56, 5]) are proposed.”
Generalizing Face Forgery Detection with High-frequency Features (2021, methodology): “We then compare our model against several methods that adopt multi-task learning for a better generalization ability, including LAE [14], ClassNSeg [35], and ForensicTrans [13].”
Interpretable and Trustworthy Deepfake Detection via Dynamic Prototypes (2021, methodology): “Lastly, we also inspect the Equal Error Rate, similar to [44].”
Video Transformer for Deepfake Detection with Incremental Learning (2021, methodology): “Nguyen et al. [39] propose a di � erent strategy to detect deepfake video.”
Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues (2020, background): “Various methods [3,33,45,46,43,58,60,30] have been proposed to detect the forged media.”
Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain (2021, unknown): “…83.00 53.60 HeadPose [48] 47.30 54.60 FWA [25] 80.10 56.90 VA-MLP [29] 66.40 55.00 VA-LogReg 78.00 55.10 Xception-c40 [37] 95.50 65.50 Multi-task [30] 76.30 54.30 Capsule [31] 96.60 57.50 DSP-FWA [25] 93.00 64.60 SMIL [23] 96.80 56.30 Two-branch [28] 93.20 73.40 F 3 -Net [35] 97.97 65.17 Our…”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, methodology): “[164] proposed a multi-task, learning-based CNN network to simultaneously detect and localize manipulated content from the videos.”
UCF: Uncovering Common Features for Generalizable Deepfake Detection (2023, background): “, forgery region location [30], neuron behaviors [43], optical flow [4], landmark geometric features [38], and attentional networks [8, 48, 42].”
Two-branch Recurrent Network for Isolating Deepfakes in Videos (2020, methodology): “5 Multi-task [44] 76.”
M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection (2021, methodology): “[72] uses a two-stream architecture to capture facial manipulation clues and patch inconsistency separately, while [42] simultaneously identifies forged faces and locates the manipulated regions with multi-task learning.”
On the Detection of Digital Face Manipulation (2019, background): “There are two main approaches to localize manipulated image regions: segmenting the entire image [9, 39], and repeatedly performing binary classification via a sliding window [42].”
Media Forensics and DeepFakes: An Overview (2020, methodology): “This same approach has been followed in [193] by using a deeper network and including a segmentation task.”
Learning Self-Consistency for Deepfake Detection (2020, background): “Most of these defense methods [60, 26, 37, 51, 46, 5, 23, 21, 6] target detecting suspicious artifacts left in the stitching process, such as eye blinking [26], face warping [27], blending boundaries [23], and fake prototypes [48].”
The Creation and Detection of Deepfakes (2020, background): “9 [118] 2019 • • • • • CNN-DE • 256x256 • • 92.”
Detecting Deepfakes with Self-Blended Images (2022, methodology): “Most previous detection methods [8,16,26,30,36,48,53, 64] perform well on the in-dataset scenario where they detect forgeries they learned in training; however, some studies [15,21,33,61] have found that the detection performance significantly drops in the cross-dataset scenario where fake Landmark ”
Self-supervised Learning of Adversarial Example: Towards Good Generalizations for Deepfake Detection (2022, methodology): “Using multi-task learning strategies to boost deepfake detection has been explored in previous works, including LAE [14], ClassNSeg [34], and Forensic-Trans [9].”