M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection

By Junke Wang, Zuxuan Wu, Jingjing Chen, Yu-Gang Jiang. International Conference on Multimedia Retrieval, 2021

409 citations recorded by the source corpus.

The widespread dissemination of Deepfakes demands effective approaches that can detect perceptually convincing forged images. In this paper, we aim to capture the subtle manipulation artifacts at different scales using transformer models. In particular, we introduce a Multi-modal Multi-scale TRansformer (M2TR), which operates on patches of different sizes to detect local inconsistencies in images at different spatial levels. M2TR further learns to detect forgery artifacts in the frequency domain to complement RGB information through a carefully designed cross modality fusion block. In addition, to stimulate Deepfake detection research, we introduce a high-quality Deepfake dataset, SR-DF, which consists of 4,000 DeepFake videos generated by state-of-the-art face swapping and facial reenactment methods. We conduct extensive experiments to verify the effectiveness of the proposed method, which outperforms state-of-the-art Deepfake detection methods by clear margins.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/21e0858665cddf51689fc680f72ec4e00b68ae04

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Baoyuan Wu
2. Guangming Lu
3. Bingzhi Chen
4. Zhiyuan Yan
5. David Zhang
6. Yanbo Fan
7. Yong Zhang
8. Zheng Zhang

Citations in the corpus

• UCF: Uncovering Common Features for Generalizable Deepfake Detection (2023, background): “Other notable directions are focusing on some specific representations, i.e., forgery region location [33], neuron behaviors [49], optical flow [4], landmark geometric features [42], 3D decomposition [59], erasing technology [47], and attentional networks [10, 55, 48].”
• DS-TransUNet: Dual Swin Transformer U-Net for Medical Image Segmentation (2021, methodology): “M2TR [23] introduces a multi-scale transformer that 2) and Eq.”