Interpretable and Trustworthy Deepfake Detection via Dynamic Prototypes

By Loc Trinh, Michael Tsang, Sirisha Rambhatla, Yan Liu. 2021

101 citations recorded by the source corpus.

In this paper we propose a novel human-centered approach for detecting forgery in face images, using dynamic prototypes as a form of visual explanations. Currently, most state-of-the-art deepfake detections are based on black-box models that process videos frame-by-frame for inference, and few closely examine their temporal inconsistencies. However, the existence of such temporal artifacts within deepfake videos is key in detecting and explaining deepfakes to a supervising human. To this end, we propose Dynamic Prototype Network (DPNet) – an interpretable and effective solution that utilizes dynamic representations (i.e., prototypes) to explain deepfake temporal artifacts. Extensive experimental results show that DPNet achieves competitive predictive performance, even on unseen testing datasets such as Google’s DeepFakeDetection, DeeperForensics, and Celeb-DF, while providing easy referential explanations of deepfake dynamics. On top of DPNet’s prototypical framework, we further formulate temporal logic specifications based on these dynamics to check our model’s compliance to desired temporal behaviors, hence providing trustworthiness for such critical detection systems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lesia Semenova
2. Chudi Zhong
3. Zhi Chen
4. Haiyang Huang
5. Cynthia Rudin
6. Mingze Xu
7. Wei Xia
8. Hui Ding

In Computer Vision and Pattern Recognition

1. Cynthia Rudin
2. Zhi Chen
3. Lesia Semenova
4. Chudi Zhong
5. Haiyang Huang
6. Wei Xia
7. Mingze Xu
8. Hui Ding

In Safety, Robustness & Interpretability

1. Lesia Semenova
2. Zhi Chen
3. Chudi Zhong
4. Haiyang Huang
5. Wei Xia
6. Yuanjun Xiong
7. Cynthia Rudin
8. Octavio Loyola-González

Citations in the corpus

• Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “While Trinh et al. (2021) recently extended the ideas in Chen et al. (2019) and developed a dynamic prototype network (DPNet) which learns prototypical video patches from deep-faked and real videos, how to efficiently compare various videos and find similar videos (for either nearest-neighbor or…”
• Learning Self-Consistency for Deepfake Detection (2020, background): “Most of these defense methods [60, 26, 37, 51, 46, 5, 23, 21, 6] target detecting suspicious artifacts left in the stitching process, such as eye blinking [26], face warping [27], blending boundaries [23], and fake prototypes [48].”