Interpretable Deep Learning under Fire

By Ningfei Wang, S. Ji, Hua Shen, Ting Wang. 2018

195 citations recorded by the source corpus.

Providing explanations for deep neural network (DNN) models is crucial for their use in security-sensitive domains. A plethora of interpretation models have been proposed to help users understand the inner workings of DNNs: how does a DNN arrive at a specific decision for a given input? The improved interpretability is believed to offer a sense of security by involving human in the decision-making process. Yet, due to its data-driven nature, the interpretability itself is potentially susceptible to malicious manipulations, about which little is known thus far. Here we bridge this gap by conducting the first systematic study on the security of interpretable deep learning systems (IDLSes). We show that existing \imlses are highly vulnerable to adversarial manipulations. Specifically, we present ADV^2, a new class of attacks that generate adversarial inputs not only misleading target DNNs but also deceiving their coupled interpretation models. Through empirical evaluation against four major types of IDLSes on benchmark datasets and in security-critical applications (e.g., skin cancer diagnosis), we demonstrate that with ADV^2 the adversary is able to arbitrarily designate an input's prediction and interpretation. Further, with both analytical and empirical evidence, we identify the prediction-interpretation gap as one root cause of this vulnerability -- a DNN and its interpretation model are often misaligned, resulting in the possibility of exploiting both models simultaneously. Finally, we explore potential countermeasures against ADV^2, including leveraging its low transferability and incorporating it in an adversarial training framework. Our findings shed light on designing and operating IDLSes in a more secure and informative fashion, leading to several promising research directions.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Measuring Association Between Labels and Free-Text Rationales (2020, background): “Zhang et al. (2020) show that saliency maps and model predictions can be independently adver-sarially attacked in vision and clinical tasks, and conclude this is due to a misalignment between the saliency map generator and model predictor.”
Aggregating explanation methods for stable and robust explainability. (2019, background): “The lack of transferability results are in line with the findings of (Zhang et al., 2018b).”
Fooling Neural Network Interpretations via Adversarial Model Manipulation (2019, background): “Firstly, unlike [9, 37], which attack each given input image to fool the interpreter, we change the model parameters via fine-tuning a pre-trained model, and do not perturb the input data.”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, methodology): “In EXNLP, Jhamtani and Clark (2020), Zhang et al. (2020a) and Zellers et al. (2019a) apply a two-stage methodology to explanation collection, by either extracting explanation candidates automatically (Jhamtani and Clark, 2020) or having crowdworkers write them (Zhang et al., 2020a; Zellers et al.,…”
Aggregating explainability methods for neural networks stabilizes explanations (2019, background): “…into explainability is reﬂected in a lot of recent work into the problem (Kindermans et al., 2017; Selvaraju et al., 2017; Bach et al., 2015; Zhang et al., 2018; Zhou et al., 2016; Ancona et al., 2018; Ribeiro et al., 2016; Rieger et al., 2018; Kim et al., 2018; Lundberg & Lee, 2017;…”