Interpretable Explanations of Black Boxes by Meaningful Perturbation

By Ruth C. Fong, Andrea Vedaldi. 2017

1,723 citations recorded by the source corpus.

As machine learning algorithms are increasingly applied to high impact yet high risk tasks, such as medical diagnosis or autonomous driving, it is critical that researchers can explain how such algorithms arrived at their predictions. In recent years, a number of image saliency methods have been developed to summarize where highly complex neural networks “look” in an image for evidence for their predictions. However, these techniques are limited by their heuristic nature and architectural constraints. In this paper, we make two main contributions: First, we propose a general framework for learning different kinds of explanations for any black box algorithm. Second, we specialise the framework to find the part of an image most responsible for a classifier decision. Unlike previous works, our method is model-agnostic and testable because it is grounded in explicit and interpretable image perturbations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Women also Snowboard: Overcoming Bias in Captioning Models (2018, background): “However, [23] consider somewhat simple data and generating explanations for more complex tasks (like image captioning) is a difficult and active area of research [24,25,26,27,28,29].”
Random Erasing Data Augmentation (2017, background): “Fong and Vedaldi (Fong and Vedaldi 2017) blur an image to suppress the SoftMax probability of the target class to learn saliency region.”
Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “Sensitivity / Saliency: [85, 226] [222, 223]”
Model-Reuse Attacks on Deep Learning Systems (2018, background): “random noise to each dimension of x∗, which however ignores the fact that some parts of x∗ are more critical than the rest with respect to its classification [21].”
Interpretations are useful: penalizing explanations to align neural networks with prior knowledge (2019, neutral): “…Rieger & Hansen, 2019), decomposition-based (Murdoch & Szlam, 2017; Shrikumar et al., 2016; Bach et al., 2015) and others (Dabkowski & Gal, 2017; Fong & Vedaldi, 2017; Ribeiro et al., 2016; Zintgraf et al., 2017), with many similarities among the meth-ods (Ancona et al., 2018; Lundberg & Lee,…”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, background): “There is no guarantee there exists globally important feature that is invariant in a machine learning task.”
Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks (2019, methodology): “The resulting explanations (Fig 1 b) are valid model inputs and their faithfulness can, thus, be directly verified (as in methods from [17, 14, 6, 11]).”
Learning to Explain With Complemental Examples (2018, background): “The primary stream is visualizing where the classifier weighs for its prediction by assigning an importance to each element in the input space, by propagating the prediction to the input space [24, 3, 29, 23, 30, 8, 31], or by learning the instance-wise importance of elements [4, 5, 15] with an auxi”
Sanity Checks for Saliency Maps (2018, methodology): “Aside gradient-based approaches, other methods ‘learn’ an explanation per sample for a model [20, 17, 12, 15, 11, 21].”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, methodology): “While there are several different techniques for obtaining local-explanations [96]–[100], recent work by Lundberg and Lee [101] have shown that there are equivalences among these techniques.”
Unsupervised Learning of Neural Networks to Explain Neural Networks (2018, methodology): “Many studies for network interpretability mainly visualized image appearance corresponding to a neural unit inside a CNN [39, 22, 29, 7, 38, 6] or extracted image regions that were responsible for network output [24, 8, 14, 9, 27, 17].”
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges (2018, methodology): “Because of this property we can categorize the following methods to the attribution category: occlusion [21], erasure [31], perturbation [18], adversarial examples [44] and prediction difference analysis [60].”
Multimodal Explanations: Justifying Decisions and Pointing to the Evidence (2018, background): “Perhaps the most prevalent form of visual explanation rely on producing heat maps/attention maps which indicate which region of an image is most important for a decision [13, 29, 37, 41].”
Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values (2018, methodology): “While there are several different approaches for obtaining local-explanations (Ribeiro et al., 2016; Fong & Vedaldi, 2017; Dabkowski & Gal, 2017; Zintgraf et al., 2017; Pieter-Jan Kindermans, 2018), recent work by Ancona et al. (2017) and Lundberg & Lee (2017) both show that there are equivalences…”
A Survey of Methods for Explaining Black Box Models (2018, background): “(Right) From Reference [30] the mask and the level of accuracy on the image considering and”
Interpreting CNNs via Decision Trees (2018, neutral): “[12] propagated gradients of feature maps w.”
Visual Feature Attribution Using Wasserstein GANs (2017, methodology): “Examples include Guided Backprop [53], Excitation Backprop [62], Integrated Gradients [54], meaningful perturbations [13].”