Real Time Image Saliency for Black Box Classifiers

By Piotr Dabkowski, Yarin Gal. 2017

638 citations recorded by the source corpus.

In this work we develop a fast saliency detection method that can be applied to any differentiable image classifier. We train a masking model to manipulate the scores of the classifier by masking salient parts of the input image. Our model generalises well to unseen images and requires a single forward pass to perform saliency detection, therefore suitable for use in real-time systems. We test our approach on CIFAR-10 and ImageNet datasets and show that the produced saliency maps are easily interpretable, sharp, and free of artifacts. We suggest a new metric for saliency and test our method on the ImageNet object localisation task. We achieve results outperforming other weakly supervised methods.

Subfields: Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “In [226] a real-time image saliency method is proposed, which is applicable to differentiable image classifiers.”
Interpretations are useful: penalizing explanations to align neural networks with prior knowledge (2019, neutral): “…Baehrens et al., 2010; Rieger & Hansen, 2019), decomposition-based (Murdoch & Szlam, 2017; Shrikumar et al., 2016; Bach et al., 2015) and others (Dabkowski & Gal, 2017; Fong & Vedaldi, 2017; Ribeiro et al., 2016; Zintgraf et al., 2017), with many similarities among the meth-ods (Ancona et al.,…”
Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks (2019, methodology): “The resulting explanations (Fig 1 b) are valid model inputs and their faithfulness can, thus, be directly verified (as in methods from [17, 14, 6, 11]).”
Learning to Explain With Complemental Examples (2018, background): “The primary stream is visualizing where the classifier weighs for its prediction by assigning an importance to each element in the input space, by propagating the prediction to the input space [24, 3, 29, 23, 30, 8, 31], or by learning the instance-wise importance of elements [4, 5, 15] with an auxi”
Sanity Checks for Saliency Maps (2018, methodology): “Dabkowski and Gal [17] proposed an entropy-based metric to quantify the amount of relevant information an explanation mask captures.”
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning (2018, background): “Nevertheless, legislation like the European Union’s General Data Protection Regulation require that companies deploying ML and other forms of analysis on certain sensitive data provide such interpretable outputs if the prediction made by a ML model is used to make decisions without a human in the lo”
Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values (2018, methodology): “While there are several different approaches for obtaining local-explanations (Ribeiro et al., 2016; Fong & Vedaldi, 2017; Dabkowski & Gal, 2017; Zintgraf et al., 2017; Pieter-Jan Kindermans, 2018), recent work by Ancona et al. (2017) and Lundberg & Lee (2017) both show that there are equivalences…”
Learning how to explain neural networks: PatternNet and PatternAttribution (2017, methodology): “As a result, our method produces explanations at least as fast as the work by Dabkowski & Gal (2017) on real time saliency.”
Getting a CLUE: A Method for Explaining Uncertainty Estimates (2020, background): “| |· It is common for approaches to generating saliency maps to employ constraints that encourage the contiguity of highlighted pixels (Chang et al., 2019; Dabkowski & Gal, 2017).”
Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning (2019, neutral): “…classify images in computer vision applications (Simonyan et al., 2014; Zeiler & Fergus, 2014; Springenberg et al., 2015; Ribeiro et al., 2016; Dabkowski & Gal, 2017; Fong & Vedaldi, 2017; Selvaraju et al., 2017; Shrikumar et al., 2017; Smilkov et al., 2017; Zhang et al., 2018) and to explain…”
Deformable ConvNets V2: More Deformable, Better Results (2018, background): “Recent works on effective receptive fields [30] and salient regions [40, 41, 12, 6] reveal that only a small proportion of pixels in the theoretical receptive field contribute significantly to the final network prediction.”
Explanations can be manipulated and geometry is to blame (2019, background): “[11] Piotr Dabkowski and Yarin Gal.”
Visualizing and Understanding Atari Agents (2017, background): “More recent approaches by [2] and [6] use masked interpolations between the original image I and some other image A, where A is chosen to introduce as little new information as possible.”
RISE: Randomized Input Sampling for Explanation of Black-box Models (2018, background): “There are several ways of removing pixels from an image [4], e.”
Explaining Classifiers with Causal Concept Effect (CaCE) (2019, background): “Many approaches have been proposed to tackle this challenge: for example by developing a model with interpretable components built-in [8, 16] or by building post-training interpretability methods [13, 14, 4, 2, 1, 9].”
A Benchmark for Interpretability Methods in Deep Neural Networks (2018, methodology): “For each dataset and estimator, we generate new train and test sets that each correspond to a different fraction of feature modification t = [0, 10, 30, 50, 70, 90].”