RISE: Randomized Input Sampling for Explanation of Black-box Models

By Vitali Petsiuk, Abir Das, Kate Saenko. 2018

1,552 citations recorded by the source corpus.

Deep neural networks are being used increasingly to automate data analysis and decision making, yet their decision-making process is largely unclear and is difficult to explain to the end users. In this paper, we address the problem of Explainable AI for deep neural networks that take images as input and output a class probability. We propose an approach called RISE that generates an importance map indicating how salient each pixel is for the model's prediction. In contrast to white-box approaches that estimate pixel importance using gradients or other internal network state, RISE works on black-box models. It estimates importance empirically by probing the model with randomly masked versions of the input image and obtaining the corresponding outputs. We compare our approach to state-of-the-art importance extraction methods using both an automatic deletion/insertion metric and a pointing metric based on human-annotated object segments. Extensive experiments on several benchmark datasets show that our approach matches or exceeds the performance of other methods, including white-box approaches. Project page: this http URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks (2019, methodology): “We use the deletion metric [32] to evaluate the faith-”
Not Using the Car to See the Sidewalk — Quantifying and Controlling the Effects of Context in Classification and Segmentation (2018, background): “Related work [16] proposes erasing randomly sampled pixels to visualize important regions for a black-box models decision.”
Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting (2020, methodology): “However, recent work in explainable artiﬁcial intelligence (XAI) has developed meth-ods to create human-interpretable explanations for model decisions (Simonyan et al., 2013; Zhang et al., 2018; Petsiuk et al., 2018; Zhou et al., 2016; Selvaraju et al., 2017).”
A Disentangling Invertible Interpretation Network for Explaining Latent Representations (2020, background): “[41] shows how saliency maps representing the importance of pixels for a classifier’s decision can be obtained without access to the classifiers gradients.”
XOC: Explainable Observer-Classifier for Explainable Binary Decisions (2019, background): “, by visualizing features [36, 43, 34, 37], saliency maps [35, 29], interpretable features [1], and modular networks [3].”
Regional Multi-Scale Approach for Visually Pleasing Explanations of Deep Neural Networks (2018, methodology): “presented the randomized input sampling for explanation (RISE) method, which generates an importance map indicating how salient each pixel is for prediction of the black-box model [18].”
A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and concepts (2021, methodology): “Similar to RISE, their idea is to find a minimal occlusion mask that if used to perturb the image (e.g., blur, noise, or blacken) maximally changes the outcome.”
SurvNAM: The machine learning survival model explanation (2021, methodology): “Many aforementioned explanation methods are based on perturbation techniques [41, 42, 43, 44].”
From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2022, background): “[197] believe that “keeping humans out of the loop for evaluation makes it more fair and true to the classifier’s own view on the problem rather than representing a human’s view”.”
Understanding Deep Networks via Extremal Perturbations and Smooth Masks (2019, methodology): “Some attribution methods, on the other hand, directly perform an analysis of the effect of perturbing the network’s input on its output [30, 19, 6, 4].”
Proactive Pseudo-Intervention: Causally Informed Contrastive Learning For Interpretable Vision Models (2020, methodology): “Several methods follow a similar idea, but use other importance measures or occlusion strategies [43, 45, 50].”
Benchmarking and survey of explanation methods for black box models (2021, background): “– Deletion and Insertion (Petsiuk et al. 2018) are metrics that remove the features that the explanation method f found important and see how the performance of b degrades.”
Information-Bottleneck Approach to Salient Region Discovery (2019, background): “A similar task of identifying salient regions in input samples is frequently addressed in the studies of neural network interpretation [5, 7, 8, 20, 22, 23, 31].”
Explainable AI: A Review of Machine Learning Interpretability Methods (2020, positive): “In order to interpret the predictions of deep neural networks for images, the RISE algorithm [39] creates a saliency map for any black-box model, indicating how important each pixel of the image with respect to the network’s prediction.”
Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey (2020, methodology): “[57] Study saliency maps by randomized masking of inputs ResNet50, VGG16 -”