Understanding Deep Networks via Extremal Perturbations and Smooth Masks

By Ruth C. Fong, Mandela Patrick, Andrea Vedaldi. 2019

494 citations recorded by the source corpus.

Attribution is the problem of finding which parts of an image are the most responsible for the output of a deep neural network. An important family of attribution methods is based on measuring the effect of perturbations applied to the input image, either via exhaustive search or by finding representative perturbations via optimization. In this paper, we discuss some of the shortcomings of existing approaches to perturbation analysis and address them by introducing the concept of extremal perturbations, which are theoretically grounded and interpretable. We also introduce a number of technical innovations to compute these extremal perturbations, including a new area constraint and a parametric family of smooth perturbations, which allow us to remove all tunable weighing factors from the optimization problem. We analyze the effect of perturbations as a function of their area, demonstrating excellent sensitivity to the spatial properties of the network under stimulation. We also extend perturbation analysis to the intermediate layers of a deep neural network. This application allows us to show how compactly an image can be represented (in terms of the number of channels it requires). We also demonstrate that the consistency with which images of a given class rely on the same intermediate channel correlates well with class accuracy.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

Citations in the corpus

Transformer Interpretability Beyond Attention Visualization (2020, methodology): “Perturbation methods [13, 14] consider the change to the decision of the network, as small changes are applied to the input.”
Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Survey (2019, methodology): “A discussion of some existing approaches to perturbation analysis is done in the study of [109], Figure 17.”
Information-Bottleneck Approach to Salient Region Discovery (2019, background): “A similar task of identifying salient regions in input samples is frequently addressed in the studies of neural network interpretation [5, 7, 8, 20, 22, 23, 31].”
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, methodology): “Perturbation To investigate important features in the input, a straightforward way is to measure the effect of perturbations applied to the input [37, 38].”
Visualization of Supervised and Self-Supervised Neural Networks via Attribution Guided Factorization (2020, methodology): “Extremal Perturbation methods (Fong, Patrick, and Vedaldi 2019; Fong and Vedaldi 2017) are applicable to black box models, but suffer from high computational complexity.”
Explaining by Removing: A Unified Framework for Model Explanation (2020, methodology): “Extremal Perturbations (Fong et al., 2019) is an extension of MP with several modifications.”
Toward Interpretable Machine Learning: Transparent Deep Neural Networks and Beyond (2020, methodology): “Both the TorchRay [37] and Captum [76] packages for Python and PyTorch enable the use of interpretability methods for neural network models deﬁned in context of PyTorch’s high level neural network description modules.”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, methodology): “Several authors [10, 6, 7, 79] use an auto-regressive waypoint network to predict differential waypoints.”
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications (2021, methodology): “Both the TorchRay [46] and Captum [95] packages for Python and PyTorch enable the use of interpretability methods for neural network models deﬁned in context of PyTorch’s high level neural network description modules.”
Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers (2021, methodology): “There are multiple families of explainability methods, including saliency-based methods [8, 34, 23, 49, 45, 48], methods that consider activations [10] using the forward pass or the backprop [46], perturbation based methods [11, 12], and methods based on Shapley-values [22, 6].”
Interpreting Super-Resolution Networks with Local Attribution Maps (2020, methodology): “For FSRCNN, the range of width is selected to be [16, 128] and the range of depth is [2, 16], with 15 models in total.”
Proactive Pseudo-Intervention: Causally Informed Contrastive Learning For Interpretable Vision Models (2020, background): “More complicated work generates an explanation by optimizing for a perturbed version of the image [18, 17, 10, 14].”