A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations

By Weili Nie, Yang Zhang, Ankit B. Patel. 2018

170 citations recorded by the source corpus.

Backpropagation-based visualizations have been proposed to interpret convolutional neural networks (CNNs), however a theory is missing to justify their behaviors: Guided backpropagation (GBP) and deconvolutional network (DeconvNet) generate more human-interpretable but less class-sensitive visualizations than saliency map. Motivated by this, we develop a theoretical explanation revealing that GBP and DeconvNet are essentially doing (partial) image recovery which is unrelated to the network decisions. Specifically, our analysis shows that the backward ReLU introduced by GBP and DeconvNet, and the local connections in CNNs are the two main causes of compelling visualizations. Extensive experiments are provided that support the theoretical analysis.

Subfields: Artificial Intelligence, Biophysics, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Interpretations are useful: penalizing explanations to align neural networks with prior knowledge (2019, background): “However, many of these methods have been poorly evaluated so far (Adebayo et al., 2018; Nie et al., 2018), casting doubt on their usefulness in practice.”
Sanity Checks for Saliency Maps (2018, methodology): “[27] theoretically assessed backpropagation based methods and found that Guided BackProp and DeconvNet, under certain conditions, are invariant to network reparamaterizations, particularly random Gaussian initialization.”
Definitions, methods, and applications in interpretable machine learning (2019, background): “Moreover, it has been shown that some feature importance scores for CNNs are doing (partial) image recovery which is unrelated to the network decisions (81).”
Learning Perceptually-Aligned Representations via Adversarial Robustness (2019, background): “In fact, explanations can often be model and data independent [NZP18; Ade+18] hence being inherently unable to explain the predictions of the model.”
Gradient-Based Attribution Methods (2019, background): “However, the lack of a ground-truth explanation makes it very challenging to evaluate these methods quantitatively, while qualitative evaluations have been shown to be insufficient or biased [2, 8, 20].”
Explaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximation (2019, background): “In fact, some recent works showed that attribution methods can actually produce un-reliable or misleading results, despite these being visually appealing to humans (Kindermans et al., 2017; Ghorbani et al., 2019; Adebayo et al., 2018; Nie et al., 2018).”
Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking (2020, methodology): “They do not guarantee that the model ignores low-scoring features, attracting criticism in recent years (Nie et al., 2018; Sixt et al., 2019; Jain & Wallace, 2019).”
Full-Gradient Representation for Neural Network Visualization (2019, background): “These include unintuitive properties such as being insensitive to model randomization [6], partly recovering the input [12] or being insensitive to the model’s invariances [5].”
Multiresolution Knowledge Distillation for Anomaly Detection (2020, methodology): “Several methods [2, 30] reveal some ﬂaws in GBP by demonstrating that it reconstructs the image instead of explaining the outcome function.”
Understanding Impacts of High-Order Loss Approximations and Features in Deep Learning Interpretation (2019, background): “Although these gradient-based methods can produce visually pleasing results, they often weakly approximate the underlying model (Feng et al., 2018; Nie et al., 2018).”
Debugging Tests for Model Explanations (2020, result): “The ﬁnding that methods that modify backprop combined with positive aggregation are invariant to higher layer parameters corroborates the recent work of Sixt et al. [60] along with previous evidence by Nie et al. [44] and Mahendran and Vedaldi [38].”
Bridging Adversarial Robustness and Gradient Interpretability (2019, background): “Although they produce sharp visualizations, they have been proven to be doing partial image recovery which is unrelated to DNN decisions (Nie et al., 2018).”