Full-Gradient Representation for Neural Network Visualization

By Suraj Srinivas, Franccois Fleuret. Neural Information Processing Systems, 2019

366 citations recorded by the source corpus.

We introduce a new tool for interpreting neural nets, namely full-gradients, which decomposes the neural net response into input sensitivity and per-neuron sensitivity components. This is the first proposed representation which satisfies two key properties: completeness and weak dependence, which provably cannot be satisfied by any saliency map-based interpretability method. Using full-gradients, we also propose an approximate saliency map representation for convolutional nets dubbed FullGrad, obtained by aggregating the full-gradient components. We experimentally evaluate the usefulness of FullGrad in explaining model behaviour with two quantitative tests: pixel perturbation and remove-and-retrain. Our experiments reveal that our method explains model behavior correctly, and more comprehensively than other methods in the literature. Visual inspection also reveals that our saliency maps are sharper and more tightly confined to object regions than other methods.

Subfields: Artificial Intelligence, Biophysics

View Paper: https://www.semanticscholar.org/paper/0e09d47723cccf0a9bad7cb1badaa367ea37fe14

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

Citations in the corpus

Saliency Guided Experience Packing for Replay in Continual Learning (2021, methodology): “To assess the sensitivity of EPR to the choice of saliency algorithms, we also used Grad-CAM++ [6], Smooth-Grad [46] and FullGrad [47] as saliency method in EPR.”
XOC: Explainable Observer-Classifier for Explainable Binary Decisions (2019, background): “FullGrad [37] combines importance scores of both the intermediate feature maps and the input to capture both local and global relevance with respect to the network’s output.”
Improving Emotional TTS with an Emotion Intensity Input from Unsupervised Extraction (2021, background): “An increasing number of techniques exist with varying complexity [18, 19, 20, 21, 22]; we discuss some below.”
Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers (2021, neutral): “Examples include class agnostic methods [33, 37, 35, 36].”
Visualization of Supervised and Self-Supervised Neural Networks via Attribution Guided Factorization (2020, methodology): “As can be seen, LRP, FullGrad and RAP output similar visualizations for both classes.”
IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers (2021, methodology): “After that, gradientbased methods [49, 46, 47] are proposed to generate precise heatmaps, by computing the gradient with respect to input during the backpropagation.”
Transformer Interpretability Beyond Attention Visualization (2020, methodology): “The FullGrad method [37] offers a more complete modeling of the gradient by also considering the gradient with respect to the bias term, and not just with respect to the input.”