Influence-Directed Explanations for Deep Convolutional Networks

By Klas Leino, Linyi Li, Shayak Sen, Anupam Datta, Matt Fredrikson. 2018

80 citations recorded by the source corpus.

We study the problem of explaining a rich class of behavioral properties of deep neural networks. Distinctively, our influence-directed explanations approach this problem by peering inside the network to identify neurons with high influence on a quantity and distribution of interest, using an axiomatically-justified influence measure, and then providing an interpretation for the concepts these neurons represent. We evaluate our approach by demonstrating a number of its unique capabilities on convolutional neural networks trained on ImageNet. Our evaluation demonstrates that influence-directed explanations (1) identify influential concepts that generalize across instances, (2) can be used to extract the “essence” of what the network learned about a class, and (3) isolate individual features the network uses to make decisions and distinguish related classes.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Model Fusion via Optimal Transport (2019, methodology): “Next, we use uniform distributions to initialize the histogram (or probability mass values) for each layer, although we note that it is possible to additionally use other measures of neuron importance (Dhamdhere et al., 2019; Datta et al., 2018; Sundararajan et al., 2017).”
The Limitations of Federated Learning in Sybil Settings (2020, background): “Recent work on feature influence in deep neural networks [1, 16, 32] may better capture the intent of sybil-based poisoning attacks in deep learning and we leave this analysis as future work.”
Towards Frequency-Based Explanation for Robust CNN (2020, methodology): “…yiliny2, ashriva2, vrawal, zihaod}@andrew.cmu.edu distribution of data with which the model is trained (e.g. Koh & Liang (2017), Yeh et al. (2018), Leino et al. (2018)); Among all behaviors of deep models, the robustness draws increasing attentions due to the security and fairness concerns.”
Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Models (2020, methodology): “, 2016) and for deep neural networks specifically (Leino et al., 2018; Sundararajan et al., 2017).”
From Local Explanations to Global Understanding with Explainable AI for Trees (2020, unknown): “Previous approaches to understanding a model globally have focused on using simple global approximations 2 , finding new inter-pretable features 20 or quantifying the impact of specific internal nodes in a deep network 21–23 (Supplementary Results 2).”
Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference (2019, background): “The unique influence measure satisfying the first three properties is internal influence [25], given by Equation 9.”
Stakeholders in Explainable AI (2018, methodology): “…seeks to place the ﬁeld on a more rigorous scientiﬁc and engineering basis by, for example, examining axiomatic approaches to model inter-pretability (Leino et al. 2018; Sundararajan, Taly, and Yan 2017), exploring more sophisticated methods for revealing the inner workings of deep networks (Olah…”
Feature-Wise Bias Amplification (2018, methodology): “To extend this notion to internal layers of a deep network h, we consider the slice abstraction (Leino et al., 2018) comprised of a pair of functions f : X0 → X , and g : X → R, such that h = g ◦ f .”