Hierarchical interpretations for neural network predictions

By Chandan Singh, W. James Murdoch, Bin Yu. 2018

162 citations recorded by the source corpus.

Deep neural networks (DNNs) have achieved impressive predictive performance due to their ability to learn complex, non-linear relationships between variables. However, the inability to effectively visualize these relationships has led to DNNs being characterized as black boxes and consequently limited their applications. To ameliorate this problem, we introduce the use of hierarchical interpretations to explain DNN predictions through our proposed method, agglomerative contextual decomposition (ACD). Given a prediction from a trained DNN, ACD produces a hierarchical clustering of the input features, along with the contribution of each cluster to the final prediction. This hierarchy is optimized to identify clusters of features that the DNN learned are predictive. Using examples from Stanford Sentiment Treebank and ImageNet, we show that ACD is effective at diagnosing incorrect predictions and identifying dataset bias. Through human experiments, we demonstrate that ACD enables users both to identify the more accurate of two DNNs and to better trust a DNN's outputs. We also find that ACD's hierarchy is largely robust to adversarial perturbations, implying that it captures fundamental aspects of the input and ignores spurious noise.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

SELFEXPLAIN: A Self-Explaining Architecture for Neural Text Classifiers (2021, methodology): “Trustability: In addition to plausibility, we also evaluate user trust of the explanations (Singh et al., 2019; Jin et al., 2020).”
Interpretations are useful: penalizing explanations to align neural networks with prior knowledge (2019, background): “, 2018), and using those interactions to create a hierarchy of features displaying the model’s prediction process (Singh et al., 2019; 2020).”
Definitions, methods, and applications in interpretable machine learning (2019, background): “Researchers have proposed a variety of interpretation forms, including feature heatmaps (71), feature hierarchies (82) and identifying important elements in the training set (69).”
Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignment (2019, background): “Furthermore, we follow the addition of Singh et al. (2019) of only adding the intercept interactions b-b to the inside part if the current time step is part of the phrase in focus, which we denote as b-b ∈ x.”
Transformation Importance with Applications to Cosmology (2020, methodology): “In the work here, the choice of attribution method attr is CD, as it can disentangle the importance of features and their interactions, and has been rigorously evaluated using real data [1], human experiments [23], and during model training [24].”
Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking (2020, methodology): “Outside of graph-speciﬁc methods, one line of research involves decomposing the output into a part attributed to a speciﬁc subset of features and a part attributed to the remaining features (Shapley, 1953; Murdoch et al., 2019; Singh et al., 2019; Jin et al., 2020).”
Explaining Patterns in Data with Language Models via Interpretable Autoprompting (2022, methodology): “…interpretation A popular method for interpreting neural networks is to inspect an LLM’s individual predictions via feature importances [29, 30], feature-interaction importances [31, 32], extractive rationales [33, 34], or natural-language explanations for individual predictions [35, 36].”
TOWARDS HIERARCHICAL IMPORTANCE ATTRIBU- (2019, methodology): “As a follow-up study, Singh et al. (2019) extends CD algorithm to other families of neural network architectures, and proposes agglomerative contextual decomposition algorithm (ACD).”
Contextualizing Hate Speech Classifiers with Post-hoc Explanation (2020, background): “, 2017) or phrase-level importance (Murdoch et al., 2018; Singh et al., 2019) of inputs to predictions.”