Interpreting CNN knowledge via an Explanatory Graph

By Quanshi Zhang, Ruiming Cao, Feng Shi, Ying Nian Wu, Song-Chun Zhu. 2017

256 citations recorded by the source corpus.

This paper learns a graphical model, namely an explanatory graph, which reveals the knowledge hierarchy hidden inside a pre-trained CNN. Considering that each filter in a conv-layer of a pre-trained CNN usually represents a mixture of object parts, we propose a simple yet efficient method to automatically disentangles different part patterns from each filter, and construct an explanatory graph. In the explanatory graph, each node represents a part pattern, and each edge encodes co-activation relationships and spatial relationships between patterns. More importantly, we learn the explanatory graph for a pre-trained CNN in an unsupervised manner, i.e., without a need of annotating object parts. Experiments show that each graph node consistently represents the same object part through different images. We transfer part patterns in the explanatory graph to the task of part localization, and our method significantly outperforms other approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Unsupervised Learning of Neural Networks to Explain Neural Networks (2018, background): “[40] also disentangled feature representations in middle layers of a CNN into a graphical model of object parts in an unsupervised manner.”
Interpreting CNNs via Decision Trees (2018, methodology): “Theoretically, our method also supports other techniques of mining interpretable features in middle layers [38, 27].”
Interpretable Convolutional Neural Networks (2017, methodology): “The second metric measures the instability of part locations, which was proposed in [35].”
Visual interpretability for deep learning: a survey (2018, methodology): “[Zhang et al. , 2018b].”
Examining CNN representations with respect to Dataset Bias (2017, methodology): “Methods of (Bansal, Farhadi, and Parikh 2014; Zhang et al. 2014) computed the distributions of a CNN’s prediction errors among testing samples, in order to summarize failure modes of the CNN.”
Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction (2018, background): “…of the output 100 , (2) analyze the function of intermediate layers 94 , (3) tease apart contributions from different network layers with ablation studies 94 , (4) build hierarchical explanatory graphs across layers 101 , and (5) design network architectures that are interpretable by design 102 .”
Definitions, methods, and applications in interpretable machine learning (2019, methodology): “When visualizing convolutional neural networks trained on image data, work has been done on visualizing filters (62, 63), maximally activating responses of individual neurons or classes (64), understanding intra-class variation (65), and grouping different neurons (66).”
Interactively Transferring CNN Patterns for Part Localization (2017, methodology): “The AOG has been a typical object representation for years [11,22]. Here, we brieﬂy introduce the AOG structure, which has been widely used to represent neural patterns of CNNs in studies of [19] and [18]. The AOG organizes latent patterns hidden in the CNN to explain the semantic hierarchy of an ob”
Explaining Knowledge Distillation by Quantifying the Knowledge (2020, methodology): “used an explanatory graph [48] and a decision tree [50] to represent hierarchical compositional part representations in CNNs.”
Interpretable Convolutional Filters with SincNet (2018, background): “In recent years, a growing interest has been thus devoted to the development of interpretable machine learning [6, 7], as witnessed by the numerous works in the field, ranging from visualization [8, 9], diagnosis of DNNs [10], explanatory graphs [11], and explainable models [12], just to name a few.”
Evaluating Explanation Without Ground Truth in Interpretable Machine Learning (2019, background): “Representative examples can be found in knowledge distillation ( 17–19 ) and mimic learning ( 20–22 ), where the common focus is to obtain an interpretable model out of the black-box neural model.”
A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and concepts (2021, methodology): “Explanatory Graphs [146] Their idea is that semantic concepts in an image usually consist of sub-objects to which they have a constant relative spatial relation (e.”
Explainable Artificial Intelligence: a Systematic Review (2020, background): “Similarly, Explanatory Graph [227] produces graphs from CNNs where each node represents a ‘part pattern’, which correspond to the peak activation in a layer related to a part of the input, and each edge connects two nodes in adjacent layers to encode co-activation relationships and spatial relations”
Hierarchical interpretations for neural network predictions (2018, methodology): “Interpreting DNNs is a growing field (Murdoch et al., 2019) spanning a range of techniques including feature visualization (Olah et al., 2017; Yosinski et al., 2015), analyzing learned weights (Tsang et al., 2017) and others (Frosst & Hinton, 2017; Andreas et al., 2016; Zhang et al., 2017).”
Semantic bottleneck for computer vision tasks (2018, background): “[34] and [35] described geometric metrics to assess the quality of the visual explanation with respect to landmarks or objects in the image.”
Explainability in Deep Reinforcement Learning (2020, background): “(interpretable) representations of the conv-layers of these networks [114, 113], as well as end-to-end learning of interpretable networks, working directly with comprehensible patterns, which are also a trending angle [108].”