Interpreting CNNs via Decision Trees

By Quanshi Zhang, Yu Yang, Ying Nian Wu, Song-Chun Zhu. 2018

353 citations recorded by the source corpus.

This paper aims to quantitatively explain the rationales of each prediction that is made by a pre-trained convolutional neural network (CNN). We propose to learn a decision tree, which clarifies the specific reason for each prediction made by the CNN at the semantic level. I.e., the decision tree decomposes feature representations in high conv-layers of the CNN into elementary concepts of object parts. In this way, the decision tree tells people which object parts activate which filters for the prediction and how much each object part contributes to the prediction score. Such semantic and quantitative explanations for CNN predictions have specific values beyond the traditional pixel-level analysis of CNNs. More specifically, our method mines all potential decision modes of the CNN, where each mode represents a typical case of how the CNN uses object parts for prediction. The decision tree organizes all potential decision modes in a coarse-to-fine manner to explain CNN predictions at different fine-grained levels. Experiments have demonstrated the effectiveness of the proposed method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical Learning Theory

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “Informativeness [5, 44, 21, 25, 26, 45, 30, 32, 34, 35, 37, 38, 41, 46, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 63, 64, 65, 66, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 86, 87, 88, 89, 59, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 1”
Unsupervised Learning of Neural Networks to Explain Neural Networks (2018, methodology): “We used [45] to compute quantitative contributions of different parts to the classification score on the right.”
Visual interpretability for deep learning: a survey (2018, methodology): “[Zhang et al., 2018c] has further proposed a decision tree to encode decision modes in fully-connected layers.”
Neural Prototype Trees for Interpretable Fine-grained Image Recognition (2020, methodology): “Other tree approaches for image classification apply post-hoc explanation techniques, by showing example images per node [2, 55], visualizing typical CNN filters of each node that can be manually labelled [55], showing class activation maps [25] or manual inspection of leaf labels and the meaning of”
Conservative Q-Improvement: Reinforcement Learning for an Interpretable Decision-Tree Policy (2019, methodology): “Some work has explored combining neural nets and decision tree forms [Tanno et al. , 2018; Zhao, 2001], including using decision trees to explain neural nets [Zhang et al. , 2018].”
XOC: Explainable Observer-Classifier for Explainable Binary Decisions (2019, methodology): “We ﬁx the CNN’s weights from the upper-bound model and use it to extract image features for the decision tree and our model.”
Explainable Artificial Intelligence: a Systematic Review (2020, methodology): “Finally, [245] and [246] proposed two algorithms that extract DTs from the weights of a DNN.”
Abduction-Based Explanations for Machine Learning Models (2018, background): “The ﬁrst line focuses on explaining a complex ML model as a whole using an interpretable model (Frosst and Hinton 2017; Zhang et al. 2018).”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “Techniques like creating programs in domain-specific languages (Verma et al., 2019b;a; Trivedi et al., 2021), extracting decision trees (Zhang et al., 2019) or symbolic causal graphs (Ren et al., 2023) from neural networks align well with the goals of mechanistic interpretability.”
Visualizing the decision-making process in deep neural decision forest (2019, background): “Another orthogonal line of research attempts to learn more interpretable representation [9] and organize the inference process into a decision tree [10].”
Questioning the AI: Informing Design Practices for Explainable AI User Experiences (2020, background): “…to changes of a feature (often in a visualization format) [8, 33, 36, 51] What if , How to be that, How to still be this Contrastive or counterfactual features Describe the feature(s) that will change the prediction if perturbed, absent or present [27, 91, 100] Why , Why not , How to be that”
Explainable artificial intelligence: a comprehensive review (2021, background): “The evidence for the mentioned information can be witnessed through the increasing number of research on the generation and explanation of the decision tree (Sagi and Rokach 2020; Främling 2020; Zhang et al. 2019).”
SegNBDT: Visual Decision Rules for Segmentation (2020, background): “Work in this area is limited to [12, 36] with complementary downfalls: Li et al.”
Evaluating Explanation Without Ground Truth in Interpretable Machine Learning (2019, background): “For example, in literature ( 21 ), the authors employ a set of decision trees as the interpretable proxy to explain the working mechanism of neural networks, where the trees are learned by summarizing common generic decision modes.”
Interpreting the Latent Space of GANs via Correlation Analysis for Controllable Concept Manipulation (2020, background): “categories according to the corresponding interpreting results: visualizing pre-trained models[10][11], diagnose pre-trained models[12][13] and construct interpretable models[14][15].”
Trustworthy AI: From Principles to Practices (2021, background): “With the rise of deep learning in the past decade, explainer approximation on the DNN has advanced as the knowledge distillation problem on explainers such as trees [125, 384].”
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, background): “However, this rationale process can be proxied by graph models [138] or decision trees [140], which provide relatively a more interpretable rationale path to humans.”