Interpretable Convolutional Neural Networks

By Quanshi Zhang, Ying Nian Wu, Song-Chun Zhu. 2017

851 citations recorded by the source corpus.

This paper proposes a method to modify a traditional convolutional neural network (CNN) into an interpretable CNN, in order to clarify knowledge representations in high conv-layers of the CNN. In an interpretable CNN, each filter in a high conv-layer represents a specific object part. Our interpretable CNNs use the same training data as ordinary CNNs without a need for any annotations of object parts or textures for supervision. The interpretable CNN automatically assigns each filter in a high conv-layer with an object part during the learning process. We can apply our method to different types of CNNs with various structures. The explicit knowledge representation in an interpretable CNN can help people understand the logic inside a CNN, i.e. what patterns are memorized by the CNN for prediction. Experiments have shown that filters in an interpretable CNN are more semantically meaningful than those in a traditional CNN. The code is available at https://github.com/zqs1022/interpretableCNN.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, positive): “To increase the interpretability of classical CNNs, the authors in [113] used a loss for each ﬁlter in high level convolutional layers to force each ﬁlter to learn very speciﬁc object components.”
Explaining Explanations: An Overview of Interpretability of Machine Learning (2018, background): “Special loss functions have been suggested for encouraging feed-forward networks to also disentangle their units; this can be used to create interpretable CNNs that have individual units that detect coherent meaningful patches instead of difﬁcult-to-interpret mixtures of patterns [67].”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “Even for deep neural networks for computer vision, even on the most challenging of benchmark datasets, a plethora of machine learning techniques have been designed that do not sacrifice accuracy but gain substantial interpretability (Zhang et al., 2018; Chen et al., 2019; Koh et al., 2020; Chen et a”
Unsupervised Learning of Neural Networks to Explain Neural Networks (2018, methodology): “For example, disentangling middlelayer representations of a CNN into object parts [44] may sometimes decrease the classification performance.”
Interpreting CNNs via Decision Trees (2018, methodology): “[40] proposed to learn CNNs with disentangled intermediate-layer representations.”
Visual interpretability for deep learning: a survey (2018, methodology): “Section 7 introduces methods to learn new models via human-computer interactions [Zhang et al. , 2017b ] and active question-answering with very limited human supervision [Zhang et al. , 2017a].”
Concept whitening for interpretable image recognition (2020, methodology): “rt and Denoyer(2019), who develop a concept-based deep learning method that is inherently interpretable, but it relies on speciﬁc properties of textual data that do not readily transfer to image data.Zhang et al. (2018) add losses to the ﬁlters to encourage them to detect object parts that can also ”
This looks like that: deep learning for interpretable image recognition (2018, background): “Recently there have also been attempts to quantify the interpretability of visual representations in a CNN, by measuring the overlap between highly activated image regions and labeled visual concepts [1, 51].”
Data-Free Learning of Student Networks (2019, background): “[31] assigned each filter in a higher convolution layer with a part of object, which demonstrates that each filter stands for different semantics.”
A Disentangling Invertible Interpretation Network for Explaining Latent Representations (2020, background): “[54] learn part specific convolutional filters by restricting filter activations to localized regions.”
XOC: Explainable Observer-Classifier for Explainable Binary Decisions (2019, methodology): “We ﬁx the CNN’s weights from the upper-bound model and use it to extract image features for the decision tree and our model.”
RAPID: Rapid and Precise Interpretable Decision Sets (2019, background): “While providing interpretation as a post-processing task especially for deep neural network models has received much attention [6]–[9], these models typically fall short of providing a fully “transparent” explanation of model predictions [10].”
Aggregating explanation methods for stable and robust explainability. (2019, methodology): “t evaluation methods. 2. Related Work The new-found interest into explainability is reﬂected in a lot of recent work into the problem (Kindermans et al.,2017; Selvaraju et al. ,2017 ;Bach et al. 2015 Zhang et al. 2018 Zhou et al. ,2016;Ancona et al. 2018 Ribeiro et al. Rieger et al. ,2018 ;Kim et al”
Explainability Methods for Graph Convolutional Neural Networks (2019, methodology): “Recently, explain-ability methods have been devised for deep networks and speciﬁcally CNNs [32, 42, 31, 39, 40, 41].”
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques (2019, background): “Methods that propose certain restrictions on the neural network architecture [12, 62] to make it inter-pretable, yet maintain richness of the hypothesis space to model complicated decision boundaries would fall under the global directly interpretable category.”
Combining Compositional Models and Deep Networks For Robust Object Classification under Occlusion (2019, background): “[22] show that part detectors can be encouraged in DCNNs by restricting the activations in feature maps to have a localized distribution.”
The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies (2020, background): “in a neural network to represent an object [55], or to develop neural networks with the structure of an additive index model [56].”
Explainable AI: A Brief Survey on History, Research Areas, Approaches and Challenges (2019, background): “In later work [25], the authors introduce additional losses to force each convolutional ﬁlter in CNN to represent a speciﬁc object part directly, and produce an interpretable CNN.”