Network Dissection: Quantifying Interpretability of Deep Visual Representations

By David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, Antonio Torralba. 2017

1,795 citations recorded by the source corpus.

We propose a general framework called Network Dissection for quantifying the interpretability of latent representations of CNNs by evaluating the alignment between individual hidden units and a set of semantic concepts. Given any CNN model, the proposed method draws on a data set of concepts to score the semantics of hidden units at each intermediate convolutional layer. The units with semantics are labeled across a broad range of visual concepts including objects, parts, scenes, textures, materials, and colors. We use the proposed method to test the hypothesis that interpretability is an axis-independent property of the representation space, then we apply the method to compare the latent representations of various networks when trained to solve different classification problems. We further analyze the effect of training iterations, compare networks trained with different initializations, and measure the effect of dropout and batch normalization on the interpretability of deep visual representations. We demonstrate that the proposed method can shed light on characteristics of CNN models and training methods that go beyond measurements of their discriminative power.

Subfields: Artificial Intelligence, Biophysics, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Natural Adversarial Examples (2019, background): “, 2017 [3] argue that Places365 classifiers learn qualitatively distinct filters (e.”
Unified Perceptual Parsing for Scene Understanding (2018, methodology): “1 Datasets In order to accomplish segmentation of a wide range of visual concepts from multiple levels, we utilize the Broadly and Densely Labeled Dataset (Broden) [26], a heterogeneous dataset that contains various visual concepts.”
Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning (2017, neutral): “For example, Bau et al (2017) quantiﬁed the number of object-selective units in our network, as well as other recent networks trained with self-supervised learning.”
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization (2016, methodology): “We devise a way to identify important neurons through GradCAM and combine it with neuron names [4] to provide textual explanations for model decisions.”
Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “For quantifying the interpretability of the latent representations of CNNs, the authors in [125] used a diﬀerent approach called network dissection.”
Explaining Explanations: An Overview of Interpretability of Machine Learning (2018, methodology): “One example of a such a method is network dissection [51], which measures the ability of individual units solve a segmentation problem over a broad set of labeled visual concepts.”
Dataset Distillation (2018, methodology): “Researchers have presented various approaches for understanding and visualizing learned models (Zeiler & Fergus, 2014; Zhou et al., 2015; Mahendran & Vedaldi, 2015; Bau et al., 2017; Koh & Liang, 2017).”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, background): “Reference [36] suggests network dissection for the interpretability of visual representations and offers a way to quantify it as well.”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, background): “[152] have a different perception of explainability, they see it as a latent property that can be influenced by different manipulable factors (such as the number of inputs, the complexity of the model, or even the user interface) and that affects different measurable outcomes (such as an end user’s ”
Unsupervised Learning of Neural Networks to Explain Neural Networks (2018, background): “However, as mentioned in [2], an interpretable model cannot always ensure a high discrimination power, which limits the applicability of above interpretable models.”
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning (2018, background): “Nevertheless, legislation like the European Union’s General Data Protection Regulation require that companies deploying ML and other forms of analysis on certain sensitive data provide such interpretable outputs if the prediction made by a ML model is used to make decisions without a human in the lo”
Interpreting CNNs via Decision Trees (2018, background): “As analyzed in [2], filters in high conv-layers are more prone to represent object parts, while those in low convlayers usually describe textures.”
Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) (2017, methodology): “This simple approach is supported by recent work using local linearity (Alain & Bengio, 2016; Raghu et al., 2017; Bau et al., 2017; Szegedy et al., 2013).”
Interpretable Convolutional Neural Networks (2017, methodology): “As discussed in [2], besides the discrimination power, the interpretability is another crucial property of a network.”
What Does Explainable AI Really Mean? A New Conceptualization of Perspectives (2017, neutral): “High dimensional data visualizations like t-SNE [10] and receptive ﬁeld visualization on convolutional neural networks [2] are examples of comprehensible models.”
Methods for interpreting and understanding deep neural networks (2017, methodology): “Furthermore, while some methods aim at reaching a comprehensive functional understanding of the model [32,41,8], we will focus here on interpreting the outputs of a DNN, and explaining individual predictions.”
Visual interpretability for deep learning: a survey (2018, methodology): “In this section, we simply introduce two latest evaluation metrics for the interpretability of CNN ﬁlters, i.e. the ﬁlter interpretability proposed by [Bau et al. , 2017] and the location instability proposed by [Zhang et al. , 2018a].”
Learning-based Video Motion Magnification (2018, background): “Deep neural networks achieve high performance in a wide variety of vision tasks, but their inner working is still largely unknown [27].”