Interpreting Deep Visual Representations via Network Dissection

By Bolei Zhou, David Bau, Aude Oliva, Antonio Torralba. 2017

361 citations recorded by the source corpus.

The success of recent deep convolutional neural networks (CNNs) depends on learning hidden representations that can summarize the important factors of variation behind the data. In this work, we describe Network Dissection, a method that interprets networks by providing meaningful labels to their individual units. The proposed method quantifies the interpretability of CNN representations by evaluating the alignment between individual hidden units and visual semantic concepts. By identifying the best alignments, units are given interpretable labels ranging from colors, materials, textures, parts, objects and scenes. The method reveals that deep representations are more transparent and interpretable than they would be under a random equivalently powerful basis. We apply our approach to interpret and compare the latent representations of several network architectures trained to solve a wide range of supervised and self-supervised tasks. We then examine factors affecting the network interpretability such as the number of the training iterations, regularizations, different initialization parameters, as well as networks depth and width. Finally we show that the interpreted units can be used to provide explicit explanations of a given CNN prediction for an image. Our results highlight that interpretability is an important property of deep neural networks that provides new insights into what hierarchical structures can learn.

Subfields: Artificial Intelligence, Biophysics, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Safety, Robustness & Interpretability

Citations in the corpus

Unified Perceptual Parsing for Scene Understanding (2018, background): “Nevertheless, since it is originally collected to discover the alignment between visual concepts and hidden units of Convolutional Neural Networks (CNNs) for network interpretability [26,29], we ﬁnd that samples from diﬀerent classes are unbalanced.”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “Even if we create a vector in the latent space that is aimed towards a single concept (as is done in Kim et al., 2018; Zhou et al., 2018b), that vector could activate highly on multiple concepts, which means the signal for the two concepts is not disentangled.”
Multimodal Explanations: Justifying Decisions and Pointing to the Evidence (2018, methodology): “Some methods find discriminative visual patches [7, 11] whereas others aim to understand what specific neurons represent [12, 38, 39].”
Interpretable Basis Decomposition for Visual Explanation (2018, methodology): “We derive an interpretable basis from the fully annotated image dataset Broden [26].”
GANalyze: Toward Visual Definitions of Cognitive Image Properties (2019, methodology): “The internal representations of a CNN can be unveiled using methods like network dissection [14, 15, 16] including for a CNN trained on memorability [5].”
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications (2021, neutral): “Further methods do not seek to explain in terms of input features but in terms of the latent space, where the directions in the latent space code for higher level concepts, such as color, material, object part, or object [17], [18], [205].”
Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing (2019, methodology): “[64] analyze the neuron activations on pixel-level annotated datasets, and quantify the network interpretability by correlating the neuron activations with groundtruth annotations.”
Visualization of Supervised and Self-Supervised Neural Networks via Attribution Guided Factorization (2020, background): “…classes include those that generate salient feature maps (Dabkowski and Gal 2017; Simonyan, Vedaldi, and Zisserman 2013; Mahendran and Vedaldi 2016; Zhou et al. 2016; Zeiler and Fergus 2014; Zhou et al. 2018), Activation Maximization (Erhan et al. 2009) and Excitation Backprop (Zhang et al. 2018).”
Understanding the role of individual units in a deep neural network (2020, methodology): “To investigate these questions, we introduce network dissection (9, 10), our method for systematically mapping the semantic concepts found within a deep convolutional neural network (CNN).”
Towards Non-I.I.D. image classification: A dataset and baselines (2019, background): “What’s more, the ability of adjusting distribution shift controllably can indeed bring more explainabil-ity to the models and experiments, expecially for deep learning [10,11] .”
Learning Compositional Representations for Few-Shot Recognition (2018, background): “There has been a lot of effort for qualitative interpretation of these representations [45, 46].”
Transformer Interpretability Beyond Attention Visualization (2020, background): “Methods that do not fall into these two main categories include saliency based methods [8, 35, 25, 48, 45, 47], Activation Maximization [12] and Excitation Backprop [46].”
Explainable Neural Computation via Stack Neural Module Networks (2018, background): “One line of research in this direction attempts to generate post-hoc explanations of decisions from generic model architectures, either by finding interpretable local surrogates in the form of linear models [29], logical rules [9,38] or natural language descriptions [2,39], or by visualizing salient”
Attribute Prototype Network for Zero-Shot Learning (2020, background): “While modern deep neural networks [13] encode local information and some CNN neurons are linked to object parts [53], the encoded local information is not necessarily best suited for zero-shot learning.”
Evaluating Explanation Without Ground Truth in Interpretable Machine Learning (2019, methodology): “As for the annotations with semantic segmentation, recent work [40] employs the pixel-level difference as the metric to measure the corresponding persuasibility of explanations.”