Understanding the role of individual units in a deep neural network

By David Bau, Jun-Yan Zhu, Hendrik Strobelt, Àgata Lapedriza, Bolei Zhou, Antonio Torralba. Proceedings of the National Academy of Sciences of the United States of America, 2020

535 citations recorded by the source corpus.

Deep neural networks excel at finding hierarchical representations that solve complex tasks over large datasets. How can we humans understand these learned representations? In this work, we present network dissection, an analytic framework to systematically identify the semantics of individual hidden units within image classification and image generation networks. First, we analyze a convolutional neural network (CNN) trained on scene classification and discover units that match a diverse set of object concepts. We find evidence that the network has learned many object classes that play crucial roles in classifying scene classes. Second, we use a similar analytic method to analyze a generative adversarial network (GAN) model trained to generate scenes. By analyzing changes made when small sets of units are activated or deactivated, we find that objects can be added and removed from the output scenes while adapting to the context. Finally, we apply our analytic framework to understanding adversarial attacks and to semantic image editing.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/973c01a6458928a1b3797daf13298969f03c0a1f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets (2023, background): “Early work focused on whether individual neurons represent features (Wang et al., 2022; Sajjad et al., 2022; Bau et al., 2020), but features may more generally be represented by directions in a LLM’s latent space (i.e. linear combinations of neurons) (Dalvi et al., 2018; Gurnee et al., 2023;…”
Nine Potential Pitfalls when Designing Human-AI Co-Creative Systems (2021, background): “teractively in creative contexts and for digital content creation and manipulation: Some examples include (1) generating or modifying images with generative adversarial networks (GANs) [1, 2, 3], (2) generating controllable movements for virtual characters with re-”
Unsupervised Point Cloud Pre-training via Occlusion Completion (2020, methodology): “tically meaningful representations as described via network dissection [4, 5]; 5) Better clustering quality under jittering, translation, and rotation transformations.”
Acquisition of chess knowledge in AlphaZero (2021, background): “This remarkable fact (which is also implicitly used by concept-based approaches) is probed in further detail by ‘network dissection’ approaches [1, 7].”
Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, background): “Although the literature contains many examples of such seemingly monosemantic neurons [6, 7, 5, 8, 9, 10], an obvious problem arises when a network has to represent more features than it has neurons.”
HyperInverter: Improving StyleGAN Inversion via Hypernetwork (2021, background): “Several studies [7, 16, 40, 41, 50] have been conducted to examine the latent spaces learned by GANs, which convey a wide range of interpretable semantics.”
Emergent Linear Representations in World Models of Self-Supervised Sequence Models (2023, background): “Yet evidence has accumulated that models extract features – articulable properties of the input 2 – and represent them in its internal activations (Geva et al., 2021; Bau et al., 2020; Gurnee et al., 2023; Belinkov, 2022; Burns et al., 2022; Goh et al., 2021; Elhage et al., 2022a).”
Quo vadis artificial intelligence? (2022, background): “For example, a typical CNN (Convolutional Neural Network) block usually involves the processes of convolution, pooling, normalization, activation, and a fully connected layer [49].”
Obtaining genetics insights from deep learning via explainable artificial intelligence (2022, background): “…to learn semantically interpretable features with increasing levels of abstraction 27 presence of edges in input images at various orientations on a fine scale, and then the deeper layers use combinations of edge detectors to identify higher-level image parts such as ears, nose and mouth 28 .”
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications (2021, background): “Orthogonal to the quest for better and more holistic machine learning models, Explainable AI (XAI) [159], [73], [112], [128], [18] has developed as a subﬁeld of machine learning that seeks to augment the training process, the learned representations and the decisions with human-interpretable explana”