Understanding Neural Networks Through Deep Visualization

By Jason Yosinski, Jeff Clune, Anh Totti Nguyen, Thomas J. Fuchs, Hod Lipson. 2015

1,959 citations recorded by the source corpus.

Recent years have produced great advances in training large, deep neural networks (DNNs), including notable successes in training convolutional neural networks (convnets) to recognize natural images. However, our understanding of how these models work, especially what computations they perform at intermediate layers, has lagged behind. Progress in the field will be further accelerated by the development of better tools for visualizing and interpreting neural nets. We introduce two such tools here. The first is a tool that visualizes the activations produced on each layer of a trained convnet as it processes an image or video (e.g. a live webcam stream). We have found that looking at live activations that change in response to user input helps build valuable intuitions about how convnets work. The second tool enables visualizing features at each layer of a DNN via regularized optimization in image space. Because previous versions of this idea produced less recognizable images, here we introduce several new regularization methods that combine to produce qualitatively clearer, more interpretable visualizations. Both tools are open source and work on a pre-trained convnet with minimal setup.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Unrolled Generative Adversarial Networks (2016, background): “(14) This concept of backpropagating to generate images has been widely used in visualizing features from discriminative networks (Simonyan et al., 2013; Yosinski et al., 2015; Nguyen et al., 2016) and has been applied to explore the visual manifold of GANs in (Zhu et al.”
Generating Images with Perceptual Similarity Metrics based on Deep Networks (2016, background): “Mahendran & Vedaldi (2015), as well as Simonyan et al. (2014) and Yosinski et al. (2015), apply gradient-based and Mahendran & Vedaldi (2015).”
Axiomatic Attribution for Deep Networks (2017, background): “…been on networks trained on computer vision tasks, and deals with understanding what a speciﬁc neuron computes (Erhan et al., 2009; Le, 2013) and interpreting the representations captured by neurons during a prediction (Mahendran & Vedaldi, 2015; Dosovitskiy & Brox, 2015; Yosinski et al., 2015).”
Perceptual Losses for Real-Time Style Transfer and Super-Resolution (2016, background): “Images can be generated to maximize class prediction scores [8,9] or individual features [9] in order to understand the functions encoded in trained networks.”
Deep Clustering for Unsupervised Learning of Visual Features (2018, methodology): “[69] with a cross entropy function between the target filter and the other filters of the same layer.”
Synthesizing the preferred inputs for neurons in neural networks via deep generator networks (2016, methodology): “The DNN being visualized is the same as in [9] (the CaffeNet architecture with weights from [7]).”
Attribute2Image: Conditional Image Generation from Visual Attributes (2015, methodology): “Our algorithm actually shares a similar spirit with recently proposed neural network visualization [42] and texture synthesis algorithms [6].”
Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks (2019, background): “Interpreting DNN predictions and activations in internal layers is still an open research challenge [18], and finding a heuristic that generalizes across DNNs is difficult.”
SentiNet: Detecting Localized Universal Attacks Against Deep Learning Systems (2018, neutral): “everal techniques have been proposed to explain and interpret the prediction of a model. One strategy attempts to “quantify” the relevance of individual pixels of the input, e.g., using saliency maps [46]. While effective in practice, focusing on individual pixels may result in a mask of non-contigu”
DeepXplore: Automated Whitebox Testing of Deep Learning Systems (2017, background): “It has recently been shown that each neuron in a DNN tends to independently extract a specific feature of the input instead of collaborating with other neurons for feature extraction [58, 87].”
Towards Open Set Deep Networks (2015, background): “Adversarial Set: Finally, consider an adversarial input image [8, 27, 31], which is constructed to be close to one class but is mislabeled as another.”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, methodology): “[37] 7 X Optimization + Activation maximization on DBN (Deep Belief Network) [75] 7 X + Activation maximization, multifaceted feature visualization [76] 7 X Visualization via regularized optimization [77] 7 X Semantic dictionary [38] 7 X Decision trees N.A. N.A. Propositional logic, rule-based [81] ”
Generative Counterfactual Introspection for Explainable Deep Learning (2019, methodology): “Most existing model introspection approaches [4]–[6] focus on studying the correlation between inputs and outputs (or predictions), e.”