Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks

By Hendrik Strobelt, Sebastian Gehrmann, Bernd Huber, Hanspeter Pfister, Alexander M. Rush. 2016

85 citations recorded by the source corpus.

Recurrent neural networks, and in particular long short-term memory networks (LSTMs), are a remarkably effective tool for sequence modeling that learn a dense black-box hidden representation of their sequential input. Researchers interested in better understanding these models have studied the changes in hidden state representations over time and noticed some interpretable patterns but also significant noise. In this work, we present LSTMVis a visual analysis tool for recurrent neural networks with a focus on understanding these hidden state dynamics. The tool allows a user to select a hypothesis input range to focus on local state changes, to match these states changes to similar patterns in a large data set, and to align these results with domain specific structural annotations. We further show several use cases of the tool for analyzing specific hidden state properties on datasets containing nesting, phrase structure, and chord progressions, and demonstrate how the tool can be used to isolate patterns for further statistical analysis.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure (2017, neutral): “…(2016) study cell and gate activations under diﬀerent conditions; Tang et al. (2017) plot distributions of cell activations and temporal traces of t-SNE cell vectors of a speech recognition RNN; and Strobelt et al. (2016) present an elaborate tool 3 that facilitates visual analysis of gated RNNs.”
Understanding Neural Networks through Representation Erasure (2016, background): “Strobelt et al. (2016) develop an interactive system that allows users to select LSTM intermediate states and align these state changes to domain specific structural annotations.”
Automatic Rule Extraction from Long Short Term Memory Networks (2016, background): “First, Hendrik et al. (2016) and Karpathy et al. (2016) analyse the movement of the raw gate activations over a sequence.”
Recurrent Neural Networks Approach to the Financial Forecast of Google Assets (2017, positive): “This idea is inspired by the LSTMVis tool, see [18], that can be used to understand hidden state dynamics in LSTM.”
Opportunities and obstacles for deep learning in biology and medicine (2017, methodology): “To facilitate such analyses, LSTMVis [357] allows interactive exploration of the hidden state of LSTMs on different inputs.”
Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs (2018, background): “, 2015) (Strobelt et al., 2016) has focused on analysing the movement of raw gate activations over a sequence.”
Do Convolutional Neural Networks Learn Class Hierarchy? (2017, background): “LSTMVis [63] features a powerful viewer for text data.”
Predicting process behaviour using deep learning (2016, methodology): “…assists in identifying cells that are active for some input but not others (the ”hypothesis selection process”, Strobelt et al., 2016), and then conﬁrming such knowledge encoding hypotheses by comparing activation patterns against similar input (the ”matching process”, Strobelt et al., 2016)).”
A Transdisciplinary Review of Deep Learning Research and Its Relevance for Water Resources Scientists (2017, methodology): “Strobelt et al. (2016) developed amore generic visual analysis tool for LSTM languagemodels.”
Translating Neuralese (2017, methodology): “Natural language already provides a rich set of tools for describing beliefs, observations, and plans—our thesis is that these tools provide a useful complement to the visualization and ablation techniques used in previous work on understanding complex models (Strobelt et al., 2016; Ribeiro et al., ”
Learning Finite State Representations of Recurrent Policy Networks (2018, background): “There have been efforts made in the past to understand the internals of Recurrent Networks (Karpathy et al., 2015; Arras et al., 2017; Strobelt et al., 2016; Murdoch & Szlam, 2017; Jacobsson, 2005; Omlin & Giles, 1996).”
The What-If Tool: Interactive Probing of Machine Learning Models (2019, unknown): “This line of work has been especially common in relation to deep neural networks, whose workings remain somewhat mysterious, e.g. Torralba’s DrawNet [32] or Strobelt’s LSTMVis [29].”
Deep Learning for Entity Matching: A Design Space Exploration (2018, background): “To gain insights into why DL outperformsMagellan on textual and dirty data, we focus on Hybrid and use first derivative saliency [38, 47, 48, 73] to analyze what it learns.”
Interactive Visualization and Manipulation of Attention-based Neural Machine Translation (2017, methodology): “Although we here treat RNN as a black-box, there exists various methods to understand RNNs, e.g. by observing intermediate values (Strobelt et al., 2016; Karpathy et al., 2015; Li et al., 2015) or by removing some parts of them (Goh, 2016; Li et al., 2016).”
Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamples (2017, background): “Indeed, several lines of work attempt to glimpse into the RNN black-box [25,24,32,33,20,23,22,30,4,7,21,19,28,1,18].”
Explanation and Justification in Machine Learning : A Survey Or (2017, background): “…and Ma, 2005], most notably of Convolutional Neural Nets (CNNs) in image classiﬁcation [Simonyan et al. , 2013; Zeiler and Fer-gus, 2013] and of Recurrent Neural Nets (RNNs) in Natural Language Processing (NLP) applications [Karpathy et al. , 2015; Li et al. , 2016; Strobelt et al. , 2016 ] .”
Length Adaptive Recurrent Model for Text Classification (2017, background): “[12] visualizes the gate activation in each LSTM cell to explore the long dependencies behavior in LSTM. [21] tries to track the dynamic states in LSTM to discover the special pattern in datasets containing nesting, phrase structure, and chord progressions.”
Visualizing Dataflow Graphs of Deep Learning Models in TensorFlow (2018, background): “Some projects present visualizations for specific types of neural networks such as convolutional network [39] and recurrent networks [2, 48].”