Visualizing and Understanding Recurrent Networks

By Andrej Karpathy, Justin Johnson, Li Fei-Fei. 2015

1,142 citations recorded by the source corpus.

Recurrent Neural Networks (RNNs), and specifically a variant with Long Short-Term Memory (LSTM), are enjoying renewed interest as a result of successful applications in a wide range of machine learning problems that involve sequential data. However, while LSTMs provide exceptional results in practice, the source of their performance and their limitations remain rather poorly understood. Using character-level language models as an interpretable testbed, we aim to bridge this gap by providing an analysis of their representations, predictions and error types. In particular, our experiments reveal the existence of interpretable cells that keep track of long-range dependencies such as line lengths, quotes and brackets. Moreover, our comparative analysis with finite horizon n-gram models traces the source of the LSTM improvements to long-range structural dependencies. Finally, we provide analysis of the remaining errors and suggests areas for further study.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Dissecting Contextual Word Embeddings: Architecture and Representation (2018, background): “Our intrinsic analysis is most similar to Belinkov et al. (2017); however, we probe span representations in addition to word representations, evaluate the transferability of the biLM representations to semantic tasks in addition to syntax tasks, and consider a wider variety of neural architectures i”
Semi-supervised sequence tagging with bidirectional language models (2017, background): “Karpathy et al. (2015) visualized character level LSTM states and showed that individual cells capture long-range dependencies such as line lengths, quotes and brackets.”
A Survey of Machine Learning for Big Code and Naturalness (2017, background): “[103] PD (c) Characters RNN (LSTM) — Kushman and Barzilay [110] PD (c |m) Token Grammar (CCG) Code Synthesis Maddison and Tarlow [126] PD (c) Syntax with scope NN —”
Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks (2015, background): “While stacking more convolution layers tends to make ConvNets perform better, the same is not always true for RNNs [19].”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “Much recent work has focused on visualizing activations on specific examples in modern neural networks for language (Karpathy et al., 2015; Kádár et al., 2017; Qian et al., 2016a; Liu et al., 2018) and speech (Wu and King, 2016; Nagamine et al.”
Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure (2017, positive): “Gate activation statistics In the same paper, Karpathy et al. (2015) study gate activations, focusing in particular on the fraction of time that gates spend being left- or right-saturated (activations less 0.”
What do Neural Machine Translation Models Learn about Morphology? (2017, background): “work visualizes hidden unit activations in recurrent neural networks that are trained for a given task (Elman, 1991; Karpathy et al., 2015; Kádár et al., 2016; Qian et al., 2016a).”
Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks (2016, methodology): “This is the approach taken by Li et al. (2015), Hill et al. (2016) and Kiros et al. (2015).”
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies (2016, neutral): “Previous research has shown that RNNs (in particular LSTMs) can learn artiﬁcial context-free languages (Gers and Schmidhuber, 2001) indentation in a programming language (Karpathy et al., 2016).”
Rationalizing Neural Predictions (2016, background): “Efforts in this area include analyzing and visualizing state activation (Hermans and Schrauwen, 2013; Karpathy et al., 2015; Li et al., 2016), learning sparse interpretable word vectors (Faruqui et al., 2015b), and linking word vectors to semantic lexicons or word properties (Faruqui et al., 2015a;…”
Learning to Generate Reviews and Discovering Sentiment (2017, background): “Previous work analysing LSTM language models showed the existence of interpretable units that indicate position within a line or presence inside a quotation (Karpathy et al., 2015).”
DenseCap: Fully Convolutional Localization Networks for Dense Captioning (2015, background): “These models have previously been used in language modeling [2, 15, 33, 44], where they are known to learn powerful long-term interactions [22].”
Analyzing the Behavior of Visual Question Answering Models (2016, background): “(Karpathy et al., 2016) constructed a series of oracles to measure the performance of a character level”
Linguistic Knowledge and Transferability of Contextual Representations (2019, neutral): “…a variety of other methods to study the learned representations in neural models, such as directly examining the activations of individual neurons (Karpathy et al., 2015; Li et al., 2015; Shi et al., 2016a, inter alia ), ablating components of the model and dataset (Kuncoro et al., 2017; Gaddy et…”
Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks (2017, methodology): “Techniques for analyzing neural network models include visualization of hidden units (Elman, 1991; Karpathy et al., 2015; K´ad´ar et al., 2016; Qian et al., 2016a), which provide illuminating, but often anecdotal information on how the network works.”
Visualizing and Understanding Neural Models in NLP (2015, background): “In parallel to our research, independent researches (Karpathy et al., 2015) have been conducted to explore similar direction from an error-analysis point of view, by analyzing predictions and errors from a recurrent neural models.”
Understanding Neural Networks through Representation Erasure (2016, methodology): “Efforts to understand neural vector space models in natural language processing (NLP) include visualizing state activation (Hermans and Schrauwen, 2013; Karpathy et al., 2015), interpreting semantic dimensions by presenting human users with a list of words and ask them to choose the one that does no”
Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, positive): “The authors in [281] propose a visualization technique based on ﬁnite horizon n-grams that discriminates interpretable cells within LSTM and GRU networks.”