Visualizing and Understanding Neural Models in NLP

By Jiwei Li, Xinlei Chen, Eduard H. Hovy, Dan Jurafsky. 2015

763 citations recorded by the source corpus.

While neural networks have been successfully applied to many NLP tasks the resulting vector-based models are very difficult to interpret. For example it's not clear how they achieve {\em compositionality}, building sentence meaning from the meanings of words and phrases. In this paper we describe four strategies for visualizing compositionality in neural models for NLP, inspired by similar work in computer vision. We first plot unit values to visualize compositionality of negation, intensification, and concessive clauses, allow us to see well-known markedness asymmetries in negation. We then introduce three simple and straightforward methods for visualizing a unit's {\em salience}, the amount it contributes to the final composed meaning: (1) gradient back-propagation, (2) the variance of a token from the average word node, (3) LSTM-style gates that measure information flow. We test our methods on sentiment using simple recurrent nets and LSTMs. Our general-purpose methods may have wide applications for understanding compositionality and other semantic properties of deep networks , and also shed light on why LSTMs outperform simple recurrent nets,

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

SELFEXPLAIN: A Self-Explaining Architecture for Neural Text Classifiers (2021, methodology): “, 2019) and gradient (Li et al., 2016) based explanation methods.”
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “Behavioral analyses, on the other hand, aim to assess a model’s behavior by its performance on constructed examples (e.g., Isabelle et al., 2017; Naik et al., 2018), or by visualizing important input features via saliency methods (e.g., Li et al., 2016; Murdoch et al., 2018).”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “The important or salient features can then be visualized in selected examples (Li et al., 2016a; Aubakirova and Bansal, 2016; Sundararajan et al., 2017; Arras et al., 2017a,b; Ding et al., 2017; Murdoch et al., 2018; Mudrakarta et al., 2018; Montavon et al., 2018; Godin et al., 2018).”
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (2018, methodology): “…of recurrent networks and related models by using visualization, input/hidden representation deletion techniques or by looking at the word-by-word behaviour of the network (e.g., Nagamine et al., 2015; Hupkes et al., 2017; Li et al., 2016; Linzen et al., 2016; Kàdàr et al., 2017; Li et al., 2017).”
Probing for semantic evidence of composition by means of simple classification tasks (2016, methodology): “In an alternative approach, Li et al. (2015) use visualization techniques to better examine the particular aspects of compositionality captured by their models.”
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies (2016, neutral): “K´ad´ar et al. (2016) and Li et al. (2016) suggest visualization techniques for word-level RNNs trained to perform tasks that aren’t explicitly syntactic (image captioning and sentiment analysis).”
Rationalizing Neural Predictions (2016, background): “Efforts in this area include analyzing and visualizing state activation (Hermans and Schrauwen, 2013; Karpathy et al., 2015; Li et al., 2016), learning sparse interpretable word vectors (Faruqui et al., 2015b), and linking word vectors to semantic lexicons or word properties (Faruqui et al., 2015a;…”
Does String-Based Neural MT Learn Source Syntax? (2016, neutral): “Li et al. (2016) explore the syntactic behavior of an RNN-based sentiment analyzer, including the compositionality of negation, intensiﬁcation, and concessive clauses, by plotting a 60-dimensional heat map of hidden unit values.”
Linguistic Knowledge and Transferability of Contextual Representations (2019, background): “…to study the learned representations in neural models, such as directly examining the activations of individual neurons (Karpathy et al., 2015; Li et al., 2015; Shi et al., 2016a, inter alia ), ablating components of the model and dataset (Kuncoro et al., 2017; Gaddy et al., 2018; Khandelwal…”
Understanding Neural Networks through Representation Erasure (2016, methodology): “Li et al. (2015) use the first-order derivative to examine the saliency of input features, but they rely on the overly strong assumption that the decision score is a linear combination of input features.”
Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Task (2017, background): “Additionally, there is an increasing amount of work on developing methods of exposing exactly what linguistic knowledge NLP models develop (K´ad´ar et al., 2016; Li et al., 2015) and what linguistic information is encoded in models’ produced representations (Adi et al., 2016; Ettinger et al., 2016).”
Explaining Recurrent Neural Network Predictions in Sentiment Analysis (2017, methodology): “A thorough model description can be found in the Appendix, and for details on the training we refer to (Li et al., 2016a).”
Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations (2017, methodology): “, 2010] provides a particularly good introduction; see also [Selvaraju et al., 2016; Simonyan et al., 2013; Li et al., 2015; Hechtlinger, 2016]).”
Unmasking Clever Hans predictors and assessing what machines really learn (2019, methodology): “Explanation methods were successfully applied to a wide set of complex real-world problems such as the analysis of faces [81, 123, 124], EEG and fMRI data [23, 50], human gait [48], videos [125], speech [126], reinforcement learning [25], biological data [127], text [26,128,129], or comparing human ”
Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context (2018, background): “They can remember sentence lengths, word identity, and word order (Adi et al., 2017), can capture some syntactic structures such as subject-verb agreement (Linzen et al., 2016), and can model certain kinds of semantic compositionality such as negation and intensiﬁca-tion (Li et al., 2016).”