Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs

By W. James Murdoch, Peter J. Liu, Bin Yu. 2018

223 citations recorded by the source corpus.

The driving force behind the recent success of LSTMs has been their ability to learn complex and non-linear relationships. Consequently, our inability to describe these relationships has led to LSTMs being characterized as black boxes. To this end, we introduce contextual decomposition (CD), an interpretation algorithm for analysing individual predictions made by standard LSTMs, without any changes to the underlying model. By decomposing the output of a LSTM, CD captures the contributions of combinations of words or variables to the final prediction of an LSTM. On the task of sentiment analysis with the Yelp and SST data sets, we show that CD is able to reliably identify words and phrases of contrasting sentiment, and how they are combined to yield the LSTM's final prediction. Using the phrase-level labels in SST, we also demonstrate that CD is able to successfully extract positive and negative negations from an LSTM, something which has not previously been done.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Dissecting Contextual Word Embeddings: Architecture and Representation (2018, methodology): “Li et al. (2016) examined the impact of erasing portions of a network’s representations on the output, Sundararajan et al. (2017) used a gradient based method to attribute predictions to inputs, and Murdoch et al. (2018) decomposed LSTMs to interpret classiﬁcation predictions.”
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “Behavioral analyses, on the other hand, aim to assess a model’s behavior by its performance on constructed examples (e.g., Isabelle et al., 2017; Naik et al., 2018), or by visualizing important input features via saliency methods (e.g., Li et al., 2016; Murdoch et al., 2018).”
Interpretations are useful: penalizing explanations to align neural networks with prior knowledge (2019, methodology): “We then describe Contextual Decomposition (CD), the explanation method proposed by (Murdoch et al., 2018).”
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges (2018, methodology): “Other methods that belong to this category are found in (Baehrens et al., 2010; Murdoch et al., 2018; Ribeiro et al., 2016b).”
Interpretable and Steerable Sequence Learning via Prototypes (2019, background): “To address this challenge, a variety of methods have been developed to unveil the inner-workings of deep sequence models through visualizing the changes in hidden states [17, 31], extracting feature importance [1, 22, 23] and constructing rules that mimic the behavior of RNNs [5].”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “The important or salient features can then be visualized in selected examples [12, 203, 247, 248, 317].”
Explaining and Interpreting LSTMs (2019, methodology): “For the CD explanation method [56] we employ the code from the authors to generate one relevance value per time step (this necessitates to run the CD decomposition as many times as there are time steps in the input sequence).”
Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases (2021, methodology): “the model, if its decisions are based on concise rationales it selects, as opposed to inferring explanations from the model’s decisions in a post-hoc manner (Ribeiro et al., 2016; Alvarez-Melis and Jaakkola, 2017; Murdoch et al., 2018).”
Hierarchical interpretations for neural network predictions (2018, methodology): “We now present empirical validation of ACD on both LSTMs trained on SST and CNNs trained on MNIST and ImageNet.”
Explaining Character-Aware Neural Networks for Word-Level Prediction: Do They Discover Linguistic Rules? (2018, unknown): “For details on CD for LSTMs, we refer the reader to the afore-mentioned paper.”
Investigating Gender Bias in Language Models Using Causal Mediation Analysis (2020, background): “Another approach identiﬁes important input features that contribute to a model’s prediction via saliency methods (Li et al., 2016; Arras et al., 2017; Murdoch et al., 2018), which also typically ignore the model’s internal structure, although one could compute them with respect to internal…”
TOWARDS HIERARCHICAL IMPORTANCE ATTRIBU- (2019, methodology): “Contextual decomposition (CD) (Murdoch et al., 2018) and its hierarchical extension (Singh et al., 2019) go beyond the additive assumption and compute the contribution solely made by a word/phrase to the model prediction (i.e., individual contribution), by decomposing the output variables of the…”
Contextualizing Hate Speech Classifiers with Post-hoc Explanation (2020, background): “These explanations reveal either word-level (Ribeiro et al., 2016; Sundararajan et al., 2017) or phrase-level importance (Murdoch et al., 2018; Singh et al., 2019) of inputs to predictions.”
Edinburgh Research Explorer Understanding Learning Dynamics Of Language Models with SVCCA (unknown): “One avenue for interpreting structure is based on (Murdoch et al., 2018), which decomposes the cell context in a LSTM in order to study the interactions between words in a sequence.”
Improving the Interpretability of Neural Sentiment Classifiers via Data Augmentation (2019, background): “…explainable models has been widely recognized on general text classification (Lei et al., 2016; Liu et al., 2018; Nguyen, 2018), other NLP tasks (Zhao et al., 2018; Feng and Boyd-Graber, 2018; Thorne et al., 2019), and machine learning (Bien et al., 2011; Koh and Liang, 2017; Murdoch et al., 2018).”