Understanding Neural Networks through Representation Erasure

By Jiwei Li, Will Monroe, Dan Jurafsky. 2016

626 citations recorded by the source corpus.

While neural networks have been successfully applied to many natural language processing tasks, they come at the cost of interpretability. In this paper, we propose a general methodology to analyze and interpret decisions from a neural model by observing the effects on the model of erasing various parts of the representation, such as input word-vector dimensions, intermediate hidden units, or input words. We present several approaches to analyzing the effects of such erasure, from computing the relative difference in evaluation metrics, to using reinforcement learning to erase the minimum set of input words in order to flip a neural model's decision. In a comprehensive analysis of multiple NLP tasks, including linguistic feature classification, sentence-level sentiment analysis, and document level sentiment aspect prediction, we show that the proposed methodology not only offers clear explanations about neural model decisions, but also provides a way to conduct error analysis on neural models.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Dissecting Contextual Word Embeddings: Architecture and Representation (2018, methodology): “Li et al. (2016) examined the impact of erasing portions of a network’s representations on the output, Sundararajan et al. (2017) used a gradient based method to attribute predictions to inputs, and Murdoch et al. (2018) decomposed LSTMs to interpret classiﬁcation predictions.”
How Can We Know What Language Models Know? (2019, background): “…by using extrinsic probing tasks to examine whether certain linguistic properties can be predicted from those representations (Shi et al., 2016; Linzen et al., 2016; Belinkov et al., 2017), or by ablations to the models to investigate how behavior varies (Li et al., 2016b; Smith et al., 2017).”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “For example, Li et al. (2016b) erased speciﬁc dimensions in word embeddings or hidden states and computed the change in probability assigned to different labels.”
Universal Adversarial Triggers for Attacking and Analyzing NLP (2019, background): “Adversarial attacks also facilitate interpretation, e.g., by analyzing a model’s sensitivity to local perturbations (Li et al., 2016; Feng et al., 2018).”
Learning to Generate Reviews and Discovering Sentiment (2017, background): “Models such as word2vec have also been observed to have small subsets of dimensions strongly associated with specific tasks (Li et al., 2016).”
Semantically Equivalent Adversarial Rules for Debugging NLP models (2018, background): “Producing such adversarial examples systematically can significantly aid in debugging ML models, as it allows users to detect problems that happen in the real world, instead of oversensitivity only to malicious attacks such as intentionally scrambling, misspelling, or removing words (Bansal et al., ”
Attention is not Explanation (2019, methodology): “Li et al. (2016) summarized this commonly held view in NLP: “Attention provides an important way to explain the workings of neural models".”
Understanding Black-box Predictions via Influence Functions (2017, methodology): “Copyright 2017 by the author(s). point (Ribeiro et al., 2016) or by perturbing the test point to see how the prediction changes (Simonyan et al., 2013; Li et al., 2016b; Datta et al., 2016; Adler et al., 2016).”
Efficient Formal Safety Analysis of Neural Networks (2018, methodology): “Besides formal analysis of safety properties, we believe that our method for efficiently estimating tight and rigorous output ranges of a network for a given input range will also be useful for guiding the training process of robust networks [32, 26] and improving explainability of the decisions mad”
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges (2018, methodology): “Because of this property we can categorize the following methods to the attribution category: occlusion (Gu¸cl¨ut¨urk et al., 2017), erasure (Li et al., 2016), perturbation (Fong and Vedaldi, 2017), adversarial examples (Papernot et al., 2017) and prediction diﬀerence analysis (Zintgraf et al.,…”
Adversarial Examples: Attacks and Defenses for Deep Learning (2017, background): “[82] aimed to fool a deep learning-based sentiment classifier by removing the minimum subset of words in the given text.”
Explaining Recurrent Neural Network Predictions in Sentiment Analysis (2017, methodology): “, 2016), or to general deep neural networks for text (Denil et al., 2014; Li et al., 2016a; Arras et al., 2016a; Li et al., 2016b; Murdoch and Szlam, 2017).”
Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations (2017, methodology): “This approach to explanations can be domain- and model-specific (e.g. “annotator rationales” used to explain text classifications in [Li et al., 2016; Lei et al., 2016; Zhang et al., 2016]).”
Representation of Linguistic Form and Function in Recurrent Neural Networks (2016, methodology): “The strength of association between A mj and C can be measured by their mutual infor-Figure Similarly to Li et al. (2016), the activation value distributions are discretized into percentile bins per dimension, such that each bin contains 5% of the marginal density.”
Is Attention Interpretable? (2019, methodology): “The idea of a minimal set of inputs necessary to uphold a decision is not new; Li et al. (2016) use reinforcement learning to attempt to construct such a minimal set of words, Lei et al. (2016) train an encoder to constrain the input prior to clas-siﬁcation, and much of the work that has been done…”
Pathologies of Neural Models Make Interpretations Difficult (2018, methodology): “The importance of each feature is measured by the change in model confidence when that feature is perturbed—in this case, removed (Ribeiro et al., 2016; Li et al., 2016b).”