Pathologies of Neural Models Make Interpretations Difficult

By Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, Jordan L. Boyd-Graber. 2018

343 citations recorded by the source corpus.

One way to interpret neural model predictions is to highlight the most important input features—for example, a heatmap visualization over the words in an input sentence. In existing interpretation methods for NLP, a word’s importance is determined by either input perturbation—measuring the decrease in model confidence when that word is removed—or by the gradient with respect to that word. To understand the limitations of these methods, we use input reduction, which iteratively removes the least important word from the input. This exposes pathological behaviors of neural models: the remaining words appear nonsensical to humans and are not the ones determined as important by interpretation methods. As we confirm with human experiments, the reduced examples lack information to support the prediction of any label, but models still make the same predictions with high confidence. To explain these counterintuitive results, we draw connections to adversarial examples and confidence calibration: pathological behaviors reveal difficulties in interpreting neural models trained with maximum likelihood. To mitigate their deficiencies, we fine-tune the models by encouraging high entropy outputs on reduced examples. Fine-tuned models become more interpretable under input reduction, without accuracy loss on regular examples.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Pretrained Transformers Improve Out-of-Distribution Robustness (2020, background): “, 2018), and reducing inputs (Feng et al., 2018).”
Semantic Noise Matters for Neural Natural Language Generation (2019, background): “…reports on hallucinations (i.e. adding information not grounded in the input), as observed for image captioning (Rohrbach et al., 2018), sports report generation (Wiseman et al., 2017), machine translation (Koehn and Knowles, 2017; Lee et al., 2019), and question answering (Feng et al., 2018).”
What Do NLP Researchers Believe? Results of the NLP Community Metasurvey (2022, background): “, 2021), and explainability and interpretability of black-box models (Feng et al., 2018; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “A number of studies evaluated the effect of erasing or masking certain neural network components, such as word embedding dimensions, hidden units, or even full words (Li et al., 2016b; Feng et al., 2018; Khandelwal et al., 2018; Bau et al., 2018).”
Attention is not Explanation (2019, background): “While such measures do enjoy a clear intrinsic (to the model) semantics, their interpretation in the context of nonlinear neural networks can nonetheless be difficult for humans (Feng et al., 2018).”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, methodology): “This is strikingly evident when models achieve high test accuracy but fail on simple input perturbations (Jia and Liang, 2017; Feng et al., 2018; Ribeiro et al., 2018a), challenge examples (Naik et al.”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, neutral): “(2019), and question answering Feng et al. (2018).”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, methodology): “Gradient-based techniques are the most commonly used methods in this category [25, 73, 74].”
Is Attention Interpretable? (2019, methodology): “To examine the importance of ℐ′ , we run the classification layer of the model twice (Figure 1): once without any modification, and once after renormalizing the attention distribution with ℐ′ ’s attention weights zeroed out, similar to other erasure-based work (Li et al., 2016; Feng et al., 2018).”
ERASER: A Benchmark to Evaluate Rationalized NLP Models (2019, background): “These enjoy a clear semantics (describing how perturbing inputs locally affects outputs), but may nonetheless exhibit counterintuitive behaviors (Feng et al., 2018).”
Neural Module Networks for Reasoning over Text (2019, background): “Recent models have performed well on certain QA datasets, sometimes rivaling humans (Zhang et al., 2019), but it has become increasingly clear that they primarily exploit surface level lexical cues (Jia & Liang, 2017; Feng et al., 2018) and compositional QA still remains a challenge.”
Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets (2019, background): “Skill s1 is inspired by Feng et al. (2018) and Sugawara et al.”
Errudite: Scalable, Reproducible, and Testable Error Analysis (2019, background): “, by adding distracting sentences to the context in MC (Jia and Liang, 2017), or feeding partial questions or wrong images into models (Agrawal et al., 2016; Mudrakarta et al., 2018; Feng et al., 2018).”
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020, methodology): “…and evaluation for faith- 9Whether for attention (Baan et al., 2019; Pruthi et al., 2019; Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019), saliency methods (Alvarez-Melis and Jaakkola, 2018; Kindermans et al., 2019), or others (Ghorbani et al., 2019; Feng et al., 2018).”
Evaluating NLP Models via Contrast Sets (2020, methodology): “This is strikingly evident when models achieve high test accuracy but fail on simple input perturbations (Jia and Liang, 2017; Feng et al., 2018), challenge examples (Naik et al., 2018), and covariate and label shifts (Ben-David et al., 2010; Shimodaira, 2000; Lipton et al., 2018). ar X iv :2 00 4.”
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions (2020, background): “We also note that Feng et al. (2018) ﬁnd that gradient-based methods sometimes yield counter-intuitive results when iterative input reductions are performed.”
Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations (2019, methodology): “The methods that provide feature importance explanations [27, 23, 7, 20, 11] aim to provide the user with the subset of input tokens that contributed the most to the prediction of the model.”
NILE : Natural Language Inference with Faithful Natural Language Explanations (2020, background): “Difﬁculty in building interpretation models and the lack of robustness of the same are some of the major issues in existing deep neural networks systems (Feng et al., 2018; Ghorbani et al., 2019; Oana-Maria et al., 2019b).”
How Do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking (2020, methodology): “They are often motivated as approximations of erasure (Baehrens et al., 2010; Simonyan et al., 2013; Feng et al., 2018) and some-times even evaluated using erasure (e.g., Serrano and Smith (2019); Jain and Wallace (2019)).”