Attention is not not Explanation

By Sarah Wiegreffe, Yuval Pinter. 2019

1,207 citations recorded by the source corpus.

Attention mechanisms play a central role in NLP systems, especially within recurrent neural network (RNN) models. Recently, there has been increasing interest in whether or not the intermediate representations offered by these modules may be used to explain the reasoning for a model’s prediction, and consequently reach insights regarding the model’s decision-making process. A recent paper claims that ‘Attention is not Explanation’ (Jain and Wallace, 2019). We challenge many of the assumptions underlying this work, arguing that such a claim depends on one’s definition of explanation, and that testing it needs to take into account all elements of the model. We propose four alternative tests to determine when/whether attention can be used as explanation: a simple uniform-weights baseline; a variance calibration based on multiple random seed runs; a diagnostic framework using frozen weights from pretrained models; and an end-to-end adversarial attention training protocol. Each allows for meaningful interpretation of attention mechanisms in RNN models. We show that even when reliable adversarial distributions can be found, they don’t perform well on the simple diagnostic, indicating that prior work does not disprove the usefulness of attention mechanisms for explainability.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Are NLP Models really able to Solve Simple Math Word Problems? (2021, positive): “As shown by Wiegreffe and Pinter (2019), analyzing the attention weights of our constrained model is a reliable way to explain its prediction since each hidden representation consists of information about only that token as opposed to the case of an RNN where each hidden representation may have…”
Quantifying Attention Flow in Transformers (2020, background): “Although it is wrong to equate attention with explanation (Pruthi et al., 2019; Jain and Wallace, 2019), it can offer plausible and meaningful interpretations (Wiegreffe and Pinter, 2019; Vashishth et al., 2019; Vig, 2019).”
SELFEXPLAIN: A Self-Explaining Architecture for Neural Text Classifiers (2021, methodology): “In natural language applications, feature attribution based on attention scores (Xu et al., 2015) has been the predominant method for developing inherently interpretable neural classiﬁers.”
Measuring Association Between Labels and Free-Text Rationales (2020, methodology): “Analysis of NLP Models Structural tests for analyzing models’ internals include probing (Tenney et al., 2019) and attention analysis (Jain and Wal-lace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Tutek and Snajder, 2020).”
What Do NLP Researchers Believe? Results of the NLP Community Metasurvey (2022, background): “, 2021), and explainability and interpretability of black-box models (Feng et al., 2018; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021, background): “A popular way of explaining NLP models is to attribute importance weights to the input tokens, either using attention scores (Wiegreffe and Pinter, 2019) or by summarizing the model behavior on perturbed instances (e.”
MERLOT: Multimodal Neural Script Knowledge Models (2021, background): “While we do not claim that these attention weights provide a full explanation of the model behavior [51, 98], they do play some role in the model’s decision [116], and we ﬁnd that our masking strategy improves performance on downstream tasks by around 1% (Table 4), versus a SpanBERT baseline [54].”
Transformer Feed-Forward Layers Are Key-Value Memories (2020, background): “Also related are interpretability methods that explain predictions (Han et al., 2020; Wiegreffe and Pinter, 2019), however, our focus is entirely different: we do not interpret individual predictions, but aim to understand the mechanism of transformers.”
Eliciting Knowledge from Language Models Using Automatically Generated Prompts (2020, background): “Attention visualization, another common technique, has a similar failure mode: attention scores may be correlated with, but not caused by the underlying target knowledge, leading to criticism against their use as explanations (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
Explainable Recommendation: A Survey and New Perspectives (2018, neutral): “attention modules do not provide meaningful explanations and should not be treated as though they do, while Wiegreffe and Pinter (2019) later challenged many of the assumptions underlying the prior work, arguing that such a claim depends on one’s definition of explanation, and showed that the prior ”
Attributing Fair Decisions with Attention Interventions (2021, background): “There is a body of work in NLP literature that tried to analyze the effect of the attention weights on interpretability of the model (Wiegreffe and Pinter, 2019; Jain and Wallace, 2019; Serrano and Smith, 2019).”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “Interpreting attention on language sequences is a well-established area of research [12, 86, 89].”
ERASER: A Benchmark to Evaluate Rationalized NLP Models (2019, background): “However, attention weights do not in general provide faithful explanations for predictions (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Zhong et al., 2019; Pruthi et al., 2019; Brunner et al., 2019; Moradi et al., 2019; Vashishth et al., 2019).”
exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models (2019, background): “However, interpreting attention patterns without understanding the attended-to embeddings, or relying on attention alone for a faithful interpretation, can lead to faulty interpretations [Brunner et al., 2019, Jain and Wallace, 2019, Wiegreffe and Pinter, 2019].”
Learning to Deceive with Attention-Based Explanations (2019, background): “attention are all restricted to classification tasks (Jain et al., 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019).”
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021, background): “Moreover, they provide attention-based explanations whose interpretability is unclear (Serrano and Smith, 2019; Brunner et al., 2020; Wiegreffe and Pinter, 2019).”
On Identifiability in Transformers (2019, background): “However, their identiﬁability affects the interpretability of the output, i.e., whether attention weights can provide the basis for explanatory insights on the model’s predictions (cf. Jain & Wallace (2019) and Wiegreffe & Pinter (2019)).”