Attention is not Explanation

By Sarthak Jain, Byron C. Wallace. 2019

1,871 citations recorded by the source corpus.

Attention mechanisms have seen wide adoption in neural NLP models. In addition to improving predictive performance, these are often touted as affording transparency: models equipped with attention provide a distribution over attended-to input units, and this is often presented (at least implicitly) as communicating the relative importance of inputs. However, it is unclear what relationship exists between attention weights and model outputs. In this work we perform extensive experiments across a variety of NLP tasks that aim to assess the degree to which attention weights provide meaningful “explanations” for predictions. We find that they largely do not. For example, learned attention weights are frequently uncorrelated with gradient-based measures of feature importance, and one can identify very different attention distributions that nonetheless yield equivalent predictions. Our findings show that standard attention modules do not provide meaningful explanations and should not be treated as though they do.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Improving language models by retrieving from trillions of tokens (2021, background): “Finally, samples from large models are difficult to interpret, making mitigating these issues all the more challenging (Belinkov et al., 2020; Jain and Wallace, 2019).”
Quantifying Attention Flow in Transformers (2020, background): “Although it is wrong to equate attention with explanation (Pruthi et al., 2019; Jain and Wallace, 2019), it can offer plausible and meaningful interpretations (Wiegreffe and Pinter, 2019; Vashishth et al., 2019; Vig, 2019).”
SELFEXPLAIN: A Self-Explaining Architecture for Neural Text Classifiers (2021, background): “However, such interpretations were shown to be unreliable (Ser-rano and Smith, 2019; Pruthi et al., 2020) and un-faithful (Jain and Wallace, 2019; Wiegreffe and Pin-ter, 2019).”
Measuring Association Between Labels and Free-Text Rationales (2020, methodology): “We propose to measure labelrationale association within self-rationalizing models that produce free-text rationales, much like how prior work has quantified the faithfulness of attention (Jain and Wallace, 2019; Brunner et al., 2020).”
WT5?! Training Text-to-Text Models to Explain their Predictions (2020, methodology): “…“for free” by examining the weight assigned by the neural network to different regions in the input (Graves, 2013; Raffel et al., 2017; Huang et al., 2018), but this introspection method has been shown to be unreliable (Jain & Wallace, 2019; Serrano & Smith, 2019; Pruthi et al., 2019).”
What Do NLP Researchers Believe? Results of the NLP Community Metasurvey (2022, background): “, 2021), and explainability and interpretability of black-box models (Feng et al., 2018; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
MERLOT: Multimodal Neural Script Knowledge Models (2021, background): “While we do not claim that these attention weights provide a full explanation of the model behavior [51, 98], they do play some role in the model’s decision [116], and we ﬁnd that our masking strategy improves performance on downstream tasks by around 1% (Table 4), versus a SpanBERT baseline [54].”
Of Non-Linearity and Commutativity in BERT (2021, background): “However, recent work raises concerns about the interpretability of attention distributions [3], [12], [21], [24].”
What Does BERT Look at? An Analysis of BERT’s Attention (2019, background): “Jain and Wallace (2019) argue that attention often does not “explain” model predictions.”
A Primer in BERTology: What We Know About How BERT Works (2020, methodology): “This view is currently debated (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Brunner et al., 2020), and in a multi-layer model where attention is followed by non-linear transformations, the patterns in individual heads do not provide a full picture.”
Attention is not not Explanation (2019, methodology): “…greatly exacerbated when performed independently on each instance.2 Thus, it is no surprise that Jain and Wal- 2Indeed, the most open-ended task, question answering over CNN data, produces considerable difficulty to manipulate its scores by random permutation (Figure 6e in Jain and Wallace (2019)).”
Eliciting Knowledge from Language Models Using Automatically Generated Prompts (2020, background): “Attention visualization, another common technique, has a similar failure mode: attention scores may be correlated with, but not caused by the underlying target knowledge, leading to criticism against their use as explanations (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
Interpretations are useful: penalizing explanations to align neural networks with prior knowledge (2019, background): “Some recent work proposes forcing models to attend to regions of the input which are known to be important (Burns et al., 2018; Mitsuhara et al., 2019), although it is important to note that attention is often not the same as explanation (Jain & Wallace, 2019).”
Explainable Recommendation: A Survey and New Perspectives (2018, methodology): “However, Jain and Wallace (2019) argued that standard”
Attributing Fair Decisions with Attention Interventions (2021, background): “There is a body of work in NLP literature that tried to analyze the effect of the attention weights on interpretability of the model (Wiegreffe and Pinter, 2019; Jain and Wallace, 2019; Serrano and Smith, 2019).”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “It does not attempt to establish a causal link between attention and model behavior [28, 84], nor to explain model predictions [35, 87].”
A Multiscale Visualization of Attention in the Transformer Model (2019, background): “, 2015; Belinkov and Glass, 2019), although its value in explaining individual predictions may be limited (Jain and Wallace, 2019).”
Is Attention Interpretable? (2019, background): “So far, other work looking at attention has examined whether attention sufﬁces as a holistic explanation for a model’s decision (Jain and Wallace, 2019), which is a higher bar.”