Is Attention Interpretable?

By Sofia Serrano, Noah A. Smith. 2019

844 citations recorded by the source corpus.

Attention mechanisms have recently boosted performance on a range of NLP tasks. Because attention layers explicitly weight input components’ representations, it is also often assumed that attention can be used to identify information that models found important (e.g., specific contextualized word tokens). We test whether that assumption holds by manipulating attention weights in already-trained text classification models and analyzing the resulting differences in their predictions. While we observe some ways in which higher attention weights correlate with greater impact on model predictions, we also find many ways in which this does not hold, i.e., where gradient-based rankings of attention weights better predict their effects than their magnitudes. We conclude that while attention noisily predicts input components’ overall importance to a model, it is by no means a fail-safe indicator.1

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Measuring Association Between Labels and Free-Text Rationales (2020, methodology): “Analysis of NLP Models Structural tests for analyzing models’ internals include probing (Tenney et al., 2019) and attention analysis (Jain and Wal-lace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Tutek and Snajder, 2020).”
WT5?! Training Text-to-Text Models to Explain their Predictions (2020, methodology): “…“for free” by examining the weight assigned by the neural network to different regions in the input (Graves, 2013; Raffel et al., 2017; Huang et al., 2018), but this introspection method has been shown to be unreliable (Jain & Wallace, 2019; Serrano & Smith, 2019; Pruthi et al., 2019).”
MERLOT: Multimodal Neural Script Knowledge Models (2021, background): “While we do not claim that these attention weights provide a full explanation of the model behavior [51, 98], they do play some role in the model’s decision [116], and we ﬁnd that our masking strategy improves performance on downstream tasks by around 1% (Table 4), versus a SpanBERT baseline [54].”
A Primer in BERTology: What We Know About How BERT Works (2020, methodology): “However, there is ongoing debate on the merits of attention as a tool for interpreting deep learning models (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Brunner et al., 2020).”
Attention is not not Explanation (2019, background): “Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rockt¨aschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “Depending on the task and model architecture, attention may have more or less explanatory power for model predictions [35, 51, 57, 71, 79].”
ERASER: A Benchmark to Evaluate Rationalized NLP Models (2019, background): “…computationally cheap to evaluate, in contrast to measures that require per-token measurements, e.g., importance score correlations with ‘leave-one-out’ scores (Jain and Wallace, 2019), or counting how many ‘important’ tokens need to be erased before a prediction ﬂips (Serrano and Smith, 2019).”
Explaining Question Answering Models through Text Generation (2020, background): “Thus, it is more similar to attention as an explanation (Serrano and Smith, 2019; Jain and Wallace, 2019; Pruthi et al., 2019), where the attention structure does not necessarily reveal what tokens are used for prediction.”
Learning to Deceive with Attention-Based Explanations (2019, background): “Previous studies analysing the interpretability of attention are all restricted to classiﬁcation tasks (Jain et al., 2019; Serrano and Smith, 2019; Wiegr-effe and Pinter, 2019).”
On Identifiability in Transformers (2019, background): “Serrano & Smith (2019) ﬁnd that zero-ing out attention weights based on gradient attribution changes the output of a multi-class prediction task more quickly than zero-ing out based on attention weights, thus showing that attention is not the best predictor of learned feature importance.”
Pre-trained models for natural language processing: A survey (2020, background): “However, much work on model analysis depends on the attention mechanism, and the effectiveness of attention for interpretability is still controversial [71, 155].”
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020, methodology): “One method proposed to do so is erasure , where the “most relevant” parts of the input—according to the explanation—are erased from the input, in expectation that the model’s decision will change (Arras et al., 2016; Feng et al., 2018; Serrano and Smith, 2019).”
How Do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking (2020, methodology): “…attribution methods which rely on heuristic rules to deﬁne feature salience, for example, attention-based attri-bution (Rockt¨aschel et al., 2015; Serrano and Smith, 2019; Vashishth et al., 2019) or back-propagation methods (Bach et al., 2015; Shrikumar et al., 2017; Sundararajan et al., 2017).”
Aligning Faithful Interpretations with their Social Attribution (2020, background): “Recent work on the faithfulness of attention heat-maps (Baan et al., 2019; Pruthi et al., 2019; Serrano and Smith, 2019) or saliency distributions (AlvarezMelis and Jaakkola, 2018; Kindermans et al.”
Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking (2020, methodology): “Another popular approach is to treat attention or gate scores as a measure of importance (Serrano & Smith, 2019).”