Attention Interpretability Across NLP Tasks

By Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, Manaal Faruqui. 2019

191 citations recorded by the source corpus.

The attention layer in a neural network model provides insights into the model's reasoning behind its prediction, which are usually criticized for being opaque. Recently, seemingly contradictory viewpoints have emerged about the interpretability of attention weights (Jain & Wallace, 2019; Vig & Belinkov, 2019). Amid such confusion arises the need to understand attention mechanism more systematically. In this work, we attempt to fill this gap by giving a comprehensive explanation which justifies both kinds of observations (i.e., when is attention interpretable and when it is not). Through a series of experiments on diverse NLP tasks, we validate our observations and reinforce our claim of interpretability of attention through manual evaluation.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Quantifying Attention Flow in Transformers (2020, background): “Although it is wrong to equate attention with explanation (Pruthi et al., 2019; Jain and Wallace, 2019), it can offer plausible and meaningful interpretations (Wiegreffe and Pinter, 2019; Vashishth et al., 2019; Vig, 2019).”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “Depending on the task and model architecture, attention may have more or less explanatory power for model predictions [35, 51, 57, 71, 79].”
ERASER: A Benchmark to Evaluate Rationalized NLP Models (2019, background): “However, attention weights do not in general provide faithful explanations for predictions (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Zhong et al., 2019; Pruthi et al., 2019; Brunner et al., 2019; Moradi et al., 2019; Vashishth et al., 2019).”
How Do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking (2020, methodology): “…attribution methods which rely on heuristic rules to deﬁne feature salience, for example, attention-based attri-bution (Rockt¨aschel et al., 2015; Serrano and Smith, 2019; Vashishth et al., 2019) or back-propagation methods (Bach et al., 2015; Shrikumar et al., 2017; Sundararajan et al., 2017).”
Staying True to Your Word: (How) Can Attention Become Explanation? (2020, methodology): “Jain and Wallace (2019) and Vashishth et al. (2019) demonstrate that, if we permute or substitute the weights of a learned attention distribution, our model can still retain high (and in some cases, unchanged) classiﬁcation performance and prediction conﬁdence.”
A Diagnostic Study of Explainability Techniques for Text Classification (2020, background): “Finally, existing studies focus on one (Vashishth et al., 2019) or a single group of explainability methods (DeYoung et al.”
Post-hoc Interpretability for Neural NLP: A Survey (2021, background): “Verifying this is often non-trivial, as has repeatedly been shown with Attention [9], where multiple papers have found contradicting conclusions regarding interpretability [54, 104, 121, 129].”
Understanding Multi-Head Attention in Abstractive Summarization (2019, background): “A promising recent line of work focuses on adversarial attention attacks (Jain and Wallace, 2019; Serrano and Smith, 2019; Vashishth et al., 2019), but this is not yet applied to sequence-tosequence tasks.”
Explainable CNN-attention Networks (C-Attention Network) for Automated Detection of Alzheimer's Disease (2020, methodology): “However, this debate was settled in favor of using self-attention mechanisms as a viable method for interpretation for classification tasks [39].”
ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning (2021, methodology): “We focused in more detail on the attention mechanism [70] at the core of each Transformer model [10] providing some limited understanding of the AI [71], [72].”
Unsupervised Quality Estimation for Neural Machine Translation (2020, background): “Recent work on attention inter-pretability showed that attention weights in Trans-former networks might not be readily interpretable (Vashishth et al., 2019; Vig and Belinkov, 2019).”
Explainable Deep Learning: A Field Guide for the Uninitiated (2020, background): “To reconcile these conflicting views, some studies (Vashishth et al., 2019; Wiegreffe & Pinter, 2019) conduct experiments to find situations where attention can be used to gain insight into model prediction.”