Learning to Deceive with Attention-Based Explanations

By Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, Zachary Chase Lipton. 2019

217 citations recorded by the source corpus.

Attention mechanisms are ubiquitous components in neural architectures applied to natural language processing. In addition to yielding gains in predictive accuracy, attention weights are often claimed to confer interpretability, purportedly useful both for providing insights to practitioners and for explaining why a model makes its decisions to stakeholders. We call the latter use of attention mechanisms into question by demonstrating a simple method for training models to produce deceptive attention masks. Our method diminishes the total weight assigned to designated impermissible tokens, even when the models can be shown to nevertheless rely on these features to drive predictions. Across multiple models and tasks, our approach manipulates attention weights while paying surprisingly little cost in accuracy. Through a human study, we show that our manipulated attention-based explanations deceive people into thinking that predictions from a model biased against gender minorities do not rely on the gender. Consequently, our results cast doubt on attention’s reliability as a tool for auditing algorithms in the context of fairness and accountability.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Quantifying Attention Flow in Transformers (2020, background): “Although it is wrong to equate attention with explanation (Pruthi et al., 2019; Jain and Wallace, 2019), it can offer plausible and meaningful interpretations (Wiegreffe and Pinter, 2019; Vashishth et al., 2019; Vig, 2019).”
SELFEXPLAIN: A Self-Explaining Architecture for Neural Text Classifiers (2021, background): “However, such interpretations were shown to be unreliable (Serrano and Smith, 2019; Pruthi et al., 2020) and unfaithful (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021, background): “Though ubiquitous, token scores may not always reflect their real importance (Pruthi et al., 2020).”
Of Non-Linearity and Commutativity in BERT (2021, background): “However, recent work raises concerns about the interpretability of attention distributions [3], [12], [21], [24].”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “Depending on the task and model architecture, attention may have more or less explanatory power for model predictions [35, 51, 57, 71, 79].”
ERASER: A Benchmark to Evaluate Rationalized NLP Models (2019, background): “However, attention weights do not in general provide faithful explanations for predictions (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Zhong et al., 2019; Pruthi et al., 2019; Brunner et al., 2019; Moradi et al., 2019; Vashishth et al., 2019).”
Explaining Question Answering Models through Text Generation (2020, background): “Thus, it is more similar to attention as an explanation (Serrano and Smith, 2019; Jain and Wallace, 2019; Pruthi et al., 2019), where the attention structure does not necessarily reveal what tokens are used for prediction.”
On Identifiability in Transformers (2019, background): “Pruthi et al. (2019) demonstrate that self-attention models can be manipulated to produce different attention masks with very little cost in accuracy.”
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020, methodology): “Whether for attention (Baan et al., 2019; Pruthi et al., 2019; Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019), saliency methods (Alvarez-Melis and Jaakkola, 2018; Kindermans et al.”
Telling BERT’s Full Story: from Local Attention to Global Aggregation (2020, background): “Recent work (Brunner et al., 2020; Pruthi et al., 2019) has raised concerns about the interpretability of attention maps as representative of global context aggregation.”
Explaining and Improving Model Behavior with k Nearest Neighbor Representations (2020, methodology): “…for NLP models have focused on explaining individual predictions by using gradient-based saliency maps over the input text (Lei, Barzilay, and Jaakkola 2016; Ribeiro, Singh, and Guestrin 2018; Bastings, Aziz, and Titov 2019) or interpreting attention (Brunner et al. 2020; Pruthi et al. 2020).”
Limits to Depth Efficiencies of Self-Attention (2020, result): “…linked to the need to transform the Key/Query similarity score into a distribution, has been recently challenged, as a growing body of work shows that the attention weights distribution does not directly correlate with predictions [Jain and Wallace, 2019, Pruthi et al., 2019, Brunner et al., 2020].”
Learning to Faithfully Rationalize by Construction (2020, methodology): “For example, without faithful explanations, we cannot know whether a model is exploiting sensitive features such as gender (Pruthi et al., 2020).”
Aligning Faithful Interpretations with their Social Attribution (2020, background): “Recent work on the faithfulness of attention heat-maps (Baan et al., 2019; Pruthi et al., 2019; Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019) or saliency distributions (AlvarezMelis and Jaakkola, 2018; Kindermans et al., 2019) cast doubt on their faithfulness as…”
Which transformer architecture fits my data? A vocabulary bottleneck in self-attention (2021, background): “While removing the attention score normalization via softmax is not in line with an intuitive interpretation of attention as distributing “fractions” of an overall attention budget among inputs, a growing body of work shows that the attention weights distribution does not directly correlate with pre”
Faithful Chain-of-Thought Reasoning (2023, background): “…looks plausible (i.e., convincing to humans) (Jacovi and Goldberg, 2020), this makes it easier for people (e.g., legal practitioners) to over-trust the model (e.g., a recidivism predictor) even if it has implicit biases (e.g., against racial minorities) (Pruthi et al., 2020; Slack et al., 2020).”
How Reliable are Model Diagnostics? (2021, background): “Attention-based interpretation can also be unreliable and manipulable to the point of deceiving practitioners, as Pruthi et al. (2020) and Jain and Wallace (2019) show.”