Staying True to Your Word: (How) Can Attention Become Explanation?

By Martin Tutek, Jan Snajder. 2020

31 citations recorded by the source corpus.

The attention mechanism has quickly become ubiquitous in NLP. In addition to improving performance of models, attention has been widely used as a glimpse into the inner workings of NLP models. The latter aspect has in the recent years become a common topic of discussion, most notably in recent work of Jain and Wallace; Wiegreffe and Pinter. With the shortcomings of using attention weights as a tool of transparency revealed, the attention mechanism has been stuck in a limbo without concrete proof when and whether it can be used as an explanation. In this paper, we provide an explanation as to why attention has seen rightful critique when used with recurrent networks in sequence classification tasks. We propose a remedy to these issues in the form of a word level objective and our findings give credibility for attention to provide faithful interpretations of recurrent models.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Measuring Association Between Labels and Free-Text Rationales (2020, methodology): “Analysis of NLP Models Structural tests for analyzing models’ internals include probing (Tenney et al., 2019) and attention analysis (Jain and Wal-lace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Tutek and Snajder, 2020).”