On Identifiability in Transformers

By Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, Roger Wattenhofer. 2019

212 citations recorded by the source corpus.

In this paper we delve deep in the Transformer architecture by investigating two of its core components: self-attention and contextual embeddings. In particular, we study the identifiability of attention weights and token embeddings, and the aggregation of context into hidden tokens. We show that, for sequences longer than the attention head dimension, attention weights are not identifiable. We propose effective attention as a complementary tool for improving explanatory interpretations based on attention. Furthermore, we show that input tokens retain to a large degree their identity across the model. We also find evidence suggesting that identity information is mainly encoded in the angle of the embeddings and gradually decreases with depth. Finally, we demonstrate strong mixing of input information in the generation of contextual embeddings by means of a novel quantification method based on gradient attribution. Overall, we show that self-attention distributions are not directly interpretable and present tools to better understand and further investigate Transformer models.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Measuring Association Between Labels and Free-Text Rationales (2020, methodology): “We propose to measure labelrationale association within self-rationalizing models that produce free-text rationales, much like how prior work has quantified the faithfulness of attention (Jain and Wallace, 2019; Brunner et al., 2020).”
Of Non-Linearity and Commutativity in BERT (2021, background): “Cosine similarity is a common choice to measure similarity between words in an embedding space [3], [8], [22] .”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “Interpreting attention on language sequences is a well-established area of research [12, 86, 89].”
ERASER: A Benchmark to Evaluate Rationalized NLP Models (2019, background): “However, attention weights do not in general provide faithful explanations for predictions (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Zhong et al., 2019; Pruthi et al., 2020; Brunner et al., 2020; Moradi et al., 2019; Vashishth et al., 2019).”
Learning to Deceive with Attention-Based Explanations (2019, background): “…to neighboring representations.7 In contrast, the Embedding model (which has no means to pass the information pre-attention) at- 7 A recent study (Brunner et al., 2019) similarly observes a high degree of ‘mixing’ of information across layers in Transformer models. tains only about 50% test…”
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021, background): “Moreover, they provide attention-based explanations whose interpretability is unclear (Serrano and Smith, 2019; Brunner et al., 2020; Wiegreffe and Pinter, 2019).”
When BERT Plays the Lottery, All Tickets Are Winning (2020, background): “It also adds to the ongoing discussion of interpretability of self-attention (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Brunner et al., 2020).”
RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers (2019, background): “Brunner et al. (2020) suggest that representations produced by Transformers mix the information from different positions and cease to be directly interpretable after 2+ layers, which might explain our observations.”
Self-Attention Attribution: Interpreting Information Interactions Inside Transformer (2020, neutral): “In contrast, some other work argues that self-attention distributions are not directly interpretable (Serrano and Smith 2019; Jain and Wallace 2019; Brunner et al. 2020).”
Explaining and Improving Model Behavior with k Nearest Neighbor Representations (2020, methodology): “2017) models in an attempt to make it more interpretable (Pruthi et al. 2020; Brunner et al. 2020).”
Limits to Depth Efficiencies of Self-Attention (2020, result): “…linked to the need to transform the Key/Query similarity score into a distribution, has been recently challenged, as a growing body of work shows that the attention weights distribution does not directly correlate with predictions [Jain and Wallace, 2019, Pruthi et al., 2019, Brunner et al., 2020].”
Similarity Analysis of Contextual Word Representation Models (2020, background): “One might mention in this context concerns regarding the reliability of attention weights for interpreting the importance of input words in a model (Jain and Wallace, 2019; Serrano and Smith, 2019; Brunner et al., 2020).”
Telling BERT’s Full Story: from Local Attention to Global Aggregation (2020, methodology): “We first show that self-attention distributions correlate strongly with Hidden Token Attribution (Brunner et al., 2020) (HTA) from hidden embedding to head output, this result validates HTA.”
Attention mechanism in neural networks: where it comes and where it goes (2022, methodology): “For a better understanding of Transformers, an identifiability analysis of self-attention weights is conducted in addition to presenting effective attention to improve explanatory interpretations [168].”
What do you mean, BERT? Assessing BERT as a Distributional Semantics Model (2019, background): “Recent research has however questioned the pertinence of these attention-based analyses (Serrano and Smith, 2019; Brunner et al., 2019); moreover these works have focused more on the inner workings of the networks than on their adequacy with theories of meaning.”
TabTransformer: Tabular Data Modeling Using Contextual Embeddings (2020, background): “The effectiveness and interpretability of contextual embeddings generated by Transformers have been also well studied (Coenen et al. 2019; Brunner et al. 2019).”
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation (2020, background): “The first line of research focuses on the interpretation of the network, in particular on the analysis of attention mechanisms and the interpretability of the weights and connections (Raganato and Tiedemann, 2018; Tang et al., 2018; Mareček and Rosa, 2019; Voita et al., 2019a; Brunner et al., 2020).”