Quantifying Attention Flow in Transformers

By Samira Abnar, Willem H. Zuidema. 2020

1,293 citations recorded by the source corpus.

In the Transformer model, “self-attention” combines information from attended embeddings into the representation of the focal embedding in the next layer. Thus, across layers of the Transformer, information originating from different tokens gets increasingly mixed. This makes attention weights unreliable as explanations probes. In this paper, we consider the problem of quantifying this flow of information through self-attention. We propose two methods for approximating the attention to input tokens given attention weights, attention rollout and attention flow, as post hoc methods when we use attention weights as the relative relevance of the input tokens. We show that these methods give complementary views on the flow of information, and compared to raw attention, both yield higher correlations with importance scores of input tokens obtained using an ablation method and input gradients.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Foundation Posteriors for Approximate Probabilistic Inference (2022, neutral): “Prior works [6, 60, 1, 14] using transformer networks in natural language have re-purposed the”
A Generalist Agent (2022, positive): “Using this method, we found the attention maps at the first layer the transformer to be most interpretable, agreeing with the findings of Abnar & Zuidema (2020).”
Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training (2021, methodology): “Figure 3: Inter-Modality Flow (IMF) from the input to each layer of the multi-modal Transformer F 1,j inter, j ∈ [1, 12] calculated on ViLT [25], SOHO [18], UNITER [9] and our model.”
Post-hoc Interpretability for Neural NLP: A Survey (2021, background): “For this reason, a n w trend in NLP is to develop architecture specific input feature explanations [1, 21], for example using attention.”
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, background): “Attention and gating are frequently used in NLP models, and there have been plenty of work aiming to understand the model through these weights [2, 40, 111, 112] and investigate the reliability of using them as explanations [55, 103, 129].”
Telling BERT’s Full Story: from Local Attention to Global Aggregation (2020, methodology): “Our method and results are complementary to those in (Abnar and Zuidema, 2020), we believe a combination of attention flow and HTA may provide new interesting insights.”
Aggregating Nested Transformers (2021, background): “ViT is given high expectation to improve vision tasks beyond image classification, with existing studies on generative modeling [11, 27, 28, 39], video understanding [2, 38], segmentation and detection [30, 34, 53], interpretability [1, 6, 7].”
Conformer: Local Features Coupling Global Representations for Visual Recognition (2021, methodology): “(b) Attention maps in DeiT-S and the transformer branch of Conformer-S by using the Attention Rollout method [1].”
Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models (2023, background): “Obtaining Explainable Image Generators The extent to which attention can be used as an explanation has been widely explored [1, 10,11].”
Revealing Occlusions with 4D Neural Fields (2022, methodology): “Specifically, we adapt attention rollout [1] to our architecture, and visualize the input points (across time) that contribute the most to the specific output class we want to analyze.”
ResViT: Residual Vision Transformers for Multimodal Medical Image Synthesis (2021, methodology): “Attention maps were calculated based on the Attention Rollout technique, and a single average map was extracted for a given transformer module [102].”
VidTr: Video Transformer Without Convolutions (2021, methodology): “VidTr-S 8 8 VidTr-M 16 4 VidTr-L 32 2 C-VidTr-S 8 8 [1,2,4] [6,4,2] C-VidTr-M 16 4 [1,2,4] [8,4,2]”
Universal Multimodal Representation for Language Understanding (2023, methodology): “” For comparison, the baseline is implemented following Abnar and Zuidema [86].”