Analyzing the Structure of Attention in a Transformer Language Model

By Jesse Vig, Yonatan Belinkov. 2019

479 citations recorded by the source corpus.

The Transformer is a fully attention-based alternative to recurrent networks that has achieved state-of-the-art results across a range of NLP tasks. In this paper, we analyze the structure of attention in a Transformer language model, the GPT-2 small pretrained model. We visualize attention for individual instances and analyze the interaction between attention and syntax over a large corpus. We find that attention targets different parts of speech at different layer depths within the model, and that attention aligns with dependency relations most strongly in the middle layers. We also find that the deepest layers of the model capture the most distant relationships. Finally, we extract exemplar sentences that reveal highly specific patterns targeted by particular attention heads.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Rethinking Attention with Performers (2020, methodology): “The illustration is based on Vig et al. (Vig, 2019; Vig & Belinkov, 2019).”
How Context Affects Language Models' Factual Predictions (2020, background): “…of “probes” have been developed to analyse the syntactic structures, such as syntax trees [Marvin and Linzen, 2018, Hewitt and Manning, 2019, Vig and Belinkov, 2019], negative polarity items [Warstadt and Bowman, 2019, Warstadt et al., 2019], semantic fragments [Richardson et al., 2019],…”
Knowledge Neurons in Pretrained Transformers (2021, background): “Most previous work focuses on the self-attention mechanism (Voita et al., 2019; Clark et al., 2019; Vig and Belinkov, 2019), which is generally considered as the most useful module in Transformer.”
Transformer Feed-Forward Layers Are Key-Value Memories (2020, background): “The study of the transformer architecture has focused on the role and function of self-attention layers (Voita et al., 2019; Clark et al., 2019; Vig and Belinkov, 2019) and on inter-layer differences (i.e. lower vs. upper layers) (Tenney et al., 2019; Jawahar et al., 2019).”
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “Attention to the ﬁrst token may be null attention (Vig and Belinkov, 2019).”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “Prior work in NLP also suggests that deeper layers in text-based Transformers attend to more complex properties [83] and encode higher-level representations [36, 56, 58, 76].”
A Multiscale Visualization of Attention in the Transformer Model (2019, background): “shows that attention in the Transformer correlates with syntactic constructs such as dependency relations and part-of-speech tags (Raganato and Tiedemann, 2018; Voita et al., 2019; Vig and Belinkov, 2019).”
exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models (2019, background): “Vig and Belinkov [2019] also explored the dependency relations across heads and discovered that initial layers typically encode positional relations, middle layers capture the most dependency relations, and later layers look for unique patterns and structures.”
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020, background): “Fueled by recent advances in deep-learning and language processing, NLP systems are increasingly being used for prediction and decision-making in many fields (Vig and Belinkov, 2019), including sensitive ones such as health, commerce and law (Fort and Couillault, 2016).”
Unsupervised Quality Estimation for Neural Machine Translation (2020, background): “Recent work on attention inter-pretability showed that attention weights in Trans-former networks might not be readily interpretable (Vashishth et al., 2019; Vig and Belinkov, 2019).”
Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling (2020, methodology): “There have also been models that directly apply pre-trained language models without introducing document graphs, since edges such as dependency structures and coreferences can be automatically learned by pre-trained language models (Clark et al. 2019; Tenney, Das, and Pavlick 2019; Vig and Belinkov ”
Investigating Gender Bias in Language Models Using Causal Mediation Analysis (2020, background): “, 2019) and has been observed for a range of syntactic dependency relationships (Clark et al., 2019; Htut et al., 2019; Vig and Belinkov, 2019).”
Attention Interpretability Across NLP Tasks (2019, background): “While on the other hand, some studies have discovered how attention in neural models captures several linguistic notions of syntax and coreference (Vig & Belinkov, 2019; Clark et al., 2019; Tenney et al., 2019).”
exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models (2020, background): “Previous studies have uncovered speciﬁc attention heads that learn particular dependencies (Vig and Belinkov, 2019; Clark et al., 2019).”
Contextual and Non-Contextual Word Embeddings: an in-depth Linguistic Investigation (2020, background): “…et al., 2017) to more domain speciﬁc approaches, such as interpreting attention mechanisms (Raganato and Tiedemann, 2018; Kovaleva et al., 2019; Vig and Belinkov, 2019) or designing speciﬁc probing tasks that a model can solve only if it captures a precise linguistic phenomenon using the…”