Do Attention Heads in BERT Track Syntactic Dependencies?

By Phu Mon Htut, Jason Phang, Shikha Bordia, Samuel R. Bowman. 2019

151 citations recorded by the source corpus.

We investigate the extent to which individual attention heads in pretrained transformer language models, such as BERT and RoBERTa, implicitly capture syntactic dependency relations. We employ two methods---taking the maximum attention weight and computing the maximum spanning tree---to extract implicit dependency relations from the attention weights of each layer/head, and compare them to the ground-truth Universal Dependency (UD) trees. We show that, for some UD relation types, there exist heads that can recover the dependency type significantly better than baselines on parsed English text, suggesting that some self-attention heads act as a proxy for syntactic structure. We also analyze BERT fine-tuned on two datasets---the syntax-oriented CoLA and the semantics-oriented MNLI---to investigate whether fine-tuning affects the patterns of their self-attention, but we do not observe substantial differences in the overall dependency relations extracted using our methods. Our results suggest that these models have some specialist attention heads that track individual dependency types, but no generalist head that performs holistic parsing significantly better than a trivial baseline, and that analyzing attention weights directly may not reveal much of the syntactic knowledge that BERT-style models are known to learn.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Natural Language Processing

In Model Architectures

Citations in the corpus

Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “This is consistent with the findings of Htut et al. (2019) who do not find a straightforward connection between attention weights and the model’s syntactic behavior.”
Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “These scores may come from attention weights ( Raganato and Tiedemann 2018; Clark et al. 2019; Mareˇcek and Rosa 2019; Htut et al. 2019) or from distances between word representations, perhaps including perturbations of the input sentence (Wu et al. 2020).”
Of Non-Linearity and Commutativity in BERT (2021, background): “In this field, a number of studies look into attention distributions in order to understand transformers’ decisions and their internal dynamics (Clark et al., 2019; Htut et al., 2019; Kovaleva et al., 2019).”
A Primer in BERTology: What We Know About How BERT Works (2020, methodology): “Htut et al. (2019) and Clark et al. (2019) report that there are BERT heads that attended signiﬁcantly more than a random baseline to words in certain syntactic positions.”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “At a high level, mechanisms for interpreting BERT can be categorized into three main categories: interpreting the learned embeddings [1, 16, 23, 49, 86], BERT’s learned knowledge of syntax [27, 30, 32, 45, 47, 76], and BERT’s learned knowledge of semantics [24, 76].”
Pre-Trained Models: Past, Present and Future (2021, methodology): “To better characterize the behaviors of PTMs ’ attention matrices, Htut et al. (2019) propose to take the maximum attention weight and compute the maximum spanning tree as two statistics.”
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models (2020, background): “Analysis on BERT self-attention [7,12,16] showed that BERT can learn syntactic relations, and a limited set of attention patterns are repeated across different heads.”
Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work? (2020, methodology): “…Warstadt et al., 2019a; Lin et al., 2019a; Hewitt and Manning, 2019; Jawahar et al., 2019), analyzing the attention heads of BERT (Clark et al., 2019b; Coenen et al., 2019; Lin et al., 2019a; Htut et al., 2019), and testing the linguistic generalizations of BERT across runs (McCoy et al., 2019).”
When BERT Plays the Lottery, All Tickets Are Winning (2020, background): “ers (Htut et al., 2019; Clark et al., 2019; Lin et al., 2019; Vig and Belinkov, 2019; Hewitt and Manning, 2019; Goldberg, 2019; Tenney et al., 2019) with various probing tasks, but probing tasks do not show whether a piece of knowledge is actually”
Investigating Gender Bias in Language Models Using Causal Mediation Analysis (2020, background): “The specialization of attention heads seems to be a general property of Transformers (Voita et al., 2019) and has been observed for a range of syntactic dependency relationships (Clark et al., 2019; Htut et al., 2019; Vig and Belinkov, 2019).”
Telling BERT’s Full Story: from Local Attention to Global Aggregation (2020, background): “, 2019) and (Htut et al., 2019) study the attention distributions of BERT’s heads to draw conclusions about the syntactic knowledge of the heads.”
Probing Classifiers: Promises, Shortcomings, and Alternatives (2021, background): “These scores may come from attention weights (Raganato and Tiedemann 2018; Clark et al. 2019; Mareˇcek and Rosa 2019; Htut et al. 2019) or from distances between word representations, perhaps including perturbations of the input sentence ( Wu et al. 2020).”
Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT (2020, methodology): “(2) A random BERT baseline, with which we randomly initialize weights of the BERT model (Htut et al., 2019), then use our methods to induce dependency trees.”
Gender Bias in Multilingual Neural Machine Translation: The Architecture Matters (2020, methodology): “Previous works have studied the relationship between the attention mechanism and the model output (Clark et al., 2019; Kovaleva et al., 2019; Reif et al., 2019; Lin et al., 2019; Mareček and Rosa, 2019; Htut et al., 2019; Raganato and Tiedemann, 2018; Tang et al., 2018).”
Picking BERT’s Brain: Probing for Linguistic Dependencies in Contextualized Embeddings Using Representational Similarity Analysis (2020, background): “One final approach that has been used to study whether linguistic dependencies are captured in BERT is to analyze whether specific attention heads track particular dependency relationships (Clark et al., 2019; Htut et al., 2019).”
The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models (2021, background): “3, 6, 7, 8 and the ﬁndings of Htut et al. (2019) and Clark et al. (2019) on BERT, but TISA makes this directly visible in an interpretable model, without having to probe correlations in a black box.”