What Does BERT Look at? An Analysis of BERT’s Attention

By Kevin Clark, Urvashi Khandelwal, Omer Levy, Christopher D. Manning. 2019

2,073 citations recorded by the source corpus.

Large pre-trained neural networks such as BERT have had great recent success in NLP, motivating a growing body of research investigating what aspects of language they are able to learn from unlabeled data. Most recent analysis has focused on model outputs (e.g., language model surprisal) or internal vector representations (e.g., probing classifiers). Complementary to these works, we propose methods for analyzing the attention mechanisms of pre-trained models and apply them to BERT. BERT’s attention heads exhibit patterns such as attending to delimiter tokens, specific positional offsets, or broadly attending over the whole sentence, with heads in the same layer often exhibiting similar behaviors. We further show that certain attention heads correspond well to linguistic notions of syntax and coreference. For example, we find heads that attend to the direct objects of verbs, determiners of nouns, objects of prepositions, and coreferent mentions with remarkably high accuracy. Lastly, we propose an attention-based probing classifier and use it to further demonstrate that substantial syntactic information is captured in BERT’s attention.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

TinyBERT: Distilling BERT for Natural Language Understanding (2019, methodology): “The attention based ﬁtting is inspired by the recent ﬁndings (Clark et al., 2019) that the attention weights learned by BERT can capture substantial linguistic knowledge, which encourages that the linguistic knowledge can be well transferred from teacher BERT to student TinyBERT.”
Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning. (2021, background): “Development psychologists (Cocking et al., 1988; Mestre, 2013) often argue for the necessity of learning languages and point out that those with limited language skills are in danger of under-performing at school.”
Big Bird: Transformers for Longer Sequences (2020, background): “Most pertinently, Clark et al. [19] investigated self-attention models in NLP tasks and concluded that that neighboring inner-products are extremely important.”
Longformer: The Long-Document Transformer (2020, methodology): “…of randomly initializing the new position embeddings, we initialize them by copying the 512 position embeddings from RoBERTa multiple times as analysis of BERT’s attention heads shows a strong learned bias to attending to local context, including the previous or next token (Clark et al., 2019).”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “Some examples include Tenney et al. (2019), Michel et al. (2019), Clark et al. (2019b).”
Quantifying Attention Flow in Transformers (2020, background): “…the easiest and most popular approach to interpret a model’s decisions and to gain insights about its internals (Vaswani et al., 2017; Xu et al., 2015; Wang et al., 2016; Lee et al., 2017; Dehghani et al., 2019; Rockt¨aschel et al., 2016; Chen and Ji, 2019; Coenen et al., 2019; Clark et al., 2019).”
VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, background): “Syntactic Grounding Given that many have observed that the attention heads of BERT can discover syntactic relationships (Voita et al., 2019; Clark et al., 2019), we also analyze how grounding information is passed through syntactic relationships that VisualBERT may have discovered.”
CogView: Mastering Text-to-Image Generation via Transformers (2021, positive): “One possible explanation is that those last layers tend to concentrate on current token to determine the output token, and attention to separator tokens may be used as a no-op for attention heads which does not substantially change model’s output, similar to the analysis in BERT [10].”
RuleBERT: Teaching Soft Rules to Pre-Trained Language Models (2021, methodology): “Our proposal is different from work on Question Answering (QA) with implicit reasoning based on common-sense knowledge (Clark et al., 2019a), as we rely purely on deductive logic from explicitly stated rules.”
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, background): “Some works show that self-attention distributions of pre-trained LMs capture a rich hierarchy of linguistic information (Jawahar et al., 2019; Clark et al., 2019).”
Predictability and Surprise in Large Generative Models (2022, background): “, [17]), and specifically a new approach known as mechanistic interpretability [25] - attempting to reverse engineer the computations performed by transformers (which underpin many of the generative models discussed in this paper) gives researchers a way to better understand how models behave.”
GPT Understands, Too (2021, background): “The tasks include question answering (BoolQ (Clark et al., 2019a) & MultiRC (Khashabi et al., 2018)), textual entailment (CB (De Marneffe et al., 2019) & RTE (Dagan et al., 2005)), co-reference resolution (WiC (Pilehvar and Camacho-Collados, 2018)), causal reasoning (COPA (Roemmele et al., 2011)),…”
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (2020, background): “Existing literature argues that in attention-based pre-trained models, individual layers specialize separately (Clark et al., 2019); therefore, it is useful to incorporate a notion of structure when computing d 90 .”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “ith investigating the inner working of large-scale pretrained models and trying to build a science on top of these empirical results. Some examples include Tenney et al. (2019), Michel et al. (2019), Clark et al. (2019b). Transformers aims at facilitating and increasing the scope of these studies by”
MERLOT: Multimodal Neural Script Knowledge Models (2021, background): “The model places a lot of attention on the START token, a pattern noticed by prior work as well [24], perhaps because we pool representations from those positions (for matching with the video frames).”
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (2021, background): “We study the challenge of achieving task-agnostic pre-training for multimodal video understanding, building on recent unimodal approaches such as pretrained language models for text (Peters et al., 2018; Devlin et al., 2019).”
Knowledge Neurons in Pretrained Transformers (2021, background): “Most previous work focuses on the self-attention module (Voita et al., 2019; Clark et al., 2019; Vig and Belinkov, 2019; Hao et al., 2021).”
Of Non-Linearity and Commutativity in BERT (2021, background): “In this field, a number of studies look into attention distributions in order to understand transformers’ decisions and their internal dynamics [4], [11], [16].”