Open Sesame: Getting inside BERT’s Linguistic Knowledge

By Yongjie Lin, Y. Tan, Robert Frank. 2019

314 citations recorded by the source corpus.

How and to what extent does BERT encode syntactically-sensitive hierarchical information or positionally-sensitive linear information? Recent work has shown that contextual representations like BERT perform well on tasks that require sensitivity to linguistic structure. We present here two studies which aim to provide a better understanding of the nature of BERT’s representations. The first of these focuses on the identification of structurally-defined elements using diagnostic classifiers, while the second explores BERT’s representation of subject-verb agreement and anaphor-antecedent dependencies through a quantitative assessment of self-attention vectors. In both cases, we find that BERT encodes positional information about word tokens well on its lower layers, but switches to a hierarchically-oriented encoding on higher layers. We conclude then that BERT’s representations do indeed model linguistically relevant aspects of hierarchical structure, though they do not appear to show the sharp sensitivity to hierarchical structure that is found in human processing of reflexive anaphora.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Language and Linguistics, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Language and Linguistics

In Large Language Models

In Natural Language Processing

Citations in the corpus

oLMpics-On What Language Model Pre-training Captures (2019, background): “This has instigated research exploring what is captured by the contextualized representations that these LMs compute, revealing that they encode substantial amounts of syntax and semantics (Linzen et al., 2016b; Tenney et al., 2019b,a; Shwartz and Dagan, 2019; Lin et al., 2019; Coenen et al., 2019).”
Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving (2019, methodology): “, 2019) and, most directly related to the present work, for the graph implicit in BERT’s attention weights (Coenen et al., 2019; Lin et al., 2019).”
Factual Probing Is [MASK]: Learning vs. Learning to Recall (2021, background): “…probing, which aims to explore the linguistic properties encoded in contextualized word representations (Belinkov et al., 2017; Tenney et al., 2019; Lin et al., 2019)—for example, by seeing if a classifier can predict that “chef ” is the nominal subject of “made” given the representations returned…”
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (2020, background): “…works cover a wide variety of topics, including grammatical generalization (Goldberg, 2019; Warstadt et al., 2019), syntax (Tenney et al., 2019b; Lin et al., 2019; Reif et al., 2019; Hewitt and Manning, 2019; Liu et al., 2019a), world knowledge (Petroni et al., 2019; Jiang et al., 2020),…”
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2019, neutral): “We will treat this as the best available gold standard for human prediction in context—humans completing the cloze task typically are not under any time pressure, so they have the opportunity to use all available information from the context to arrive at a prediction.”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “At a high level, mechanisms for interpreting BERT can be categorized into three main categories: interpreting the learned embeddings [1, 16, 23, 49, 86], BERT’s learned knowledge of syntax [27, 30, 32, 45, 47, 76], and BERT’s learned knowledge of semantics [24, 76].”
Pre-Trained Models: Past, Present and Future (2021, background): “…Captions (Sharma et al., 2018), SBU Captions (Ordonez et al., 2011) or existing V & L datasets designed for specific tasks including COCO (Lin et al., 2014), Flicker30 K (Plummer et al., 2015), GQA (Hudson and Manning, 2019), VQA (Antol et al., 2015) and Visual Genome (Krishna et al.,…”
jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models (2020, methodology): “Early versions of jiant have already been used in multiple works, including probing analyses (Ten-ney et al., 2019b,a; Warstadt et al., 2019; Lin et al., 2019; Hewitt and Manning, 2019; Jawahar et al., 2019), transfer learning experiments (Wang et al., 2019a; Phang et al., 2018), and dataset and…”
BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance (2019, background): “Many recent papers have sought a deeper understanding of BERT, whether to assess its encoding of sentence structure (Lin et al., 2019; He-witt and Manning, 2019; Chrupała and Alishahi, 2019; Jawahar et al., 2019; Tenney et al., 2019b); its representational structure more generally (Ab-nar et al.,…”
Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work? (2020, methodology): “…Warstadt et al., 2019a; Lin et al., 2019a; Hewitt and Manning, 2019; Jawahar et al., 2019), analyzing the attention heads of BERT (Clark et al., 2019b; Coenen et al., 2019; Lin et al., 2019a; Htut et al., 2019), and testing the linguistic generalizations of BERT across runs (McCoy et al., 2019).”
Do Attention Heads in BERT Track Syntactic Dependencies? (2019, background): “To that end, researchers have investigated the linguistic knowledge that these models learn by analyzing BERT (Goldberg, 2018; Lin et al., 2019) directly or training probing classiﬁers on the contextualized embeddings or attention heads of BERT (Tenney et al., 2019b,a; Hewitt and Man-ning, 2019).”
On Identifiability in Transformers (2019, background): “In this way, we can compare to other works in the literature (Vig, 2019; Clark et al., 2019; Klein & Nabi, 2019; Coenen et al., 2019; Lin et al., 2019) by using the publicly available pretrained BERT model in our analyses without ﬁne-tuning it to a speciﬁc task.”
When BERT Plays the Lottery, All Tickets Are Winning (2020, background): “ers (Htut et al., 2019; Clark et al., 2019; Lin et al., 2019; Vig and Belinkov, 2019; Hewitt and Manning, 2019; Goldberg, 2019; Tenney et al., 2019) with various probing tasks, but probing tasks do not show whether a piece of knowledge is actually”
Adversarial Removal of Demographic Attributes Revisited (2019, background): “…this way is common in machine learning, including bias detection studies (Elazar and Goldberg, 2018; Zhao et al., 2019) and probing studies (Ravfogel et al., 2018; Lin et al., 2019), but is known to overestimate performance (Glober-son and Roweis, 2016), in particular for high-dimensional problems.”
Logic2Text: High-Fidelity Natural Language Generation from Logical Forms (2020, background): “Recent works (Lin et al., 2019; Rogers et al., 2020) also show that Transformer-based structure can capture hierarchical syntactic structures.”
Is Supervised Syntactic Parsing Beneficial for Language Understanding Tasks? An Empirical Investigation (2020, background): “While existing body of work (Lin et al., 2019; Tenney et al., 2019; Liu et al., 2019a; Kulmizev et al., 2020; Chi et al., 2020) probes transformers for structural phenomena, our work is more pragmatically motivated.”
UnNatural Language Inference (2020, background): “…phenomena (Linzen et al., 2016) that transformer-based models pretrained on LM do acquire some knowledge of natural language syntax (Gulordava et al., 2018; Chrupała and Alishahi, 2019; Jawahar et al., 2019; Lin et al., 2019; Manning et al., 2020; Hawkins et al., 2020; Linzen and Baroni, 2021).”