Visualizing and Measuring the Geometry of BERT

By Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda B. Viégas, Martin Wattenberg. 2019

480 citations recorded by the source corpus.

Transformer architectures show significant promise for natural language processing. Given that a single pretrained model can be fine-tuned to perform well on many different tasks, these networks appear to extract generally useful linguistic features. A natural question is how such networks represent this information internally. This paper describes qualitative and quantitative investigations of one particularly effective model, BERT. At a high level, linguistic features seem to be represented in separate semantic and syntactic subspaces. We find evidence of a fine-grained geometric representation of word senses. We also present empirical descriptions of syntactic representations in both attention matrices and individual word embeddings, as well as a mathematical argument to explain the geometry of these representations.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

oLMpics-On What Language Model Pre-training Captures (2019, background): “revealing that they encode substantial amounts of syntax and semantics (Linzen et al., 2016b; Tenney et al., 2019b,a; Shwartz and Dagan, 2019; Lin et al., 2019; Coenen et al., 2019).”
Quantifying Attention Flow in Transformers (2020, background): “…the easiest and most popular approach to interpret a model’s decisions and to gain insights about its internals (Vaswani et al., 2017; Xu et al., 2015; Wang et al., 2016; Lee et al., 2017; Dehghani et al., 2019; Rockt¨aschel et al., 2016; Chen and Ji, 2019; Coenen et al., 2019; Clark et al., 2019).”
Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving (2019, methodology): “This was shown for BERT’s hidden activation states (Hewitt & Manning, 2019; Tenney et al., 2019) and, most directly related to the present work, for the graph implicit in BERT’s attention weights (Coenen et al., 2019; Lin et al., 2019).”
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (2020, background): “, 2019), syntax (Tenney et al., 2019b; Lin et al., 2019; Reif et al., 2019; Hewitt and Manning, 2019; Liu et al., 2019a), world knowledge (Petroni et al.”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, methodology): “We run both embedding probes, which assess the knowledge encoded in the output embeddings of each layer, and attention probes (Reif et al., 2019; Clark et al., 2019), which measure the knowledge contained in the attention weights for pairwise features.”
BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance (2019, background): “, 2019), or a variety of psycholinguistic phenomena (Ettinger, 2020); its internal workings (Coenen et al., 2019; Tenney et al., 2019a; Clark et al., 2019); or its inductive biases (Warstadt and Bowman, 2020).”
On Identifiability in Transformers (2019, methodology): “In this way, we can compare to other works in the literature (Vig, 2019; Clark et al., 2019; Klein & Nabi, 2019; Coenen et al., 2019; Lin et al., 2019) by using the publicly available pretrained BERT model in our analyses without ﬁne-tuning it to a speciﬁc task.”
Pre-trained models for natural language processing: A survey (2020, methodology): “and constituency trees from BERT, which proved the BERT’s ability 10 QIU XP, et al. Pre-trained Models for Natural Language Processing: A Survey March (2020) to encode sentence structure. Reif et al. [136] explored the geometry of internal representations in BERT and ﬁnd some evidences: 1) linguisti”
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining (2021, background): “The t-SNE [9] plot in Figure 6 further demonstrates the beneﬁts of SCL.”
Probing Pretrained Language Models for Lexical Semantics (2020, background): “These consist in learned classifier predicting annotations like POS tags (Pimentel et al., 2020) and word senses (Peters et al., 2018; Reif et al., 2019; Chang and Chen, 2019), or linear transformations to a space where distances mirror dependency tree structures (Hewitt and Manning, 2019).1 In…”
Large pre-trained language models contain human-like biases of what is right and wrong to do (2021, background): “[58], there is no strict evidence it should actually be Euclidean.”
Dissecting Recall of Factual Associations in Auto-Regressive Language Models (2023, unknown): “More broadly, our approach relates to studies of how LMs organize information internally (Reif et al., 2019; Hewitt and Manning, 2019).”
Asking without Telling: Exploring Latent Ontologies in Contextual Representations (2020, methodology): “Descriptive analysis In contrast to predictive methods, which assess an encoder with respect to particular data, descriptive methods analyze models on their own terms, and include clustering, visualization (Reif et al., 2019), and correlation”
Mitigating Political Bias in Language Models Through Reinforced Calibration (2021, methodology): “These methods, however, can not be easily adapted to recent LMs because the embedding of LMs are often context-aware and encoded with other meta-features such as positions (Reif et al. 2019).”
The Linear Representation Hypothesis and the Geometry of Large Language Models (2023, background): “…of representations There is a line of work that studies the geometry of word and sentence representations (e.g., Arora et al., 2016; Mimno & Thompson, 2017; Etha-yarajh, 2019; Reif et al., 2019; Li et al., 2020; Hewitt & Manning, 2019; Chen et al., 2021; Chang et al., 2022; Jiang et al., 2023).”
What BERT Sees: Cross-Modal Transfer for Visual Question Generation (2020, methodology): “We represent these captions and images vectors with the special BERT token [CLS], following previous works (Reif et al., 2019) where [CLS] is used to represent the entire sequence.”
Learning and Evaluating Contextual Embedding of Source Code (2019, methodology): “Further, Figures 1–5 show visualizations of the attention matrix of the last layer of the ﬁne-tuned CuBERT model (Coenen et al., 2019) for the code snippets.”