How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings

By Kawin Ethayarajh. 2019

1,289 citations recorded by the source corpus.

Replacing static word embeddings with contextualized word representations has yielded significant improvements on many NLP tasks. However, just how contextual are the contextualized representations produced by models such as ELMo and BERT? Are there infinitely many context-specific representations for each word, or are words essentially assigned one of a finite number of word-sense representations? For one, we find that the contextualized representations of all words are not isotropic in any layer of the contextualizing model. While representations of the same word in different contexts still have a greater cosine similarity than those of two different words, this self-similarity is much lower in upper layers. This suggests that upper layers of contextualizing models produce more context-specific representations, much like how upper layers of LSTMs produce more task-specific representations. In all layers of ELMo, BERT, and GPT-2, on average, less than 5% of the variance in a word’s contextualized representations can be explained by a static embedding for that word, providing some justification for the success of contextualized representations.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Of Non-Linearity and Commutativity in BERT (2021, methodology): “In [8] it is pointed out that the hidden embeddings at each transformer layer are directional and occupy cones of different sizes in their respective spaces, i.”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “At a high level, mechanisms for interpreting BERT can be categorized into three main categories: interpreting the learned embeddings [1, 16, 23, 49, 86], BERT’s learned knowledge of syntax [27, 30, 32, 45, 47, 76], and BERT’s learned knowledge of semantics [24, 76].”
Pre-Trained Models: Past, Present and Future (2021, background): “Ethayarajh (2019) also claim that PTMs would be open knowledge graphs and propose an unsupervised method to build knowledge graphs based on PTMs.”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, background): “We also draw a connection to the recent findings that pre-trained word embeddings suffer from anisotropy (Ethayarajh, 2019; Li et al., 2020) and prove that—through a spectrum perspective—the contrastive learning objective “flattens” the singular value distribution of the sentence embedding space, he”
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining (2021, methodology): “Next, we calculate the self-similarity of the representations of a same word when appearing in different contexts (Ethayarajh, 2019)—A less self-similar token representation indicates the Transformer is more contextualized.”
An Analysis of BERT in Document Ranking (2020, background): “Etha-yarajh et al. [5] found that deeper layers produce more context-specific representations.”
Contextual and Non-Contextual Word Embeddings: an in-depth Linguistic Investigation (2020, background): “Focusing instead on the geometry of the representation space, Ethayarajh (2019) ﬁrst showed that the contextualized word representations of ELMo, BERT and GPT-2 produce more context speciﬁc representations in the upper layers and then proposed a method for creating a new type of static embedding…”
Analyzing Redundancy in Pretrained Transformer Models (2020, background): “Recent research (Mu et al., 2018; Ethayarajh, 2019) analyzed contextualized embeddings in pretrained models and showed that the repre-”
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning (2021, neutral): “Although a large body of empirical work seeks to understand the effectiveness of pretrained models [7, 5, 12, 35, 34, 11, 27, 15], theoretical understanding is scarce.”
Shared computational principles for language processing in humans and deep language models (2022, background): “Taken together, these results suggest that the brain is coding for the semantic relationship among words contained in static embeddings while also being tuned to the unique contextual relationship between the specific word and the preceding words in the sequence 53 .”
Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning (2022, background): “Given that isotropy has both theoretical and empirical benefits for static embeddings (Mu & Viswanath, 2018), the extent of anisotropy in contextualized representations is surprising (Ethayarajh, 2019).”
How True is GPT-2? An Empirical Analysis of Intersectional Occupational Biases (2021, background): “In terms of models, GPT-2, a transformer-based language model trained on 40GB of text from over 8 million documents from the Internet, has been the subject of various studies (Wick et al., 2020; Budzianowski & Vulic, 2019; Ethayarajh, 2019).”
DirectProbe: Studying Representations without Classifiers (2021, background): “Another interesting observation is the decreasing performance and minimum distance of higher layers, which is also corroborated by Ethayarajh (2019) and Liu et al. (2019a).”
Outliers Dimensions that Disrupt Transformers Are Driven by Frequency (2022, background): “Transformer-based language models (LMs) have been shown to exhibit anisotropic behavior in their representations of both tokens and sentences (Ethayarajh, 2019; Gao et al., 2019; Rajaee and Pilehvar, 2021; Timkey and van Schijndel, 2021).”
Picking BERT’s Brain: Probing for Linguistic Dependencies in Contextualized Embeddings Using Representational Similarity Analysis (2020, methodology): “We use representations from the last layer because Ethayarajh (2019) found that BERT representations are more contextspecific in higher layers than lower layers.”