What do you mean, BERT? Assessing BERT as a Distributional Semantics Model
By Timothee Mickus, Denis Paperno, Mathieu Constant, Kees van Deemter. 2019
49 citations recorded by the source corpus.
Contextualized word embeddings, i.e. vector representations for words in context, are naturally seen as an extension of previous noncontextual distributional semantic models. In this work, we focus on BERT, a deep neural network that produces contextualized embeddings and has set the state-of-the-art in several semantic tasks, and study the semantic coherence of its embedding space. While showing a tendency towards coherence, BERT does not fully live up to the natural expectations for a semantic vector space. In particular, we find that the position of the sentence in which a word occurs, while having no meaning correlates, leaves a noticeable trace on the word embeddings and disturbs similarity relationships.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Ivan Vulic
- Caiming Xiong
- Richard Socher
- Yonatan Belinkov
- Anna Rogers
- Mohit Bansal
- Luke S. Zettlemoyer
- Goran Glavas
Citations in the corpus
- BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “At a high level, mechanisms for interpreting BERT can be categorized into three main categories: interpreting the learned embeddings [1, 16, 22, 47, 85], BERT’s learned knowledge of syntax [26, 29, 30, 43, 45, 74], and BERT’s learned knowledge of semantics [23, 74].”
- A Primer in BERTology: What We Know About How BERT Works (2020, background): “However, Mickus et al. (2019) note that representations of the same word varies depending on position of the sentence in which it occurs, likely due to NSP objective.”
- Probing Pretrained Language Models for Lexical Semantics (2020, background): “First, representations are assigned to word tokens and are affected by the current context and position within a sentence (Mickus et al., 2020).”