Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings

By Gregor Wiedemann, Steffen Remus, Avi Chawla, Chris Biemann. 2019

201 citations recorded by the source corpus.

Contextualized word embeddings (CWE) such as provided by ELMo (Peters et al., 2018), Flair NLP (Akbik et al., 2018), or BERT (Devlin et al., 2019) are a major recent innovation in NLP. CWEs provide semantic vector representations of words depending on their respective context. Their advantage over static word embeddings has been shown for a number of tasks, such as text classification, sequence tagging, or machine translation. Since vectors of the same word type can vary depending on the respective context, they implicitly provide a model for word sense disambiguation (WSD). We introduce a simple but effective approach to WSD using a nearest neighbor classification on CWEs. We compare the performance of different CWE models for the task and can report improvements above the current state of the art for two standard WSD benchmark datasets. We further show that the pre-trained BERT model is able to place polysemic words into distinct 'sense' regions of the embedding space, while ELMo and Flair NLP do not seem to possess this ability.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “At a high level, mechanisms for interpreting BERT can be categorized into three main categories: interpreting the learned embeddings [1, 16, 22, 47, 85], BERT’s learned knowledge of syntax [26, 29, 30, 43, 45, 74], and BERT’s learned knowledge of semantics [23, 74].”
Implicit Representations of Meaning in Neural Language Models (2021, background): “Within the realm of semantics, existing work has identi-ﬁed representations of word meaning (e.g., ﬁne-grained word senses; Wiedemann et al. 2019) and predicate–argument structures like frames and semantic roles (Kovaleva et al., 2019).”
Word meaning in minds and machines (2020, background): “Taking these results and related studies together (Ethayarajh, 2020; Wiedemann et al., 2019), it ’ s clear that BERT has some ability to resolve the meaning of homonyms given context.”
Analysing Lexical Semantic Change with Contextualised Word Representations (2020, methodology): “…usage similarity judgements of the WiC evaluation set (Pilehvar and Camacho-Collados, 2019), performing on a par with state-of-the-art word sense disambiguation models (Wiedemann et al., 2019), and proving useful for the supervised derivation of time-speciﬁc sense representation (Hu et al., 2019).”