Diachronic word embeddings and semantic shifts: a survey

By Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski, Erik Velldal. 2018

354 citations recorded by the source corpus.

Recent years have witnessed a surge of publications aimed at tracing temporal changes in lexical semantics using distributional methods, particularly prediction-based word embedding models. However, this vein of research lacks the cohesion, common terminology and shared practices of more established areas of natural language processing. In this paper, we survey the current state of academic research related to diachronic word embeddings and semantic shifts detection. We start with discussing the notion of semantic shifts, and then continue with an overview of the existing methods for tracing such time-related shifts with word embedding models. We propose several axes along which these methods can be compared, and outline the main challenges before this emerging subfield of NLP, as well as prospects and possible applications.

Subfields: Artificial Intelligence, Cultural Studies, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cultural Studies

In Statistical and Nonlinear Physics

Citations in the corpus

Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “There is an established consensus that the meaning of words changes over time due to complicated linguistic and social processes (e.g. Kutuzov et al., 2018; Shoemark et al., 2019).”
Dynamic Language Models for Continuously Evolving Content (2021, background): “Prior works show that new words and semantic evolution pose a crucial challenge in many NLP tasks, leading to a significant performance drop for word embedding based models (eg, word2vec [30]) [24, 32].”
Room to Glo: A Systematic Comparison of Semantic Change Detection Approaches with Word Embeddings (2019, methodology): “A major challenge in developing semantic change detection systems is obtaining ground truth data (Kutuzov et al., 2018), which has so far prevented a systematic evaluation of different approaches.”
Harm inflation: Making sense of concept creep (2020, background): “There is ample evidence from sociolinguistics that language change commonly reflects societal and cultural shifts of the sort implicated in concept creep (Blank & Koch, 2013; Kutuzov et al., 2018).”
Analysing Lexical Semantic Change with Contextualised Word Representations (2020, methodology): “…with the exception of a few methods based on word frequencies and parts of speech (Michel et al., 2011; Kulkarni et al., 2015), lexical semantic change detection has been addressed following two main approaches: form-based and sense-based (for an overview, see Kutuzov et al., 2018; Tang, 2018).”
Neural Temporality Adaptation for Document Classification: Diachronic Word Embeddings and Domain Adaptation Models (2019, methodology): “Recent research has used diachronic word embeddings to study how language changes over time (Kulkarni et al., 2015; Hamilton et al., 2016; Kutuzov et al., 2018).”
A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication (2021, background): “Several authors have studied the problems of semantic drift in WordNet and other training sets [5, 65, 83] and proposed ways to address and mitigate them from a technical perspective [39, 52].”
HistoRAG: Embedding Historical Methodology in Retrieval-Augmented Generation Through Critical Technical Practice (2026, unknown): “Discourse evolves over time (Hamilton et al., 2016; Kutuzov et al., 2018), and chunks from periods where terminology most closely matches contemporary query formulations will be systematically over-represented, while earlier periods (where the same phenomena were described in now-unfamiliar…”
A Model of the Language Process (2026, unknown): “Benchmarking the semantic change detection performance of a model is challenging due to the lack of strong standardized datasets for the task (Kutuzov et al., 2018; Dubossarsky et al., 2019; Schlechtweg et al., 2020).”