Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change

By William L. Hamilton, Jure Leskovec, Dan Jurafsky. 2016

1,030 citations recorded by the source corpus.

Understanding how words change their meanings over time is key to models of language and cultural evolution, but historical data on meaning is scarce, making theories hard to develop and test. Word embeddings show promise as a diachronic tool, but have not been carefully evaluated. We develop a robust methodology for quantifying semantic change by evaluating word embeddings (PPMI, SVD, word2vec) against known historical changes. We then use this methodology to reveal statistical laws of semantic evolution. Using six historical corpora spanning four languages and two centuries, we propose two quantitative laws of semantic change: (i) the law of conformity---the rate of semantic change scales with an inverse power-law of word frequency; (ii) the law of innovation---independent of frequency, words that are more polysemous have higher rates of semantic change.

Subfields: Cultural Studies, Sociology and Political Science, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cultural Studies

In Sociology and Political Science

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Inductive Representation Learning on Large Graphs (2017, neutral): “Of course, if we had a way to match “similar” nodes between the graphs, then it could be possible to use an alignment procedure to share information between the graphs, such as the procedure proposed by [12] for aligning the output of word embedding algorithms.”
Time-Aware Language Models as Temporal Knowledge Bases (2021, background): “There is extensive prior work on learning diachronic embeddings of individual words (e.g., Wijaya and Yeniterzi, 2011; Hamilton et al., 2016; Bamler and Mandt, 2017).”
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data (2020, background): “Thus, standing meanings evolve over time as speakers can different experiences (e.g. McConnell-Ginet, 1984), and a reﬂection of such change can be observed in their changing textual distribution (e.g. Herbelot et al., 2012; Hamilton et al., 2016).”
Mind the Gap: Assessing Temporal Generalization in Neural Language Models (2021, background): “In NLP, most recent work in this area models lexical change by training word embeddings (Hamilton et al., 2016; Szymanski, 2017; Yin et al., 2018) and deep neural networks (Rosenfeld and Erk, 2018; Bjerva et al., 2019) on data of different time spans.”
Dynamic Language Models for Continuously Evolving Content (2021, methodology): “The more recent works aim to automatically detect word semantic changes where the semantics of words are in distributional representation (eg, word-context matrix) [9, 13, 36] or, recently more popular, distributed representation (ie, word embeddings) [14].”
Diachronic degradation of language models: Insights from social media (2018, methodology): “These methods can also ﬁnd application in the study of other linguistic phenomena such as pol-ysemy (Hamilton et al., 2016b; Szymanski, 2017).”
KinGDOM: Knowledge-Guided DOMain Adaptation for Sentiment Analysis (2020, background): “Research in the field of cross-domain SA has proposed diverse approaches, which include learning domain-specific sentiment words/lexicons (Sarma et al., 2018; Hamilton et al., 2016b), co-occurrence based learning (Blitzer et al.”
On learning and representing social meaning in NLP: a sociolinguistic perspective (2021, background): “Exploration of sociolinguistic questions NLP methods can support (socio)linguistic research, e.g. methods to automatically identify words that have changed meaning (Hamilton et al., 2016) or words that exhibit geographical variation (Nguyen and Eisenstein, 2017).”
Inducing Domain-Specific Sentiment Lexicons from Unlabeled Corpora (2016, background): “It is possible that certain settings of word2vec could perform better, but previous work has shown that SVD-based methods have superior results on smaller datasets and rare-word similarity tasks (Levy et al., 2015; Hamilton et al., 2016), so this result is not surprising.”
Harm inflation: Making sense of concept creep (2020, methodology): “Hamilton et al. (2016b) further demonstrated the value of systematically examining nearest neighbours of a target word to evaluate semantic changes that are due to cultural shifts.”
Relational Representation Learning for Dynamic (Knowledge) Graphs: A Survey (2019, background): “ddings, it can be used for node embeddings as well. Diachronic encoders have been also explored for word embeddings to understand how the meaning (or usage) of words has changed over time (see, e.g., Hamilton et al. (2016); Bamler and Mandt (2017)). 3. In cases where there are dierent time modiers (”
Evaluating the Stability of Embedding-based Word Similarities (2018, methodology): “For example, Hamilton et al. (2016) train separate embeddings on temporal segments of a corpus and then”
Distributional Models of Word Meaning (2018, background): “…properties of lexical items (Schütze 1997, 1998, Heylen et al. 2015); and (c) the analysis of semantic change, which involves applying DS to diachronic corpora (Hamilton et al. 2016, Rodda et al. 2017) and investigating the change in productivity of syntactic constructions (Perek 2016).”
Diachronic word embeddings and semantic shifts: a survey (2018, methodology): “At the same time, Hamilton et al. (2016a) and Hamilton et al. (2016c) showed that these two approaches can be used simultaneously: they employed both ‘second order embeddings’ and orthogonal Procrustes transformations to align diachronic models. Recently, it was shown in Bamler and Mandt (2017) (‘dy”
TimeLMs: Diachronic Language Models from Twitter (2022, methodology): “For instance, by specializing language representations derived from word embedding models or neural networks (Hamilton et al., 2016; Szymanski, 2017; Rosen-feld and Erk, 2018; Del Tredici et al., 2019; Hof-mann et al., 2021).”
Diachronic Embedding for Temporal Knowledge Graph Completion (2019, background): “…embeddings is similar to diachronic word embeddings where a corpus is typically broken temporally into slices (e.g., 20-year chuncks of a 200-year corpus) and embeddings are learned for words in each chunk thus providing word embeddings that are a function of time (see, e.g., [28, 32, 18, 3]).”
Entity Cloze By Date: What LMs Know About Unseen Entities (2022, background): “Another line of work has looked at diachronic embeddings: (Wijaya and Yeniterzi, 2011; Kim et al., 2014; Hamilton et al., 2016; Bamler and Mandt, 2017), which can model changing meanings of words over time.”
Censorship of Online Encyclopedias: Implications for NLP Models (2021, background): “This precludes naive comparison of word distances across distinct corpuses [17, 33].”