Statistically Significant Detection of Linguistic Change

By Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, Steven S. Skiena. 2014

502 citations recorded by the source corpus.

We propose a new computational approach for tracking and detecting statistically significant linguistic shifts in the meaning and usage of words. Such linguistic shifts are especially prevalent on the Internet, where the rapid exchange of ideas can quickly change a word's meaning. Our meta-analysis approach constructs property time series of word usage, and then uses statistically sound change point detection algorithms to identify significant linguistic shifts. We consider and analyze three approaches of increasing complexity to generate such linguistic property time series, the culmination of which uses distributional characteristics inferred from word co-occurrences. Using recently proposed deep neural language models, we first train vector representations of words for each time period. Second, we warp the vector spaces into one unified coordinate system. Finally, we construct a distance-based distributional time series for each word to track its linguistic displacement over time. We demonstrate that our approach is scalable by tracking linguistic change across years of micro-blogging using Twitter, a decade of product reviews using a corpus of movie reviews from Amazon, and a century of written books using the Google Book Ngrams. Our analysis reveals interesting patterns of language usage change commensurate with each medium.

Subfields: Artificial Intelligence, Cultural Studies, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cultural Studies

In Natural Language Processing

Citations in the corpus

Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change (2016, background): “mean shifts (Kulkarni et al., 2014) could also be used.”
Dynamic Language Models for Continuously Evolving Content (2021, background): “[23] use a linear transformation that preserves general vector space structure to align learned embeddings across time-periods and Hamilton et al.”
Dynamic Word Embeddings via Skip-Gram Filtering (2017, background): “Several authors have analyzed different statistics of text data to analyze semantic changes of words over time (Mi-halcea & Nastase, 2012; Sagi et al., 2011; Kim et al., 2014; Kulkarni et al., 2015; Hamilton et al., 2016).”
Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Change (2019, background): “This approach has been found to outperform other LSC detection methods in various studies (Hamilton et al., 2016b; Kulkarni et al., 2015).”
Diachronic Embedding for Temporal Knowledge Graph Completion (2019, unknown): “, [28, 32, 18, 3]).”
Diachronic word embeddings and semantic shifts: a survey (2018, background): “Some examples falling into this category include tracking semantic drift of particular words (Kulkarni et al., 2015) or of word sentiment (Hamilton et al.”
Can Large Language Models Transform Computational Social Science? (2023, background): “These features can be used to study the sociolinguistics of language change (Kulkarni et al., 2015; Hamilton et al., 2016b) or the linguistic biases in educational assessments (Craig and Washington, 2002) and online moderation (Sap et al.”
Analysing Lexical Semantic Change with Contextualised Word Representations (2020, methodology): “In particular, with the exception of a few methods based on word frequencies and parts of speech (Michel et al., 2011; Kulkarni et al., 2015), lexical semantic change detection has been addressed following two main approaches: form-based and sense-based (for an overview, see Kutuzov et al., 2018;…”
Semantic Variation in Online Communities of Practice (2018, background): “The most common approach consists in creating independent vector representations for consecutive time spans and then using a transformation matrix to map vectors from one space to another one (Kulkarni et al., 2015; Zhang et al., 2015; Hamilton et al., 2016).”
O N M EMORY IN H UMAN AND A RTIFICIAL L ANGUAGE P ROCESSING S YSTEMS (2020, background): “In addition, many researchers (Mikolov et al., 2013b; Kulkarni et al., 2015; Hamilton et al., 2016) have remarked upon the ability of word embeddings (Mikolov et al., 2013a) to represent concepts and their relations, which resemble information stored in a semantic memory.”
Last Words: Computational Linguistics and Deep Learning (2015, unknown): “Kulkarni et al. (2015) use neural word embeddings to model the shift in meaning of words such as gay over the last century (exploiting the online Google Books Ngrams corpus).”
The Geometry of Culture: Analyzing the Meanings of Class through Word Embeddings (2018, background): “Simply calculating the proximity of word vectors can also provide a strong indicator of the similarity or distance between word meanings (Kulkarni et al. 2015).”
Local Modeling of Attributed Graphs: Algorithms and Applications (2016, background): “We have showed in our recent work [71] that it is possible to align embeddings into a unified coordinate system.”
Diachronic degradation of language models: Insights from social media (2018, background): “Decahose (Twitter 10”
Harm inflation: Making sense of concept creep (2020, background): “Prediction-based models, which are now dominant and have been shown to outperform count-based approaches (Kulkarni et al., 2015; Schlechtweg et al., 2019), involve training language models (such as word2vec, SGNS; Mikolov et al., 2013) incrementally for each subsequent time period (“epoch”) and…”
Neural Temporality Adaptation for Document Classification: Diachronic Word Embeddings and Domain Adaptation Models (2019, methodology): “We use incremental training (Kim et al., 2014) (abbreviated In-cre , using fastText), linear regression (Kulkarni et al., 2015), implemented in scikit-learn, and Pro-crustes (Hamilton et al., 2016), implemented in SciPy.”
A Bayesian Model of Diachronic Meaning Change (2016, background): “…changes in the distribution of terms from two time periods (Popescu and Strapparava, 2013; Cook and Stevenson, 2010), to training distributional similarity models on time slices (Gulordava and Baroni, 2011; Sagi et al., 2009), and neural language models (Kim et al., 2014; Kulkarni et al., 2015).”
Dynamic Word Embeddings (2017, background): “Current approaches to learning word embeddings in a dynamic context rely on grouping the data into time bins and training the embeddings separately on these bins (Kim et al., 2014; Kulkarni et al., 2015; Hamilton et al., 2016).”