Temporal Analysis of Language through Neural Language Models

By Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, Slav Petrov. 2014

363 citations recorded by the source corpus.

We provide a method for automatically detecting change in language across time through a chronologically trained neural language model. We train the model on the Google Books Ngram corpus to obtain word vector representations specific to each year, and identify words that have changed significantly from 1900 to 2009. The model identifies words such as cell and gay as having changed during that time period. The model simultaneously identifies the specific years during which such words underwent change.

Subfields: Artificial Intelligence, Cultural Studies, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cultural Studies

In Natural Language Processing

Citations in the corpus

Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change (2016, methodology): “SGNS has the beneﬁt of allowing incremental initialization during learning, where the embed-dings for time t are initialized with the embed-dings from time t − ∆ (Kim et al., 2014).”
Siamese CBOW: Optimizing Word Embeddings for Sentence Representations (2016, positive): “…proven to be beneﬁcial in a variety of tasks in NLP such as machine translation (Zou et al., 2013), parsing (Chen and Manning, 2014), semantic search (Reinanda et al., 2015; Voskarides et al., 2015), and tracking the meaning of words and concepts over time (Kim et al., 2014; Kenter et al., 2015).”
Entity Cloze By Date: What LMs Know About Unseen Entities (2022, background): “Another line of work has looked at diachronic embeddings: (Wijaya and Yeniterzi, 2011; Kim et al., 2014; Hamilton et al., 2016; Bamler and Mandt, 2017), which can model changing meanings of words over time.”
Dynamic Word Embeddings via Skip-Gram Filtering (2017, methodology): “In contrast, in SGP (middle) (Kim et al., 2014) and SGI (bottom) (Hamilton et al., 2016), the distribution of distances jumps from the ﬁrst to the second year but then remains largely stationary.”
Neural Temporality Adaptation for Document Classification: Diachronic Word Embeddings and Domain Adaptation Models (2019, methodology): “We use incremental training (Kim et al., 2014) (abbreviated In-cre , using fastText), linear regression (Kulkarni et al., 2015), implemented in scikit-learn, and Pro-crustes (Hamilton et al., 2016), implemented in SciPy.”
Diachronic Embedding for Temporal Knowledge Graph Completion (2019, background): “…embeddings is similar to diachronic word embeddings where a corpus is typically broken temporally into slices (e.g., 20-year chuncks of a 200-year corpus) and embeddings are learned for words in each chunk thus providing word embeddings that are a function of time (see, e.g., [28, 32, 18, 3]).”
Evaluating the Stability of Embedding-based Word Similarities (2018, neutral): “Other studies use cosine similarities between embeddings to measure the variation of language across geographical areas (Kulkarni et al., 2016; Phillips et al., 2017) and time (Kim et al., 2014).”
Analysing Lexical Semantic Change with Contextualised Word Representations (2020, background): “…detection , the task of deciding whether and to what extent the concept evoked by a word has changed between time periods (e.g., Gulordava and Ba-roni, 2011; Kim et al., 2014; Kulkarni et al., 2015; Del Tredici et al., 2019; Hamilton et al., 2016; Bam-ler and Mandt, 2017; Rosenfeld and Erk, 2018).”
Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Change (2019, neutral): “…in different families: count-based semantic spaces (Sagi et al., 2009; Gulordava and Baroni, 2011) and more recently based on neural embed-dings (Kim et al., 2014; Basile et al., 2016; Kulka-rni et al., 2015; Hamilton et al., 2016b); graph-based models (Tahmasebi and Risse, 2017a; Mitra et al.,…”
Statistically Significant Detection of Linguistic Change (2014, methodology): “[19] work, our technique warps the embeddings spaces of the different time snapshots after the training, allowing for efficient training that could be parallelized for large corpora.”
Dynamic Word Embeddings (2017, methodology): “In contrast, in SGP (middle) (Kim et al., 2014) and SGI (bottom) (Hamilton et al.”
Last Words: Computational Linguistics and Deep Learning (2015, neutral): “At a recent ACL workshop, Kim et al. (2014) use a similar approach—using word2vec—to look at recent changes in the meaning of words.”
Room to Glo: A Systematic Comparison of Semantic Change Detection Approaches with Word Embeddings (2019, methodology): “The ﬁrst is the cosine distance, a common approach in previous work (Hamilton et al., 2016b; Stewart et al., 2017; Dubossarsky et al., 2017; Kim et al., 2014).”
Can Large Language Models Transform Computational Social Science? (2023, background): “, 2014) and in historical archives (Mihalcea and Nastase, 2012; Kim et al., 2014; Kulkarni et al., 2015; Rudolph and Blei, 2018)1.”
Diachronic word embeddings and semantic shifts: a survey (2018, methodology): “Kim et al. (2014) proposed the idea of incrementally updated diachronic embedding models : that is, they train a model on the year y i , and then the model for the year y i +1 is initialized with the word vectors from y i .”