Dynamic Word Embeddings

By Robert Bamler, Stephan Mandt. 2017

249 citations recorded by the source corpus.

We present a probabilistic language model for time-stamped text data which tracks the semantic evolution of individual words over time. The model represents words and contexts by latent trajectories in an embedding space. At each moment in time, the embedding vectors are inferred from a probabilistic version of word2vec [Mikolov et al., 2013]. These embedding vectors are connected in time through a latent diffusion process. We describe two scalable variational inference algorithms--skip-gram smoothing and skip-gram filtering--that allow us to train the model jointly over all times; thus learning on all data while simultaneously allowing word and context vectors to drift. Experimental results on three different corpora demonstrate that our dynamic model infers word embedding trajectories that are more interpretable and lead to higher predictive likelihoods than competing methods that are based on static models trained separately on time slices.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Time-Aware Language Models as Temporal Knowledge Bases (2021, background): “There is extensive prior work on learning diachronic embeddings of individual words (e.g., Wijaya and Yeniterzi, 2011; Hamilton et al., 2016; Bamler and Mandt, 2017).”
Dynamic Language Models for Continuously Evolving Content (2021, methodology): “Other works attempt to simultaneously learn time-aware embeddings over all time periods and resolve the alignment problem [4, 35, 47].”
Diachronic word embeddings and semantic shifts: a survey (2018, background): “Recently, it was shown in Bamler and Mandt (2017) (‘ dynamic skip-gram ’ model) and Yao et al. (2018) (‘ dynamic Word2Vec ’ model) that it is possible to learn the word embeddings across several time periods jointly, enforcing alignment across all of them simultaneously, and positioning all the…”
Advances in Variational Inference (2017, methodology): “[11], [12] derive a structured BBVI algorithm for non-conjugate latent diffusion models.”
Neural Temporality Adaptation for Document Classification: Diachronic Word Embeddings and Domain Adaptation Models (2019, background): “These types of embeddings have been effective in capturing and learning the language usage and semantic shift over time (Kim et al., 2014; Kulkarni et al., 2015; Hamilton et al., 2016; Bamler and Mandt, 2017; Szymanski, 2017; Rudolph and Blei, 2018; Rosenfeld and Erk, 2018; Yao et al., 2018).”
Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors (2018, background): “These dynamic behaviors can arise from different sources such as cultural shift or different political backgrounds (Bamler and Mandt 2017).”
Entity Cloze By Date: What LMs Know About Unseen Entities (2022, background): “Another line of work has looked at diachronic embeddings: (Wijaya and Yeniterzi, 2011; Kim et al., 2014; Hamilton et al., 2016; Bamler and Mandt, 2017), which can model changing meanings of words over time.”
Relational Representation Learning for Dynamic (Knowledge) Graphs: A Survey (2019, background): “or node embeddings as well. Diachronic encoders have been also explored for word embeddings to understand how the meaning (or usage) of words has changed over time (see, e.g., Hamilton et al. (2016); Bamler and Mandt (2017)). 3. In cases where there are dierent time modiers (e.g., OccurredAt, Since,”
Variational Beam Search for Novelty Detection (2021, methodology): “Our ﬁrst experiments demonstrate that a spike & slab prior is more inter-pretable and results in more meaningful word semantics than the diffusion prior of (Bamler and Mandt, 2017).”
GP-VAE: Deep Probabilistic Time Series Imputation (2019, methodology): “Similar structured distributions were also employed by Bamler and Mandt [2], Blei and Laﬀerty [5].”
Room to Glo: A Systematic Comparison of Semantic Change Detection Approaches with Word Embeddings (2019, background): “Word embeddings have been especially popular (e.g., Dubossarsky et al., 2017; Hamilton et al., 2016b), and recently Bamler and Mandt (2017) and Rudolph and Blei (2018) explored dynamic embeddings for semantic change detection by training a joint model over all time periods.”
Analysing Lexical Semantic Change with Contextualised Word Representations (2020, methodology): “More recent methods capture diachronic word usage by learning dynamic word embeddings that vary as a function of time (Bamler and Mandt, 2017; Rosenfeld and Erk, 2018; Rudolph and Blei, 2018).”
Diachronic Embedding for Temporal Knowledge Graph Completion (2019, background): “…embeddings is similar to diachronic word embeddings where a corpus is typically broken temporally into slices (e.g., 20-year chuncks of a 200-year corpus) and embeddings are learned for words in each chunk thus providing word embeddings that are a function of time (see, e.g., [28, 32, 18, 3]).”
Learning Dynamic Author Representations with Temporal Language Models (2019, background): “between word representations -, [28] - a dynamic Skip-Gram model -, [29] - a model with exponential probabilistic evolu-”
Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Change (2019, methodology): “Using different base embeddings, SGNS (Bamler and Mandt, 2017), PPMI (Yao et al., 2018), and Bernoulli embeddings (Rudolph and Blei, 2018), the results show that sharing data is beneﬁcial regardless of the method.”
Deep Learning for Anomaly Detection: A Survey (2019, methodology): “Anomaly detection models leveraging the word2vec embeddings are shown to signiﬁcantly improve performance (Schnabel et al. [2015], Bertero et al. [2017], Bakarov et al. [2018], Bamler and Mandt [2017]).”