Continuous Time Dynamic Topic Models

By Chong Wang, David M. Blei, David E. Heckerman. Conference on Uncertainty in Artificial Intelligence, 2008

528 citations recorded by the source corpus.

In this paper, we develop the continuous time dynamic topic model (cDTM). The cDTM is a dynamic topic model that uses Brownian motion to model the latent topics through a sequential collection of documents, where a "topic" is a pattern of word use that we expect to evolve over the course of the collection. We derive an efficient variational approximate inference algorithm that takes advantage of the sparsity of observations in text, a property that lets us easily handle many time points. In contrast to the cDTM, the original discrete-time dynamic topic model (dDTM) requires that time be discretized. Moreover, the complexity of variational inference for the dDTM grows quickly as time granularity increases, a drawback which limits fine-grained discretization. We demonstrate the cDTM on two news corpora, reporting both predictive perplexity and the novel task of time stamp prediction.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/411277565d6182b40d301df1d2179f5c2b634bb3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey on Bayesian Deep Learning (2016, methodology): “[7, 113] are proposed to incorporate temporal information, and [6] extends LDA by assuming correlations among topics.”
Dynamic Word Embeddings via Skip-Gram Filtering (2017, background): “Probabilistic models that have been extended to latent time series models are ubiquitous (Blei & Lafferty, 2006; Wang et al., 2008; Sahoo et al., 2012; Gultekin & Paisley, 2014; Charlin et al., 2015; Ranganath et al., 2015; Jerfel et al., 2017), but none of them relate to word embeddings.”
Mind the Gap: Assessing Temporal Generalization in Neural Language Models (2021, methodology): “More speciﬁc to the LM literature, prior work proposed ways of designing LMs that efﬁciently adapt their knowledge to continuous streams of new information (Jelinek et al., 1991; Wang et al., 2008; Goyal et al., 2009; Osborne et al., 2014; Yogatama et al., 2014 et al., 2019; Sun et al., 2020).”
Towards Bayesian Deep Learning: A Survey (2016, background): “[5], [56] are proposed to incorporate temporal information and [4] extends LDA by assuming correlations among topics.”
Topic Modeling Using Latent Dirichlet allocation (2021, methodology): “They modeled topics for each group and demonstrated how the topic evolves from one group to another over time [10, 146].”
Learning under Concept Drift: an Overview (2010, background): “The state of the art Latent Dirichlet Allocation model for probabilistic document corpus modeling was recently equipped with a time dimension [18, 149].”
Syntactic Topic Models (2008, background): “Non-conjugate pairs appear in the dynamic topic model (Blei and Lafferty 2006; Wang et al. 2008), correlated topic model (Blei et al.”
Dynamic Language Models for Streaming Text (2014, background): “On the other hand, cautiously using only the most recent data risks overﬁt-ting to short-term trends and missing important time-insensitive effects (Blei and Lafferty, 2006; Wang et al., 2008).”
Learning Dynamic Author Representations with Temporal Language Models (2019, background): “Note the extensions of [21] to a multi-scale temporal version [23] or to a model with continuous-time dependencies [24].”
Variational inference in nonconjugate models (2012, unknown): “6”
TextFlow: Towards Better Understanding of Evolving Topics in Text (2011, unknown): “A dynamic topic model like dynamic LDA focuses on extending LDA to handle dynamic topics [2] [28] [27].”
A temporal model of text periodicities using Gaussian Processes (2013, background): “Extensions that model time make different assumptions, usually regarding smoothing proprieties in (Wang and McCallum, 2006; Blei and Lafferty, 2006; Wang et al., 2008; Hennig et al., 2012).”
Pitfalls of Static Language Modelling (2021, background): “…Transformer (Vaswani et al., 2017) language models are able to generalize well across time in a dynamic streaming setup (Jelinek et al., 1991; Wang et al., 2008; Yogatama et al., 2014; Osborne et al., 2014, inter alia ), which can help measure progress and spur further advances in this…”
Dynamic Word Embeddings (2017, background): “Probabilistic models that have been extended to latent time series models are ubiquitous (Blei & Lafferty, 2006; Wang et al., 2008; Sahoo et al., 2012; Gultekin & Paisley, 2014; Charlin et al., 2015; Ranganath et al., 2015; Jerfel et al., 2017), but none of them relate to word embeddings.”
On-line LDA: Adaptive Topic Models for Mining Text Streams with Applications to Topic Detection and Tracking (2008, methodology): “Most of the related work either processes archives in an off-line fashion (e.g. [16]), post-discretizes the time ([17, 13]), or uses unconjugated priors to multinomial distributions and trained on all the previous data (e.g. [3, 15]).”
SituatedQA: Incorporating Extra-Linguistic Contexts into QA (2021, methodology): “Recent work studying other NLP tasks have shown that this static paradigm misrepresents how models perform in practical settings (Osborne et al., 2014; Wang et al., 2008; Huang and Paul, 2018; Rijhwani and Preotiuc-Pietro, 2020).”