Dynamic Language Models for Streaming Text

By Dani Yogatama, Chong Wang, Bryan R. Routledge, Noah A. Smith, Eric P. Xing. 2014

35 citations recorded by the source corpus.

We present a probabilistic language model that captures temporal dynamics and conditions on arbitrary non-linguistic context features. These context features serve as important indicators of language changes that are otherwise difficult to capture using text data by itself. We learn our model in an efficient online fashion that is scalable for large, streaming data. With five streaming datasets from two different genres—economics news articles and social media—we evaluate our model on the task of sequential language modeling. Our model consistently outperforms competing models.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Variational Inference: A Review for Statisticians (2016, background): “…al. 2007; Liang, Jordan, and Klein 2009), grammar induction (Kurihara and Sato 2006; Naseem et al. 2010; Cohen and Smith 2010), models of streaming text (Yogatama et al. 2014), topic modeling (Blei, Ng, and Jordan 2003), and hidden Markov models and part-of-speech tagging (Wang and Blunsom 2013).”
Mind the Gap: Assessing Temporal Generalization in Neural Language Models (2021, methodology): “…al., 2010) has highlighted the importance of temporal splits for fairer and more realistic evaluations—and has led to research (Osborne et al., 2014; Yogatama et al., 2014) that addresses language modelling from this streaming perspective (§7)—using temporal splits (or splits beyond random ones)…”
Stochastic Gradient Markov Chain Monte Carlo (2019, methodology): “and complex models to these high-dimensional data. For example, scientiﬁc ﬁelds, such as population genetics (Raj et al., 2014), brain imaging (Andersen et al., 2018) and natural language processing (Yogatama et al., 2014), commonly use a Bayesian approach to data analysis, but the continual growth ”
Pitfalls of Static Language Modelling (2021, background): “Compared to prior work on streaming ( n -gram) language modelling (Yogatama et al., 2014; Osborne et al., 2014), the magnitude of the perplexity numbers is overall much lower in this work—although the exact perplexity values are not directly comparable.”
SituatedQA: Incorporating Extra-Linguistic Contexts into QA (2021, methodology): “Dynamic evaluation Dynamic evaluation has been studied under the language modeling (Os-borne et al., 2014; Yogatama et al., 2014), topic modeling (Wang et al., 2008) and entity linking (Rijhwani and Preotiuc-Pietro, 2020).”
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models (2022, background): “…to train and evaluate such LMs. Previous works have introduced benchmarks comprised of data sources from Twitter feeds (Osborne et al., 2014; Yogatama et al., 2014; Loureiro et al., 2022), recent news articles (Jang et al., 2022), and arXiv papers (Lazaridou et al., 2021) where the temporal…”