Exponential Reservoir Sampling for Streaming Language Models

By Miles Osborne, Ashwin Lall, Benjamin Van Durme. 2014

30 citations recorded by the source corpus.

We show how rapidly changing textual streams such as Twitter can be modelled in fixed space. Our approach is based upon a randomised algorithm called Exponential Reservoir Sampling, unexplored by this community until now. Using language models over Twitter and Newswire as a testbed, our experimental results based on perplexity support the intuition that recently observed data generally outweighs that seen in the past, but that at times, the past can have valuable signals enabling better modelling of the present.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019, background): “2014a], by leveraging recent seen content to maintain a uniform sampling [Osborne et al. 2014], or by splitting the queries and run multiple in parallel [Sampson et al.”
Deep Reinforcement Learning from Self-Play in Imperfect-Information Games (2016, methodology): “MSL was updated with exponentiallyaveraged reservoir sampling (Osborne et al., 2014), replacing entries in MSL with minimum probability 0.”
Mind the Gap: Assessing Temporal Generalization in Neural Language Models (2021, methodology): “More speciﬁc to the LM literature, prior work proposed ways of designing LMs that efﬁciently adapt their knowledge to continuous streams of new information (Jelinek et al., 1991; Wang et al., 2008; Goyal et al., 2009; Osborne et al., 2014; Yogatama et al., 2014 et al., 2019; Sun et al., 2020).”
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models (2022, background): “…widely-available resources to train and evaluate such LMs. Previous works have introduced benchmarks comprised of data sources from Twitter feeds (Osborne et al., 2014; Yogatama et al., 2014; Loureiro et al., 2022), recent news articles (Jang et al., 2022), and arXiv papers (Lazaridou et al.,…”
One Pass ImageNet (2021, background): “When the loss function is the (2)Similar ideas on encouraging recent examples in reservoir sampling can be found in [13, 14].”
Large Scale Distributed Semi-Supervised Learning Using Streaming Approximation (2015, unknown): “Preliminary: Manku and Motwani [21] presented an algorithm for computing frequency counts exceeding a user-speciﬁed threshold over data streams, and others have applied this algorithm to handle large amounts of data in NLP problems [15, 14, 35, 24].”
Pitfalls of Static Language Modelling (2021, background): “Compared to prior work on streaming ( n -gram) language modelling (Yogatama et al., 2014; Osborne et al., 2014), the magnitude of the perplexity numbers is overall much lower in this work—although the exact perplexity values are not directly comparable.”
SituatedQA: Incorporating Extra-Linguistic Contexts into QA (2021, methodology): “Recent work studying other NLP tasks have shown that this static paradigm misrepresents how models perform in practical settings (Osborne et al., 2014; Wang et al., 2008; Huang and Paul, 2018; Rijhwani and Preotiuc-Pietro, 2020).”