Beyond Word Frequency: Bursts, Lulls, and Scaling in the Temporal Distributions of Words
By Eduardo G. Altmann, Janet B. Pierrehumbert, Adilson E. Motter. 2009
210 citations recorded by the source corpus.
Background Zipf's discovery that word frequency distributions obey a power law established parallels between biological and physical processes, and language, laying the groundwork for a complex systems perspective on human communication. More recent research has also identified scaling regularities in the dynamics underlying the successive occurrences of events, suggesting the possibility of similar findings for language as well. Methodology/Principal Findings By considering frequent words in USENET discussion groups and in disparate databases where the language has different levels of formality, here we show that the distributions of distances between successive occurrences of the same word display bursty deviations from a Poisson process and are well characterized by a stretched exponential (Weibull) scaling. The extent of this deviation depends strongly on semantic type – a measure of the logicality of each word – and less strongly on frequency. We develop a generative model of this behavior that fully determines the dynamics of word usage. Conclusions/Significance Recurrence patterns of words are well described by a stretched exponential distribution of recurrence times, an empirical scaling that cannot be anticipated from Zipf's law. Because the use of words provides a uniquely precise and powerful lens on human thought and activity, our findings also have implications for other overt manifestations of collective human dynamics.
Subfields: Cultural Studies, Economics and Econometrics, Statistical and Nonlinear Physics, Statistics and Probability, Computational Linguistics & Semantics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Cultural Studies
- James L. McClelland
- Adam Santoro
- Jacob Eisenstein
- Noah A. Smith
- Brendan T. O'Connor
- Josh Tenenbaum
- Janet B. Pierrehumbert
- Paul Röttger
In Economics and Econometrics
- Alexander M. Petersen
- H. Eugene Stanley
- Shlomo Havlin
- Eric P. Xing
- Jacob Eisenstein
- Noah A. Smith
- Brendan T. O'Connor
- James L. McClelland
In Statistical and Nonlinear Physics
- H. Eugene Stanley
- Matjaz Perc
- Andrew Kyle Lampinen
- Shlomo Havlin
- Jacob Eisenstein
- Noah A. Smith
- Eric P. Xing
- Alexander M. Petersen
In Statistics and Probability
- Noah A. Smith
- Eric P. Xing
- Jacob Eisenstein
- Felix Hill
- Brendan T. O'Connor
- James L. McClelland
- Josh Tenenbaum
- Daniel Khashabi
In Computational Linguistics & Semantics
- Eric P. Xing
- Jacob Eisenstein
- Noah A. Smith
- Felix Hill
- James L. McClelland
- Josh Tenenbaum
- Daniel Khashabi
- Andrew Kyle Lampinen
Citations in the corpus
- Data Distributional Properties Drive Emergent Few-Shot Learning in Transformers (background): “…that is not uniform across sentences or documents – rather, it may tend to appear in clusters (likely induced by the topical nature of text; Altmann et al., 2009; Alvarez-Lacalle…”
- Diffusion of Lexical Change in Social Media (2012, background): “s poses a signicant additional challenge, as we are concerned not with the rst appearance of a word, but with the bursts and lulls in its popularity over arXiv:1210.5268v4 [cs.CL] 24 Nov 2014 2 time [22]. In addition, the well known \long-tail" nature of both word counts and city sizes [23] ens”
- Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2021, background): “Changes in the ways a language is used over time are widely attested (Labov, 2011; Altmann et al., 2009; Eisenstein et al., 2014); how these changes will affect NLP systems built from text corpora, and in particular their long-term performance, is not as well understood.”
- Languages cool as they expand: Allometric scaling and the decreasing need for new words (2012, background): “Languages, like biological systems demonstrate economies of scale (b < 1) manifesting from a complex dependency structure that mimics a hierarchical “circulatory system” required by the organization of language [39, 52–56] and the limits of the efficiency of the speakers/writers who exchange the wor”
- Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media (2021, background): “their initial use, often triggered by some exogenous event, followed by a slower decay (Church, 2000; Altmann et al., 2009; Pierrehumbert, 2012).”
- Data Distributional Properties Drive Emergent In-Context Learning in Transformers (2022, background): “That is, a given entity (word, person, object, etc) may have a distribution that is not uniform across time, instead tending to appear in clusters (Altmann et al., 2009; Alvarez-Lacalle et al., 2006; Lambiotte et al., 2013; Neuts, 2007; Sarkar et al., 2005; Serrano et al., 2009).”
- Zipf’s word frequency law in natural language: A critical review and future directions (2014, background): “Recently, Altmann, Pierrehumbert, and Motter (2009) showed that word recurrences on a time scale compatible with semantics (not syntax) follow a stretched exponential distribution, with a certain degree of “ burstiness.”