Word frequency distributions

By Geoffrey Sampson. Computational Linguistics, 2002

433 citations recorded by the source corpus.

Subfields: Computational Theory and Mathematics

View Paper: https://www.semanticscholar.org/paper/4813c27159d792aa330697cd3db510b070119dd1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

Citations in the corpus

Frequency in lexical processing (2016, background): “Furthermore, as pointed out by Nusbaum (1985), such a Markov model not only correctly predicts shorter words to be more frequent, it also correctly predicts that more frequent words will have denser similarity neighbourhoods (see also Baayen, 2001).”
Changing places: A cross-language perspective on frequency and family size in Dutch and Hebrew (2005, methodology): “With respect to the frequency counts, Baayen, Moscoso del Prado, Schreuder, and Wurm (2003) show that under the lognormal model (Baayen, 2001; Carroll, 1967) a change in corpus size amounts to a change in the intercept for the regression Table 4 Pairwise Pearson correlation coefficients between the…”
Beyond Word Frequency: Bursts, Lulls, and Scaling in the Temporal Distributions of Words (2009, background): “Taken together, they suggest the existence of a dynamic counterpart to the universal scaling laws in magnitude and frequency distributions [7, 8, 9, 10, 11].”
The Statistics of Word Cooccur-rences: Word Pairs and Collocations (2004, neutral): “statistical model can then be formulated that allows us to predict to what extent the observed cooccurrences may be merely due to chance (i.e. the particular choice of source corpus), or whether they provide sufficient evidence for a “true” association between the words (that holds for the entire la”
Detecting Fake Content with Relative Entropy Scoring (2008, background): “On the other hand, natural texts and sentences have many simple statistical properties that are not matched by typical word salads, such as the average sentence length, the type/token ratio, the distribution of grammatical words, etc [1].”
The WaCky wide web: a collection of very large linguistically processed web-crawled corpora (2009, background): “Connected text is known to reliably contain a high proportion of function words (Baayen 2001), therefore documents not meeting certain minimal parameters—10 types and 30 tokens per page, with function words accounting for at least a quarter of all words— are discarded.”
Know thy Corpus! Robust Methods for Digital Curation of Web corpora (2020, background): “also a more in-depth discussion by Harald Baayen in Chapter 5 of (Baayen, 2001).”
Introduction to the Special Issue on the Web as Corpus (2003, background): “Baayen (2001) presents sophisticated mathematical models for word frequency distributions, and it is likely that his mixture models have potential for modeling sublanguage mixtures.”
How Many Words Do We Know? Practical Estimates of Vocabulary Size Dependent on Word Definition, the Degree of Language Input and the Participant’s Age (2016, methodology): “This roughly corresponds to a frequency of 1 per 10 billion words and decreases the number of unigrams drastically, given that about half of the unigrams observed in corpus have a frequency of 1 (Baayen, 2001). A similar threshold was used by Brants and Franz (2006).”
DynaSent: A Dynamic Benchmark for Sentiment Analysis (2020, background): “(These measures are intended to control for the known correlation between token counts and vocabulary sizes; Baayen 2001.)”
Text Categorization with Support Vector Machines. How to Represent Texts in Input Space? (2002, background): “See e.g. Chitashvili and Baayen, 1993 for a summary.”
Introducing, evaluating ukWaC, a very large web-derived corpus of English (2008, background): “Connected text is known to reliably contain a high proportion of function words (Baayen, 2001), therefore documents not meeting certain minimal parameters – ten types and thirty tokens per page, with function words accounting for at least a quarter of all words – were discarded.”