Zipf’s word frequency law in natural language: A critical review and future directions

By S. Piantadosi. 2014

840 citations recorded by the source corpus.

The frequency distribution of words has been a key object of study in statistical linguistics for the past 70 years. This distribution approximately follows a simple mathematical form known as Zipf’s law. This article first shows that human language has a highly complex, reliable structure in the frequency distribution over and above this classic law, although prior data visualization methods have obscured this fact. A number of empirical phenomena related to word frequencies are then reviewed. These facts are chosen to be informative about the mechanisms giving rise to Zipf’s law and are then used to evaluate many of the theoretical explanations of Zipf’s law in language. No prior account straightforwardly explains all the basic facts or is supported with independent evaluation of its underlying assumptions. To make progress at understanding why language obeys Zipf’s law, studies must seek evidence beyond the law itself, testing assumptions and evaluating novel predictions with new, independent data.

Subfields: Artificial Intelligence, Cultural Studies, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cultural Studies

In Natural Language Processing

Citations in the corpus

The Curious Case of Neural Text Degeneration (2019, methodology): “The Zipfian coefficient s can be used to compare the distribution in a given text to a theoretically perfect exponential curve, where s = 1 (Piantadosi, 2014).”
CM3: A Causal Masked Multimodal Model of the Internet (2022, background): “Traditionally language tokens are said to follow a Zipﬁan distribution (Piantadosi, 2014), while image tokens are strictly uniform (see § A.2).”
Skip-Gram − Zipf + Uniform = Vector Additivity (2017, background): “In every word model that satisﬁes A1, A2, and where p ( w ) = 1 / V | for every w ∈ V , the paraphrase of C = { c 1 , . . . , c m } is As word frequencies are typically much closer to a Zipf distribution (Piantadosi, 2014), the uni-formity assumption of Theorem 2 is not realistic.”
Language Model Evaluation Beyond Perplexity (2021, background): “Since the power-law behavior of the token rank–frequency distribution is known to fall off at higher ranks (Piantadosi, 2014; Moreno-Sánchez et al., 2016), we consider solely the first 10,000 ranks in each sample, including when computing Dp.”
From Word to Sense Embeddings: A Survey on Vector Representations of Meaning (2018, background): “The skewed distribution clearly shows that word types tend to have varying number of senses in a natural text, as also discussed in other studies (Piantadosi, 2014; Bennett, Baldwin, Lau, McCarthy, & Bond, 2016; Pasini & Navigli, 2018).”
Gromov-Wasserstein Alignment of Word Embedding Spaces (2018, background): “If they are not, but words are sorted according to occurrence (as they often are in popular off-the-shelf embedding formats), one can estimate rank-probabilities such as Zipf power laws, which are known to accurately model multiple languages (Piantadosi, 2014).”
First Women, Second Sex: Gender Bias in Wikipedia (2015, background): “Addressing the gender gap from a content perspective may help to improve the quality and value of the content.”
Real-world visual statistics and infants' first-learned object names (2017, background): “Like the distribution of elements in many naturally occurring phenomena [35-37], the frequency distribution of visual objects in these corpora of”
Hyperbolic Attention Networks (2018, background): “This includes a wide variety of natural phenomena in physics [22], biology [25], and even human-made structures such as metabolic-mass relationships [5], social networks [20, 30], and frequencies of words [32, 31, 37].”
Data Distributional Properties Drive Emergent In-Context Learning in Transformers (2022, background): “Natural data also often has the property that the marginal distribution across entities is highly skewed, following a Zipfian (power law) distribution with a long tail of infrequent items (Piantadosi, 2014; Smith et al., 2018; Zipf, 1949).”
Data Distributional Properties Drive Emergent Few-Shot Learning in Transformers (background): “Intriguingly, natural languages are best described by a Zipﬁan distribution with an exponent of approximately 1 (Piantadosi, 2014).”
Twitter Sentiment Analysis (2015, background): “Statistical properties of texts, such as adherence to Zipfs law can also be used (Piantadosi, 2014).”