Zipf’s Law Arises Naturally When There Are Underlying, Unobserved Variables

By Laurence Aitchison, Nicola Corradi, Peter E. Latham. 2016

85 citations recorded by the source corpus.

Zipf’s law, which states that the probability of an observation is inversely proportional to its rank, has been observed in many domains. While there are models that explain Zipf’s law in each of them, those explanations are typically domain specific. Recently, methods from statistical physics were used to show that a fairly broad class of models does provide a general explanation of Zipf’s law. This explanation rests on the observation that real world data is often generated from underlying causes, known as latent variables. Those latent variables mix together multiple models that do not obey Zipf’s law, giving a model that does. Here we extend that work both theoretically and empirically. Theoretically, we provide a far simpler and more intuitive explanation of Zipf’s law, which at the same time considerably extends the class of models to which this explanation can apply. Furthermore, we also give methods for verifying whether this explanation applies to a particular dataset. Empirically, these advances allowed us extend this explanation to important classes of data, including word frequencies (the first domain in which Zipf’s law was discovered), data with variable sequence length, and multi-neuron spiking activity.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Molecular Biology, Information Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Molecular Biology

In Information Theory

Citations in the corpus

Measuring Data (2022, background): “Recent work has also demonstrated that Zipf’s Law arises naturally across domains with underlying, unobserved variables (Aitchison et al., 2016), suggesting it may be extended to datasets in different domains.”
Colloquium: Criticality and dynamical scaling in living systems (2017, background): “…signatures of criticality could reflect an effective averaging over un-observed variables (such as common external inputs in the case of retinal populations), lacking thus any relationship with possible functional advantages (Aitchison et al., 2016; Nonnenmacher et al., 2017) (see Appendix B).”
Detecting Out-of-Distribution Inputs to Deep Generative Models Using a Test for Typicality (2019, background): “Aitchison et al. [1] showed that power-law data can give rise to a dispersed distribution of likelihoods, and thus examining connections between long-tailed data and typicality might explain this phenomenon.”
A high-bias, low-variance introduction to Machine Learning for physicists (2018, unknown): “Overlooking this can sometimes lead to misleading conclusions, and it is important to be cognizant of these implicit assumptions (Aitchison et al., 2016; Schwab et al., 2014).”