The variation of Zipf’s law in human language

By R. Ferrer i Cancho. 2005

126 citations recorded by the source corpus.

Abstract.Words in humans follow the so-called Zipf’s law. More precisely, the word frequency spectrum follows a power function, whose typical exponent is β≈2, but significant variations are found. We hypothesize that the full range of variation reflects our ability to balance the goal of communication, i.e. maximizing the information transfer and the cost of communication, imposed by the limitations of the human brain. We show that the higher the importance of satisfying the goal of communication, the higher the exponent. Here, assuming that words are used according to their meaning we explain why variation in β should be limited to a particular domain. From the one hand, we explain a non-trivial lower bound at about β=1.6 for communication systems neglecting the goal of the communication. From the other hand, we find a sudden divergence of β if a certain critical balance is crossed. At the same time a sharp transition to maximum information transfer and unfortunately, maximum communication cost, is found. Consistently with the upper bound of real exponents, the maximum finite value predicted is about β=2.4. It is convenient for human language not to cross the transition and remain in a domain where maximum information transfer is high but at a reasonable cost. Therefore, only a particular range of exponents should be found in human speakers. The exponent β contains information about the balance between cost and communicative efficiency.

Subfields: Cultural Studies, Molecular Biology, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/d056158720cb6ca96ee33eb080d91dcb9a25b30b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cultural Studies

In Molecular Biology

In Statistical and Nonlinear Physics

Citations in the corpus

Languages cool as they expand: Allometric scaling and the decreasing need for new words (2012, background): “The Zipf law has been found to hold for a large dataset of English text [12], but interestingly deviates from the schizophrenic lexicon [13].”