Language trees and zipping.

By D. Benedetto, E. Caglioti, Vittorio Loreto. Physical Review Letters, 2001

411 citations recorded by the source corpus.

In this Letter we present a very general method for extracting information from a generic string of characters, e.g., a text, a DNA sequence, or a time series. Based on data-compression techniques, its key point is the computation of a suitable measure of the remoteness of two bodies of knowledge. We present the implementation of the method to linguistic motivated problems, featuring highly accurate results for language recognition, authorship attribution, and language classification.

View Paper: https://www.semanticscholar.org/paper/a8dd3672b6770c2008307cb77eee334234289744

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Computational methods in authorship attribution (2009, unknown): “…character n-grams for authorship attribution of texts in English (Ledger & Merriam 1994; Clement and Sharp 2003; Houvardas and Stamatatos 2006; Stamatatos 2008), Dutch (Hoorn et al. 1999), Russian (Kukushkina 2001), Italian (Benedetto et al. 2002) and Greek (Keselj et al. 2003; Peng et al. 2004).”
Clustering by compression (2003, methodology): “Previously, [1] used an asymmetric measure based on relative entropy, and the full matrix of the pair-wise distances between all 52 languages, to build a language classification tree.”
Colloquium: Criticality and dynamical scaling in living systems (2017, methodology): “…one such method relies on computing estimators of the Kolmogorov complexity (Ming and Vitányi, 2014) of sets of gene-expression time series in diverse microarrays, and computing how the difference between the information content of any two system states (Benedetto et al., 2002) changes over time.”
A survey of modern authorship attribution methods (2009, methodology): “…general text-categorization techniques and evaluate them on authorship attribution together with other tasks such as topic identification, language identification, genre detection, and so on (Benedetto et al., 2002; Teahan & Harper, 2003; Marton et al., 2005; Peng et al., 2004; Zhang & Lee, 2006).”
The similarity metric (2001, methodology): “The paper [2] is predated by our universal similarity metric work and phylogeny tree (hierarchical clustering) experiments [11], [12], [35], but it is the language tree experiment we repeated in the present paper using our own technique with somewhat better results.”
Towards parameter-free data mining (2004, methodology): “A similar experiment is reported in [2].”
Character N-Gram Tokenization for European Language Text Retrieval (2004, methodology): “We adapted a method by Benedetto et al. (2002) to calculate the relatedness of two languages, with the goal of determining whether the accuracy of no-translation retrieval can be predicted given a language pair.”
Comparative study on methods of detecting research fronts using different types of citation (2009, background): “, 2001), Italian literature (Benedetto et al., 2002), and so on.”
Universal Estimation of Information Measures for Analog Sources (2009, background): “In addition to their prominent role in information theory, they have found numerous applications, among others, in probability theory [13, 19], ergodic theory [218], statistics [64, 142], convex analysis and inequalities [69], physics [25, 27, 147, 150], chemistry [79], molecular biology [270], ecol”