Statistical keyword detection in literary corpora
By Juan P. Herrera, Pedro A. Pury. 2007
75 citations recorded by the source corpus.
Abstract.Understanding the complexity of human language requires an appropriate analysis of the statistical distribution of words in texts. We consider the information retrieval problem of detecting and ranking the relevant words of a text by means of statistical information referring to the spatial use of the words. Shannon's entropy of information is used as a tool for automatic keyword extraction. By using The Origin of Species by Charles Darwin as a representative text sample, we show the performance of our detector and compare it with another proposals in the literature. The random shuffled text receives special attention as a tool for calibrating the ranking indices.
Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Giampaolo Cristadoro
- Eduardo G. Altmann
- Mirko Degli Esposti
- Adilson E. Motter
- Janet B. Pierrehumbert
- Marcelo A. Montemurro
- Damián H. Zanette
In Molecular Biology
- Giampaolo Cristadoro
- Eduardo G. Altmann
- Mirko Degli Esposti
- Janet B. Pierrehumbert
- Marcelo A. Montemurro
- Damián H. Zanette
In Natural Language Processing
- Giampaolo Cristadoro
- Eduardo G. Altmann
- Mirko Degli Esposti
- Janet B. Pierrehumbert
- Marcelo A. Montemurro
- Damián H. Zanette
Citations in the corpus
- On the origin of long-range correlations in texts (2012, background): “In contrast to this group of function words are the most frequent nouns that have large σ τ / (cid:104) τ (cid:105) [19, 21, 34, 35] and large ˆ γ , appearing as outliers at the upper right corner of Fig.”
- Towards the Quantification of the Semantic Information Encoded in Written Language (2009, background): “Additionally, by analyzing the statistics of the distance between consecutive occurrences of the same word in a text, it was shown that many words exhibit a phenomenon of clustering or “self-attraction” [12, 20].”
- Beyond Word Frequency: Bursts, Lulls, and Scaling in the Temporal Distributions of Words (2009, background): “[18] Herrera JP and Pury PA (2008) Statistical keyword de-”