Prediction and Entropy of Printed English

By N. Sloane, A. Wyner. 1951

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

LLaMA: Open and Efficient Foundation Language Models (2023, background): “Language models are probability distributions over sequences of words, tokens or characters (Shannon, 1948, 1951).”
Beyond human-level accuracy: computational challenges in deep learning (2019, background): “For example, word and character LM desired SOTA are near estimated lower bounds on the entropy of English text [31].”
Imitating Interactive Intelligence (2020, methodology): “Imitation learning has been successfully deployed to build agents for self-driving cars (Pomerleau, 1989), robotics and biomimetic motor control (Schaal, 1999), game play (Silver et al., 2016; Vinyals et al., 2019), and language modeling (Shannon, 1951).”
Natural Language Processing (Almost) from Scratch (2011, background): “On the other hand, Sutton and McCallum (2005a) obtain results somewhat below the state-of-the-art using joint decoding for SRL and syntactic parsing.”
Generating Images with Sparse Representations (2021, methodology): “They used human vision as a model, analogous to how Shannon (1951) measured the redundancy of English by having people guess the next character in a text sequence. is trained to sequentially predict the conditional distribution over the next element in the sequence, resulting in a model that…”
Nonlinear time-series analysis revisited. (2015, background): “Since the 1950s, entropy has been a well-established measure of the complexity and predictability of a time series 36 .”
DBpedia spotlight: shedding light on the web of documents (2011, methodology): “Entropy [23] has been commonly used to measure uncertainty in probability distributions.”
The Lindley family of distributions: properties and applications (2017, background): “Two popular entropy measures are the Renyi and Shannon entropies [31],[32].”
The generalized odd log-logistic family of distributions: properties, regression models and applications (2017, background): “Section 4 is devoted to the Rényi and Shannon entropies and order statistics.”
Guess Again (and Again and Again): Measuring Password Strength by Simulating Password-Cracking Algorithms (2012, methodology): “In this method, we calculate for each password in the condition the entropy contributed by the number, content, and type of each character, using Shannon’s formula [35].”
Efficient Large-Scale Distributed Training of Conditional Maximum Entropy Models (2009, methodology): “For the largest experiment, we examined the task of predicting the next character in a sequence of text [19], which has implications for many natural language processing tasks.”
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression (2024, background): “The underlying assumption is that natural language contains redundancy (Shannon, 1951) that may be useful for human understanding but might not be necessary for LLMs.”