An Estimate of an Upper Bound for the Entropy of English

By Peter F. Brown, Stephen Della Pietra, V. D. Pietra, Jennifer C. Lai, Robert L. Mercer. 1992

Subfields: Computational Linguistics & Semantics, Information Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Linguistics & Semantics

In Information Theory

Citations in the corpus

Measuring Data (2022, methodology): “It was subsequently leveraged to evaluate the performance of language models (Brown et al., 1992; Bengio et al., 2000; Melis et al., 2017) as well as datasets, for tasks such as perplexity sampling (De la Rosa et al., 2022) and data selection (Toral et al., 2015).”
Evaluating the Search Phase of Neural Architecture Search (2019, methodology): “The quality of a candidate cell is then evaluated using the standard perplexity metric [4].”
Assessing The Factual Accuracy of Generated Text (2019, methodology): “generation tasks. For evaluation, these models generally rely on metrics like ROUGE (RecallOriented Understudy for Gisting Evaluation) [13], BLEU (Bilingual Evaluation Understudy) [23] and perplexity [3] that measure locally constrained n-gram overlap. In this paper, we propose an automatic metric f”
What Context Features Can Transformer Language Models Use? (2021, methodology): “A note on evaluation As in past work on evaluating language models (Brown et al., 1992), our evaluation of relative predictive information ultimately bottoms out in a conditional entropy (log-perplexity).”
Linguistically-Informed Specificity and Semantic Plausibility for Dialogue Generation (2019, background): “Perplexity per word (PPW) Perplexity is the exponentiation of the entropy, which estimates the expected number of bits required to encode the sentence (Brown et al., 1992; Goodman, 2001).”
Evaluating Style Transfer for Text (2019, background): “Low perplexity signi-ﬁes less uncertainty over which words can be used to continue a sequence, quantifying the ability of a language model to predict gold or reference texts (Brown et al., 1992; Young et al., 2006).”
A survey of smoothing techniques for ME models (2000, methodology): “For example, one simple smoothing technique is to linearly interpolate the ML estimate of the -gram proba- bility with an estimate of the -gram probability [14] , [15] (4) The lower-order estimate can be defined analogously, and the recursion can end with a unigram or uniform distribution.”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “In information theory, perplexity (PPL) is a measurement of how well a probability distribution or probability model predicts a sample compared with the ground-truth [14].”
An Attentional Neural Conversation Model with Improved Specificity (2016, background): “The second measure is perplexity (Brown et al., 1992), which measures the likelihood of generating a word given observations.”
Paying Attention to Attention: Highlighting Influential Samples in Sequential Analysis (2018, methodology): “To determine when the turn weights are uniform, we use perplexity (Brown et al., 1992) (see more details in subsection 3.1).”
Learning to Rank with Selection Bias in Personal Search (2016, background): “Perplexity measures how well a distribution predicts samples and is often used to evaluate or compare language models [3].”
CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation (2022, unknown): “As for the design of pattern functions, we ﬁx the base model and replace our input and output patterns ( f & g ) with those of PPL-GPT (Radford et al., 2018) and BARTScore (Yuan et al., 2021).”
Revisiting the Predictability of Language: Response Completion in Social Media (2012, methodology): “More recently, Brown et. al (1992) computed an upper bound for the entropy of English with a trigram model, using the Brown corpus.”