A Mathematical Theory of Communication

By J. Shin, Sang Joon Kim. 2006

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Theory

Citations in the corpus

LLaMA: Open and Efficient Foundation Language Models (2023, background): “Language models are probability distributions over sequences of words, tokens or characters (Shannon, 1948, 1951).”
Ethical and social risks of harm from Language Models (2021, methodology): “This research group built on prior work in statistics by Claude Shannon (Shannon, 1948) and Andrey Markov (Markov, 1913).”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “These include the Shannon Entropy (Shannon and Weaver, 1963) over unigrams and bigrams (H1, H2), the mean segmented type token ratio over segment lengths of 100 (MSTTR, Johnson, 1944), the ratio of distinct n-grams over the total number of n-grams (Distinct1,2), and the count of n-grams that only…”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “Language modeling refers to the task of modeling the probability of a sequence of tokens in a text (Shannon, 1948), where a token is a unit of text (e.g. word, subword, character or byte, etc., as discussed by Mielke et al., 2021).”
Understanding Dataset Difficulty with V-Usable Information (2021, positive): “Even if X were to be encrypted, the information X contains about Y would not be removed; in other words, the Shannon mutual information would be unchanged (Shannon, 1948).”
Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters (2018, methodology): “According to Shannon’s source coding theorem (Shannon, 1948), w∗ can be coded with no less than NH[p′] nats, which is asymptotically achieved by Huffman coding, like in Han et al.”
Plug and Play Language Models: A Simple Approach to Controlled Text Generation (2019, background): “…al. (2016; 2017); Chen et al. (2018) explored the idea of using a small neural network to steer an LM. Yu et al. (2016), and more recently Yu et al. (2019); Yee et al. (2019); Ng et al. (2019), leveraged the Shannon Noisy Channel Theory (Shannon, 1948) for improving sequence-to-sequence modeling.”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, background): “, 2022) (for factuality of summarization); 4) diversity/fluency/naturalness metrics such as perplexity, Mean Segmented Type Token Ratio (MSSTR) (Johnson, 1944), Shannon entropy over unigrams and bigrams (Shannon, 1948), the ratio of distinct n-grams over the total number of n-grams (Distinct-1, Dist”
A note on the evaluation of generative models (2015, methodology): “The negative of this log-likelihood is equivalent to the average number of bits (assuming base-2 logarithm) required to losslessly compress the discrete data with an entropy coding scheme optimized for Q (Shannon, 2001).”
From Frequency to Meaning: Vector Space Models of Semantics (2010, background): “In information theory, a surprising event has higher information content than an expected event (Shannon, 1948).”
End-to-end Optimized Image Compression (2016, background): “Data compression is a fundamental and well-studied problem in engineering, and is commonly formulated with the goal of designing codes for a given discrete data ensemble with minimal entropy (Shannon, 1948).”
The security of machine learning (2010, background): “This suggests an open problem: is it possible to develop a “theory of information for security” that can measure the value of leaked information? An answer to this question is likely to build on Shannon’s classical theory of information (Shannon 1948) as well as computationally-based variants of it ”
Differentially Private Password Frequency Lists (2016, neutral): “In this model, guessing-entropy [34], [26]”
Structural Synaptic Plasticity Has High Memory Capacity and Can Explain Graded Amnesia, Catastrophic Forgetting, and the Spacing Effect (2014, background): “It is a well-known result of information theory [69] that optimally coding an entity taken at random from a set of different entities takes ld bits of information [69] (where ld : ~ log2).”
Few-Shot Continual Active Learning by a Robot (2022, methodology): “Further, to choose most informative samples from an unlabeled set, we use a combination of predictive entropy [21, 18] and viewpoint consistency metrics [14, 16] on the GMM representation of the previously learned classes.”
Fast Lossless Neural Compression with Integer-Only Discrete Flows (2022, background): “Shannon’s source coding theorem (Shannon, 1948) states that the average code length needed to encode data is lower bounded by entropy of its distribution:”