Estimation of Entropy and Mutual Information

By Liam Paninski. 2003

1,672 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Control and Systems Engineering, Statistical and Nonlinear Physics, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Statistical and Nonlinear Physics

In Evaluation & Benchmarks

Citations in the corpus

The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (2019, background): “Even though the information-theoretic view provides insights into processes shaping the representations, direct MI estimation from ﬁnite samples for densities on multi-dimensional spaces is challenging (Paninski, 2003).”
Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, background): “As directly maximizing the mutual information is challenging [37, 3, 50], the InfoNCE loss [36] was proposed to maximize a lower bound of the mutual information I(v1; v2).”
What makes for good views for contrastive learning (2020, background): “Furthermore, theoretical and experimental work has highlighted that estimating mutual information in high dimensions is challenging and empirical work has shown that the InfoNCE and other bounds used in practice can be quite loose [43, 55, 51].”
Theoretical evidence for adversarial robustness through randomization: the case of the Exponential family (2019, methodology): “In practice, we do not have access to the real value of the entropy, but we estimate it with classical estimators [33].”
Opening the Black Box of Deep Neural Networks via Information (2017, methodology): “Good overviews on methods for mutual information estimation can be found in Paninski (2003) and Kraskov et al. (2004).”
InfoMax-GAN: Improved Adversarial Image Generation via Information Maximization and Contrastive Learning (2020, background): “Yet, since directly maximizing mutual information is often intractable in practice [26], these works often maximize on the InfoNCE [21] lower bound instead, which involves a contrastive loss minimized through having a critic correctly finding a positive sample in contrast to a set of negative sample”
Verified Uncertainty Calibration (2019, background): “This is analogous to the difficulty of measuring the mutual information between two continuous signals [27].”
On Variational Bounds of Mutual Information (2019, background): “Copyright 2019 by the author(s). samples but not the underlying distributions (Paninski, 2003; McAllester & Stratos, 2018).”
Mutual Information Neural Estimation (2018, background): “Despite being a pivotal quantity across data science, mutual information has historically been difficult to compute (Paninski, 2003).”
A Mutual Information Maximization Perspective of Language Representation Learning (2019, background): “Maximizing mutual information directly is generally intractable when the function f consists of modern encoders such as neural networks (Paninski, 2003), so we need to resort to a lower bound on I(A,B).”
Estimating the unseen: an n/log(n)-sample estimator for entropy and support size, shown optimal via new CLTs (2011, methodology): “In a different direction, Paninski gave a simple though non–constructive proof of the existence of a sublinear sample estimator for additively approximating the entropy to within a constant; the proof is via a direct application of the Stone-Weierstrass theorem to the set of Poisson functions [33, 3”
Inferring gene regulatory networks from time series data using the minimum description length principle (2006, neutral): “Recent progress in estimating information theoretic quantities has led to a number of good estimators in this regard (Beirlant et al ., 1997; Paninski, 2003, 2004; Treves and Panzeri, 1995).”
Predicting the birth of a spoken word (2015, background): “Paninski L (2003) Estimation of entropy and mutual information.”
A CLT and tight lower bounds for estimating entropy (2010, background): “We explicitly exhibit such a family of pairs of distributions D,D′ via a Laguerre polynomial construction that may be of independent interest.”
Justifying Additive Noise Model-Based Causal Discovery via Algorithmic Information Theory (2009, background): “To give an impression on the amount of information between p Y | X and p Y that can be inferred after n -fold sampling, we recall that the mutual information between E and X can be estimated up to an accuracy of O (1 /n ) [12].”
Universal Estimation of Information Measures for Analog Sources (2009, methodology): “Information-theoretic methods have also been used in neuroscience to study the dependence between stimuli and neural response [36, 100, 176, 184, 232] and to classify neurons according to their functions [126, 213].”
Gene Regulatory Network Inference from Single-Cell Data Using Multivariate Information Measures (2016, methodology): “…a number of methods have been developed either for estimating the probability distribution from a set of frequencies, such as the Dirichlet estimator and the shrinkage estimator, or for estimating the entropy directly, as in the Miller-Madow estimator (Hausser & Strimmer 2009, Paninski 2003).”
Estimating mutual information using B-spline functions – an improved similarity measure for analysing gene expression data (2004, result): “Recently, an entropy estimator was suggested [21] and showed in an extensive comparison to other commonly used estimators to be superior.”
minet: A R/Bioconductor Package for Inferring Large Transcriptional Networks Using Mutual Information (2008, background): “The empirical estimator (also called "plug-in", "maximum likelihood" or "naïve", see [23]) is the entropy of the empirical distribution.”