Estimating mutual information.
By Alexander Kraskov, H. Stögbauer, Peter Grassberger. 2003
4,216 citations recorded by the source corpus.
We present two classes of improved estimators for mutual information M(X,Y), from samples of random points distributed according to some joint probability density mu(x,y). In contrast to conventional estimators based on binnings, they are based on entropy estimates from k -nearest neighbor distances. This means that they are data efficient (with k=1 we resolve structures down to the smallest possible scales), adaptive (the resolution is higher where data are more numerous), and have minimal bias. Indeed, the bias of the underlying entropy estimates is mainly due to nonuniformity of the density at the smallest resolved scale, giving typically systematic errors which scale as functions of k/N for N points. Numerically, we find that both families become exact for independent distributions, i.e. the estimator M(X,Y) vanishes (up to statistical fluctuations) if mu(x,y)=mu(x)mu(y). This holds for all tested marginal distributions and for all dimensions of x and y. In addition, we give estimators for redundancies between more than two random variables. We compare our algorithms in detail with existing algorithms. Finally, we demonstrate the usefulness of our estimators for assessing the actual independence of components obtained from independent component analysis (ICA), for improving ICA, and for estimating the reliability of blind source separation.
Subfields: Artificial Intelligence, Cognitive Neuroscience, Information Theory, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Bernhard Schölkopf
- Sergey Levine
- Mohit Bansal
- Dacheng Tao
- Quoc V. Le
- Ruslan Salakhutdinov
- Percy Liang
In Cognitive Neuroscience
- Bernhard Schölkopf
- Sergey Levine
- Aapo Hyvärinen
- Matthias Bethge
- Dominik Janzing
- Jan Peters
- Kun Zhang
- Joseph Troy Lizier
In Information Theory
- Bernhard Schölkopf
- Dominik Janzing
- Kun Zhang
- Jan Peters
- Joris M. Mooij
- Peter Kairouz
- Bastian Steudel
- Pramod Viswanath
In Signal Processing
- Bernhard Schölkopf
- Dominik Janzing
- Yoshua Bengio
- Kun Zhang
- Jan Peters
- Joris M. Mooij
- Aapo Hyvärinen
- David Lopez-Paz
Citations in the corpus
- On Mutual Information Maximization for Representation Learning (2019, background): “(1) The fundamental properties of MI are well understood and have been extensively studied (see e.g. Kraskov et al. (2004)).”
- Opening the Black Box of Deep Neural Networks via Information (2017, methodology): “Good overviews on methods for mutual information estimation can be found in Paninski (2003) and Kraskov et al. (2004).”
- Inferring deterministic causal relations (2010, methodology): “For example, in the one-dimensional case, we have used the following estimator [Kraskov et al., 2003]: where the x -values should be ordered ascendingly, i.e., x i ≤ x i +1 , and ψ is the digamma function.”
- Physics-Informed Neural Networks for Cardiac Activation Mapping (2020, methodology): “In our computational experiments, we observe that the predictive posterior distribution p ( T | D ) is generally not Gaussian and we opt to use a non-parametric estimator for the entropy [23, 24].”
- Distinguishing Cause from Effect Using Observational Data: Methods and Benchmarks (2014, methodology): “The first method, 1sp, uses a so-called “1-spacing” estimate (e.g., Kraskov et al., 2004): Ĥ(x) := ψ(N)− ψ(1) + 1 N − 1 N−1∑ i=1 log |xi+1 − xi| , (25) where the x-values should be ordered ascendingly, i.e., xi ≤ xi+1, and ψ is the digamma function (i.e., the logarithmic derivative of the gamma…”
- Excessive Invariance Causes Adversarial Vulnerability (2018, background): “By the chain rule of mutual information (Cover & Thomas, 2006) and information preservation under homeomorphisms (Kraskov et al., 2004), we have”
- On Variational Bounds of Mutual Information (2019, methodology): “While many parametric and non-parametric (Nemenman et al., 2004; Kraskov et al., 2004; Reshef et al., 2011; Gao et al., 2015) techniques have been proposed to address MI estimation and optimization problems, few of them scale up to the dataset size and dimensionality encountered in modern machine…”
- Mutual Information Neural Estimation (2018, methodology): “…vector machines, non-parametric kernel-density estimators; see, Fraser & Swinney, 1986; Darbellay & Vajda, 1999; Suzuki et al., 2008; Kwak & Choi, 2002; Moon et al., 1995; Kraskov et al., 2004), or rely on approximate gaussianity of data distribution (e.g., Edgeworth expansion, Van Hulle, 2005).”
- Formal Limitations on the Measurement of Mutual Information (2018, background): “In the continuous setting, a classical approach to estimating mutual information is based on computing the average log of the distance to the k-th nearest neighbor in samples (Kraskov et al., 2004).”
- Compressing Neural Networks using the Variational Information Bottleneck (2018, methodology): “tic noise assumptions, we instead apply the non-parameteric mutual information estimator from (Kraskov et al., 2004) widely used for similar purposes, and compare against a single sample from VIBNet for fair comparison.”
- Generalized correlation for biomolecular dynamics (2005, methodology): “7), a small systematic underestimation of MI is observed, as discussed in Kraskov et al.(18) Taken together, accuracy can be enhanced by using the larger value of both the (analytical) linear and the (numerical) nonlinear MI.”
- On the information bottleneck theory of deep learning (2018, methodology): “Kraskov estimator We additionally investigated the widely-used nonparametric MI estimator of Kraskov et al (2004).”
- MINE: Mutual Information Neural Estimation (2018, methodology): “Experiment: estimating mutual information between two Gaussians We begin by comparing MINE to the k-means-based non-parametric estimator found in Kraskov et al. (2004). In our experiment, we consider two bivariate Gaussian random variables Xa and Xb with correlation, corr(Xa, Xb) = ρ ∈ [−0.”
- Edgeworth Approximation of Multivariate Differential Entropy (2005, methodology): “Finally, we took the benchmark case of (Kraskov et al., 2004): a 2D gaussian with unit variances along the x- and y-axes and a covariance r for which the theoretical MI = − (1)2 log(1 − r2).”
- Distinguishing Cause from Effect Using Observational Data: Methods and Benchmarks (2014, methodology): “Name Implementation References 1sp based on (24) (Kraskov et al., 2004) 3NN ITE: Shannon kNN k (Kozachenko and Leonenko, 1987) sp1 ITE: Shannon spacing V (Vasicek, 1976) sp2 ITE: Shannon spacing Vb (Van Es, 1992) sp3 ITE: Shannon spacing Vpconst (Ebrahimi et al.”
- On variational lower bounds of mutual information (2018, methodology): “While many parametric and non-parametric (Nemenman et al., 2004; Kraskov et al., 2004) techniques have been proposed to address the MI estimation and maximization problems, few of them scale up to the dataset size and dimensionality encountered in modern machine learning problems.”