Estimating Mutual Information for Discrete-Continuous Mixtures

By Weihao Gao, Sreeram Kannan, Sewoong Oh, Pramod Viswanath. 2017

184 citations recorded by the source corpus.

Estimating mutual information from observed samples is a basic primitive, useful in several machine learning tasks including correlation mining, information bottleneck clustering, learning a Chow-Liu tree, and conditional independence testing in (causal) graphical models. While mutual information is a well-defined quantity in general probability spaces, existing estimators can only handle two special cases of purely discrete or purely continuous pairs of random variables. The main challenge is that these methods first estimate the (differential) entropies of X, Y and the pair (X;Y) and add them up with appropriate signs to get an estimate of the mutual information. These 3H-estimators cannot be applied in general mixture spaces, where entropy is not well-defined. In this paper, we design a novel estimator for mutual information of discrete-continuous mixtures. We prove that the proposed estimator is consistent. We provide numerical experiments suggesting superiority of the proposed estimator compared to other heuristics of adding small continuous noise to all the samples and applying standard estimators tailored for purely continuous variables, and quantizing the samples and applying standard estimators tailored for purely discrete variables. This significantly widens the applicability of mutual information estimation in real-world applications, where some variables are discrete, some continuous, and others are a mixture between continuous and discrete components.

Subfields: Artificial Intelligence, Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

Citations in the corpus

Obfuscation via Information Density Estimation (2019, background): “…inherently connected to mutual information estimation (since the expected value of information density is equal to the mutual information) which is known to be challenging (Valiant and Valiant, 2011; Wu and Yang, 2016; Gao et al., 2017) unless an adequate parametric model is assumed (Vapnik, 2013).”
Data-Efficient Mutual Information Neural Estimator (2019, methodology): “For MI-KSG, we use off-the-shelf implementation [15] with default number of nearest neighbors k=3.”
The Nearest Neighbor Information Estimator is Adaptively Near Minimax Rate-Optimal (2017, background): “Moreover, a slightly revised version of KSG estimator is proved to be consistent even if the underlying distribution is not purely continuous nor purely discrete [19], but the optimality properties are not yet well understood.”
GILBO: One Metric to Measure Them All (2018, methodology): “…VAEs and very expensive to estimate for GANs) (Wu et al., 2017)3, an independent Wasserstein critic (Danihelka et al., 2017), or the common INCEPTION (Gao et al., 2017) and FID (Heusel et al., 2017) scores which attempt to measure how well the generated samples match the observed true data samples.”
A Theory of Usable Information Under Computational Constraints (2020, methodology): “Although non-parametric estimators of mutual information exist (Kraskov et al., 2004; Darbellay & Vajda, 1999; Gao et al., 2017), these estimators do not scale to high dimensions.”
Alleviating Semantics Distortion in Unsupervised Low-Level Image-to-Image Translation via Structure Consistency Constraint (2022, background): “Subsequent works improved the performance in more complicated cases such as discrete-continuous mixtures [11, 41], segmentation [66, 70, 72] and continue learning [61, 62].”
An Information-Theoretic Analysis of Deep Latent-Variable Models (2017, background): “Computing mutual information exactly in high dimensions is problematic (Panin-ski, 2003; Gao et al., 2017), so we turn to recently developed tools in variational inference to approximate it.”
Scalable Mutual Information Estimation Using Dependence Graphs (2018, methodology): “In general in Figures 2 and 3, EDGE has better convergence rate than EKDE and KSG estimators.”
DIME: An Information-Theoretic Difficulty Measure for AI Datasets (2019, background): “Ross (2014), Gao et al. (2017) and Beknazaryan et al. (2019) proposed approaches for estimating mutual information for the mixture case based on density ratio estimation (e.g.,binning, kNN or kernel methods), which is unsatisfactory for high dimensional data such as image and text.”