Mutual Information between Discrete and Continuous Data Sets

By Brian C. Ross. PLoS ONE, 2014

821 citations recorded by the source corpus.

Mutual information (MI) is a powerful method for detecting relationships between data sets. There are accurate methods for estimating MI that avoid problems with “binning” when both data sets are discrete or when both data sets are continuous. We present an accurate, non-binning MI estimator for the case of one discrete data set and one continuous data set. This case applies when measuring, for example, the relationship between base sequence and gene expression level, or the effect of a cancer drug on patient survival time. We also show how our method can be adapted to calculate the Jensen–Shannon divergence of two or more data sets.

Subfields: Computer Networks and Communications, Molecular Biology, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/ab99cb06acad1011ae0cc791cb7b35b74577cdfc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Molecular Biology

In Statistics and Probability

Citations in the corpus

Variational Information Maximization for Feature Selection (2016, background): “Only a few examples exist in literature [22] under this setting.”
BioSeq-Analysis2.0: an updated platform for analyzing DNA, RNA and protein sequences at sequence level and residue level based on machine learning approaches (2019, methodology): “0, users can reduce the feature vector di-mension into a user-defined length by using mutual information (58) or chi-square algorithm (59).”
Smallholder maize area and yield mapping at national scales with Google Earth Engine (2019, background): “linear cases and between continuous and categorical variables (Kraskov et al., 2004; Ross, 2014).”
Prototypical Contrastive Learning of Unsupervised Representations (2020, methodology): “We estimate the mutual information between the instance features (or their assigned prototypes) and the ground-truth class labels for all images in ImageNet [30] training set, following the method in [31].”
From explanations to feature selection: assessing SHAP values as feature selection mechanism (2020, methodology): “Thus, we tested SHAP against three common feature selection approaches: Mutual Information [35]–[37], Recursive Feature Elimination [31], and ANOVA [38].”
A statistical framework for neuroimaging data analysis based on mutual information estimated via a gaussian copula (2016, methodology): “EEG data were band-pass ﬁltered (1–30 Hz) and the current source density transformation was applied [Rousselet et al., 2014a].”
Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, methodology): “…the absolute difference between class means for each neuron (Antverg & Belinkov, 2021), the mutual information between each neuron and the labels (Ross, 2014; Voita & Titov, 2020; Pimentel et al., 2020), or the absolute magnitude of the coefficients of a dense probe trained with l 1…”
DIME: An Information-Theoretic Difficulty Measure for AI Datasets (2019, background): “Ross (2014), Gao et al. (2017) and Beknazaryan et al. (2019) proposed approaches for estimating mutual information for the mixture case based on density ratio estimation (e.g.,binning, kNN or kernel methods), which is unsatisfactory for high dimensional data such as image and text.”
Estimating Mutual Information for Discrete-Continuous Mixtures (2017, methodology): “Therefore, our estimator recovers the discrete vs continuous estimator in [42].”
Measuring Causal Specificity (2015, background): “Information theoretic variants have also been developed to deal with continuous variables ðe.g., Reshef et al. 2011; Ross 2014Þ. Applying this framework to a causal relationship allows one to measure how much knowing the value set by an intervention on a causal variable reduces one’s uncertainty…”