Equitability, mutual information, and the maximal information coefficient

By Justin B. Kinney, Gurinder S. Atwal. 2013

696 citations recorded by the source corpus.

Significance Attention has recently focused on a basic yet unresolved problem in statistics: How can one quantify the strength of a statistical association between two variables without bias for relationships of a specific form? Here we propose a way of mathematically formalizing this “equitability” criterion, using core concepts from information theory. This criterion is naturally satisfied by a fundamental information-theoretic measure of dependence called “mutual information.” By contrast, a recently introduced dependence measure called the “maximal information coefficient” is seen to violate equitability. We conclude that estimating mutual information provides a natural and practical method for equitably quantifying associations in large datasets. How should one quantify the strength of association between two random variables without bias for relationships of a specific form? Despite its conceptual simplicity, this notion of statistical “equitability” has yet to receive a definitive mathematical formalization. Here we argue that equitability is properly formalized by a self-consistency condition closely related to Data Processing Inequality. Mutual information, a fundamental quantity in information theory, is shown to satisfy this equitability criterion. These findings are at odds with the recent work of Reshef et al. [Reshef DN, et al. (2011) Science 334(6062):1518–1524], which proposed an alternative definition of equitability and introduced a new statistic, the “maximal information coefficient” (MIC), said to satisfy equitability in contradistinction to mutual information. These conclusions, however, were supported only with limited simulation evidence, not with mathematical arguments. Upon revisiting these claims, we prove that the mathematical definition of equitability proposed by Reshef et al. cannot be satisfied by any (nontrivial) dependence measure. We also identify artifacts in the reported simulation evidence. When these artifacts are removed, estimates of mutual information are found to be more equitable than estimates of MIC. Mutual information is also observed to have consistently higher statistical power than MIC. We conclude that estimating mutual information provides a natural (and often practical) way to equitably quantify statistical associations in large datasets.

Subfields: Economics and Econometrics, Experimental and Cognitive Psychology, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Experimental and Cognitive Psychology

In Statistical and Nonlinear Physics

Citations in the corpus

Mutual Information Neural Estimation (2018, background): “This important property, that guarantees the quantiﬁcation dependence without bias for the relationship, is called equitability (Kinney & Atwal, 2014).”
CausalVAE: Disentangled Representation Learning via Neural Structural Causal Models (2020, methodology): “Table 1 records the mutual information (MIC/TIC) between the learned representation and the ground truth concept labels of all compared methods.”
A statistical framework for neuroimaging data analysis based on mutual information estimated via a gaussian copula (2016, background): “It can be viewed as a statistical test against a null hypothesis that two variables are statistically independent, but in addition its effect size (measured in bits) has a number of useful properties and interpretations [Kinney and Atwal, 2014).”
Data Based Identification and Prediction of Nonlinear and Complex Dynamical Systems (2016, background): “There were claims [342, 343] that the MIC does not outperform MI, suggesting that the mutual information may be a more natural and practical way to quantify statistical associations.”
Distribution-Free Consistent Independence Tests via Center-Outward Ranks and Signs (2019, background): “…the sample space (Heller, Heller, and Gorfine 2013; Heller et al. 2016; Ma and Mao 2019; Zhang 2019), (ii) mutual information (Kraskov, Stögbauer, and Grassberger 2004; Kinney and Atwal 2014; Berrett and Samworth 2019), and (iii) the maximal information coefficient (Reshef et al. 2011, 2016, 2018).”
Lineage tracing on transcriptional landscapes links state to fate during differentiation (2018, methodology): “This reasoning reflects a formal result knownas data-processing inequality (20) (materials and methods, section 10.”
High-dimensional consistent independence testing with maxima of rank correlations (2018, background): “We remark that invariance under invertible (and not just monotonic) transformations was considered in work on self-equitable measures of dependence (Kinney and Atwal, 2014).”
A Robust-Equitable Measure for Feature Ranking and Selection (2017, methodology): “The proof follows from Theorems S3 and S4 of Kinney and Atwal (2014), since g(x) = |x− 1|α is convex when α ≥ 1.”
Toward enhanced understanding and projections of climate extremes using physics-guided data mining techniques (2014, background): “Developments in correlative analysis (Reshef et al., 2011; Khan et al., 2007; Kinney and Atwal, 2014), extended to handle correlated data at multiple spatial and temporal scales, may help quantify conceptual understanding and possibly even discover new dependencies (Khan et al., 2006).”
Demystifying fixed k-nearest neighbor information estimators (2016, methodology): “As a byproduct of our investigations, we obtain nearly tight rates of convergence of the `2 error of the well known fixed k nearest neighbor estimator of differential entropy by Kozachenko and Leonenko.”
Gene Regulatory Network Inference from Single-Cell Data Using Multivariate Information Measures (2016, background): “Information theoretical measures, in particular, are particularly parsimonious in the assumptions that they make (Kinney et al. 2014) compared to e.g. ODE-based regression approaches or simple graphical models.”
Learning Speaker Representations with Mutual Information (2018, positive): “As opposed to other metrics, such as correlation, MI can capture complex non-linear relationships between random variables [19].”