Detecting Novel Associations in Large Datasets

By David N. Reshef, Yakir A. Reshef, Hilary K Finucane, S. Grossman, G. McVean, Peter J. Turnbaugh, Eric S. Lander, Michael M. Mitzenmacher, Pardis C Sabeti. 2011

3,212 citations recorded by the source corpus.

A statistical method reveals relationships among variables in complex data sets. Identifying interesting relationships between pairs of variables in large data sets is increasingly important. Here, we present a measure of dependence for two-variable relationships: the maximal information coefficient (MIC). MIC captures a wide range of associations both functional and not, and for functional relationships provides a score that roughly equals the coefficient of determination (R2) of the data relative to the regression function. MIC belongs to a larger class of maximal information-based nonparametric exploration (MINE) statistics for identifying and classifying relationships. We apply MIC and MINE to data sets in global health, gene expression, major-league baseball, and the human gut microbiota and identify known and novel relationships.

Subfields: Artificial Intelligence, Molecular Biology, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Statistical and Nonlinear Physics

Citations in the corpus

Isolating Sources of Disentanglement in VAEs (2018, background): “Efficient estimation of mutual information is an ongoing research topic [23, 24], but we find that the simple estimator (5) is sufficient for the datasets we use.”
Distinguishing Cause from Effect Using Observational Data: Methods and Benchmarks (2014, methodology): “Other dependence measures could be used instead, e.g., the measure proposed by Reshef et al. (2011).”
A low variance consistent test of relative dependency (2014, background): “diverse, and include kernel measures of covariance [5, 19] and correlation [2, 3], distance covariances (which are instances of kernel tests) [18, 14], rankings [7], and space partitioning approaches [6, 12, 9]. For many problems in data analysis, however, the question of whether dependence exists i”
On Variational Bounds of Mutual Information (2019, methodology): “L G ] 1 6 M ay 2 01 9 While many parametric and non-parametric (Nemenman et al., 2004; Kraskov et al., 2004; Reshef et al., 2011; Gao et al., 2015) techniques have been proposed to address MI estimation and optimization problems, few of them scale up to the dataset size and dimensionality…”
Privacy-Preserving Data Exploration in Genome-Wide Association Studies (2013, background): “The analyst explores the dataset [24], choosing the most appropriate statistical tests, discovering the regions of the genome that look interesting, etc. Existing methods for privacy-preserving analysis of genetic data do not support this kind of exploration.”
Comparison of co-expression measures: mutual information, correlation, and model based indices (2012, methodology): “MIC Another mutual information based method is the recently proposed the maximal information coefficient (MIC) [35].”
Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE (2021, background): “Estimation of mutual information (MI) is central to many scientiﬁc investigations and engineering problems [36–38], thus motivating a plethora of practical procedures [39–46].”
Agricultural intensification reduces microbial network complexity and the abundance of keystone taxa in roots (2018, positive): “Co-occurrence patterns in fungal communities were assessed by performing network analysis using the maximal information coefficient (MIC) scores in MINE statistics [80].”
Isolating Sources of Disentanglement in Variational Autoencoders (2018, background): “Efficient estimation of mutual information is an ongoing research topic (Belghazi et al., 2018; Reshef et al., 2011), but we find that the simple estimator (6) is sufficient for the datasets we use.”
On the uniqueness of distance covariance (2012, background): “On recent developments in measuring and testing independence also see Reshef et al. (2011) and Parzen and Mukhopadhyay (submitted for publication).”
A Statistically and Numerically Efficient Independence Test Based on Random Projections and Distance Covariance (2017, background): “Test of independence is a fundamental problem in statistics, with many existing work including the maximal information coefficient (MIC) [1], the copula based measures [2,3], the kernel based criterion [4] and the distance correlation [5,6], which motivated our current work.”
Equitability, mutual information, and the maximal information coefficient (2013, background): “claim in their paper (1) that mutual information does not satisfy the heuristic notion of equitability.”