Entropy Inference and the James-Stein Estimator, with Application to Nonlinear Gene Association Networks

By Jean Hausser, Korbinian Strimmer. 2008

416 citations recorded by the source corpus.

We present a procedure for effective estimation of entropy and mutual information from small-sample data, and apply it to the problem of inferring high-dimensional gene association networks. Specifically, we develop a James-Stein-type shrinkage estimator, resulting in a procedure that is highly efficient statistically as well as computationally. Despite its simplicity, we show that it outperforms eight other entropy estimation procedures across a diverse range of sampling scenarios and data-generating models, even in cases of severe undersampling. We illustrate the approach by analyzing E. coli gene expression data and computing an entropy-based gene-association network from gene expression data. A computer program is available that implements the proposed shrinkage estimator.

Subfields: Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

Citations in the corpus

Kernel Mean Shrinkage Estimators (2014, background): “5), Bock (1975), and Hudson (1978). Although Stein phenomenon has been extensively studied in the statistics community, it has not received much attention in the machine learning community, see e.g., Hausser and Strimmer (2009). The James-Stein estimator is a special case of a larger class of estima”
Bayesian calibration of building energy models with large datasets (2017, neutral): “…. . . . . . . . . . . . 116 xi List of Tables 1.1 Error criteria for model to be deemed calibrated . . . . . . . . . . . . . . . . . . 14 2.1 Common choices for a Dirichlet prior that can be used to smooth the sampled data (Equation 2.15), taken from Hausser and Strimmer (2009). . . . . . . . . . .”
Inferring Correlation Networks from Genomic Survey Data (2012, methodology): “Sample entropies were computed according to the method descried by Chao and Shen [25], as implemented in the R ‘entropy’ package [26].”
stm: An R Package for Structural Topic Models (2019, methodology): “(14)In practice when using FREX for labeling we regularize our estimate of the topic-specific distribution over words using a James-Stein shrinkage estimator (Hausser and Strimmer 2009).”
Automatic Targetless Extrinsic Calibration of a 3D Lidar and Camera by Maximizing Mutual Information (2012, methodology): “We chose MI because of ongoing active research in fast and robust MI estimation techniques, such as James-Stein-type shrinkage estimators (Hausser and Strimmer 2009), which have the potential to be directly employed in the proposed framework, though are not currently.”
Minimax Estimation of Functionals of Discrete Distributions (2014, methodology): “If we trace the progress on entropy estimation in the non-asymptotic regime, we find several thrusts in various communities, including, for example, the Miller–Madow bias-corrected estimator and its variants [21]–[23], the jackknified estimator [24], the shrinkage estimator [25], the Bayes estimator”
Maximum Likelihood Estimation of Functionals of Discrete Distributions (2014, methodology): “Among them we mention the Miller–Madow bias-corrected estimator and its variants [29]–[31], the jackknife estimator [32], the shrinkage estimator [33], the coverage adjusted estimator [34], the Best Upper Bound (BUB) estimator [14], the B-Splines estimator [35], and [36], [37] etc.”
Entropy and the species accumulation curve: a novel entropy estimator via discovery rates of new species (2013, methodology): “Hausser and Strimmer (2009) derived a James-Stein type of entropy estimator, but they also require S to be known.”
Equitability, mutual information, and the maximal information coefficient (2013, background): “m sparse data is a nontrivial problem [7, 11]. Indeed, many approaches for estimating mutual information (or entropy, a closely related quantity) have been proposed; a non-comprehensive list includes [3, 6, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24]. While comparisons of various estimation ”
Gene Regulatory Network Inference from Single-Cell Data Using Multivariate Information Measures (2016, methodology): “Like other studies (Simoes & Emmert-Streib 2011, Olsen et al. 2009, Hausser et al. 2009), we find that the methods chosen to discretize data and estimate entropies and probability distributions affect algorithm performance considerably (Fig.”
Aitchison’s Compositional Data Analysis 40 Years on: A Reappraisal (2022, background): “The shrinkage intensity can be determined in a data-driven way by minimising the mean-squared error with respect to the unknown multinomial parameters [55].”