Entropy and Inference, Revisited

By Ilya Nemenman, Fariel Shafee, William Bialek. Neural Information Processing Systems, 2001

297 citations recorded by the source corpus.

We study properties of popular near–uniform (Dirichlet) priors for learning undersampled probability distributions on discrete nonmetric spaces and show that they lead to disastrous results. However, an Occam–style phase space argument expands the priors into their infinite mixture and resolves most of the observed problems. This leads to a surprisingly good estimator of entropies of discrete distributions.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/007584cbd3bb0b5e20b1c8b240e6040dc40ab2b4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Neural Decision Forests for Semantic Image Labelling (2014, background): “This is due to an excessive dominance of the prior with respect to the likelihood term when few samples are considered [26].”
Variational Inference for Sparse and Undirected Models (2016, methodology): “One solution to this bias was foreshadowed by methods for estimating entropy and Mutual Information, which used hierarchical priors to integrate over a large range of possible model complexities (Nemenman et al., 2002; Archer et al., 2013).”
A Hilbert Space Embedding for Distributions (2007, methodology): “These methods have by and large a common issue: to compute quantities such as the mutual information, entropy, or Kullback-Leibler divergence, we require sophisticated space partitioning and/or bias correction strategies [10, 9].”
Maximum Likelihood Estimation of Functionals of Discrete Distributions (2014, background): “Nemenman, Shafee, and Bialek [42] argued in an intuitive way why Dirichlet prior is bad for entropy estimation and proposed to use mixtures of Dirichlet priors.”
A dependence maximization view of clustering (2007, methodology): “It is difficult to compute mutual information in high dimensions, however, since sophisticated bias correction methods must be employed (Nemenman et al., 2002).”
Minimax Estimation of Functionals of Discrete Distributions (2014, methodology): “If we trace the progress on entropy estimation in the non-asymptotic regime, we find several thrusts in various communities, including, for example, the Miller–Madow bias-corrected estimator and its variants [21]–[23], the jackknified estimator [24], the shrinkage estimator [25], the Bayes estimator”
Universal Estimation of Information Measures for Analog Sources (2009, background): “In addition, for memoryless sources, different approaches [139, 177, 184, 265] have been designed to overcome difficulties in the under-sampled regime.”
Estimation of Entropy and Mutual Information (2003, methodology): “... to the martingale E(H | x1 ,..., xi); the computation of the asymptotic mean follows by methods almost identical to those used in the proof of theorem 3 (sorting and linearity of expectation, effectively), and the asymptotic variance follows upon combining the formulas of Darling (1953) and Shao”
Gamma-band synchronization in the neocortex: novel analysis methods and their application to sensory and motivational systems (2013, background): “We pointed out that probability vectors with the same entropy can still be assigned different prior probabilities according to the prior NSB distribution, as was noted by (Nemenman et al., 2001) as well.”
Equitability, mutual information, and the maximal information coefficient (2013, background): “Indeed, many approaches for estimating mutual information (or entropy, a closely related quantity) have been proposed; a non-comprehensive list includes [3, 6, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24].”
Inferring interaction partners from protein sequences using mutual information (2018, methodology): “This NSB (Nemenman-Shafee-Bialek) correction is successful in undersampled cases [35, 36].”
Supervised feature selection via dependence estimation (2007, methodology): “Then This property is by contrast with the mutual information, which can require sophisticated bias correction strategies (e.g. Nemenman et al., 2002).”
Causality detection based on information-theoretic approaches in time series analysis (2007, methodology): “[153] has this property.”