Parametric Bayesian Estimation of Differential Entropy and Relative Entropy

By Maya R. Gupta, Santosh Kumar Srivastava. 2010

45 citations recorded by the source corpus.

Given iid samples drawn from a distribution with known parametric form, we propose the minimization of expected Bregman divergence to form Bayesian estimates of differential entropy and relative entropy, and derive such estimators for the uniform, Gaussian, Wishart, and inverse Wishart distributions. Additionally, formulas are given for a log gamma Bregman divergence and the differential entropy and relative entropy for the Wishart and inverse Wishart. The results, as always with Bayesian estimates, depend on the accuracy of the prior parameters, but example simulations show that the performance can be substantially improved compared to maximum likelihood or state-of-the-art nonparametric estimators.

Subfields: Statistical and Nonlinear Physics, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistical and Nonlinear Physics

1. Barnab'as P'oczos
2. Jeff Schneider
3. Liang Xiong
4. Siamak Ravanbakhsh

In Statistics and Probability

1. Jeff Schneider
2. Barnab'as P'oczos
3. Liang Xiong
4. Bharath K. Sriperumbudur
5. Arthur Gretton

Citations in the corpus

• Nonparametric Divergence Estimation with Applications to Machine Learning on Distributions (2011, methodology): “Gupta and Srivastava (2010) developed algorithms for estimating the Shannon entropy and the kl divergence for certain parametric families.”
• On the Estimation of α-Divergences (2011, methodology): “This error originates in the work of Kozachenko and Leonenko (1987) and can also be found in other works.”
• On the Estimation of alpha-Divergences (2011, unknown): “Gupta and Srivastava (2010) developed algorithms for estimating the Shannon entropy and the kl divergence for certain parametric families .”