On the Kozachenko-Leonenko entropy estimator

By Nicolas G. Fournier, Sylvain Delattre. 2016

61 citations recorded by the source corpus.

We study in details the bias and variance of the entropy estimator proposed by Kozachenko and Leonenko for a large class of densities on $\mathbb{R}^d$. We then use the work of Bickel and Breiman to prove a central limit theorem in dimensions $1$ and $2$. In higher dimensions, we provide a development of the bias in terms of powers of $N^{-2/d}$. This allows us to use a Richardson extrapolation to build, in any dimension, an estimator satisfying a central limit theorem and for which we can give some some explicit (asymptotic) confidence intervals.

Subfields: Statistical and Nonlinear Physics, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/7f1bfb39194acab99a7ea3f661b9f9cf5ba69ef7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistical and Nonlinear Physics

In Statistics and Probability

Citations in the corpus

DINOv2: Learning Robust Visual Features without Supervision (2023, methodology): “The KoLeo regularizer derives from the Kozachenko-Leonenko differential entropy estimator (see Beirlant et al. (1997); Delattre & Fournier (2017)) and encourages a uniform span of the features within a batch.”
Efficient multivariate entropy estimation via $k$-nearest neighbour distances (2016, background): “In the case k = 1, this function appears in Delattre and Fournier (2017), where the authors show that, under certain regularity conditions, lim n→∞ nVar Ĥn − V (f) = Ψ ′(1) + ∫ [0,∞)3 e−s−t T1(r, s, t) st dr ds dt− 1 + 2 log 2.”
The Nearest Neighbor Information Estimator is Adaptively Near Minimax Rate-Optimal (2017, methodology): “Recent works [3, 52, 11] analyzed the performance of the KL estimator under various assumptions on how close the density could be to zero, with no matching lower bound up to logarithmic factors in general.”
Breaking the Bandwidth Barrier: Geometrical Adaptive Entropy Estimation (2016, background): “The O(1/n) convergence rate of the variance is established in [3, 18, 2, 4] under various assumptions.”