A class of Rényi information estimators for multidimensional densities

By Nikolai N. Leonenko, Luc Pronzato, V. Savani. 2008

282 citations recorded by the source corpus.

A class of estimators of the Renyi and Tsallis entropies of an unknown distribution f in R^m is presented. These estimators are based on the k-th nearest-neighbor distances computed from a sample of N i.i.d. vectors with distribution f. We show that entropies of any order q, including Shannon's entropy, can be estimated consistently with minimal assumptions on f. Moreover, we show that it is straightforward to extend the nearest-neighbor method to estimate the statistical distance between two distributions using one i.i.d. sample from each.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Statistics and Probability

Citations in the corpus

Nonparametric Divergence Estimation with Applications to Machine Learning on Distributions (2011, methodology): “To estimate the L 2 divergence, we will borrow some ideas from Leonenko et al. (2008) and Goria et al. (2005), who considered Shannon and Rényi-α entropy estimation from a single sample.”
On the Estimation of alpha-Divergences (2011, background): “We also have the following (Leonenko et al., 2008a).”
Empirical estimation of entropy functionals with confidence (2010, methodology): “For further applications, see, for example, Leonenko etal [26].”
A closed-form expression for the Sharma–Mittal entropy of exponential families (2011, methodology): “[11] proposed a method to estimate entropies using the k-nearest neighbor graph (k-NN) of an independently and identically distributed sample set x1, .”
On the Estimation of α-Divergences (2011, methodology): “We also have the following (Leonenko et al., 2008a).”
Estimation of Renyi Entropy and Mutual Information Based on Generalized Nearest-Neighbor Graphs (2010, methodology): “A variant of these estimators, where each sample point is connected to its k -th nearest neighbor only, were recently studied by Goria et al. (2005) for Shannon entropy estimation ( i.e. the special case α = 1 ) and Leonenko et al. (2008) for R´enyi α -entropy estimation.”
Correction: A class of Rényi information estimators for multidimensional densities (2010, background): “In particular , the so - called Sharma and Mittal entropy H q , s = [ 1 − ( I q ) ( s−1 ) / ( q−1 ) ] / ( s − 1 ) , with q , s > 0 , q , s = 1 and I q given by ( 1 .”
Estimation of Information Theoretic Measures for Continuous Random Variables (2008, background): “The idea of using k-nn density estimation to estimate the divergence was put forward in [26, 18], where they prove mean-square consistency of their estimator for finite k.”
Practical Bounds on Image Denoising: From Estimation to Information (2011, background): “This was later extended by considering -nearest neighbor distances in [36]–[38] (see also [ 39 ]).”
Minimax and maximin space-filling designs: some properties and methods for construction (2017, unknown): “For instance, Penrose and Yukich (2011) show that for the k -th NN graph, we have a.s. and L 2 convergence for β > 0 and L q convergence ( q = 1 , 2) for β ∈ ( − d / q , 0 ) ; see also Leonenko et al. (2008).”
Detecting and quantifying causal associations in large nonlinear time series datasets (2017, background): “While the Kozachenko-Leonenko estimator is asymptotically unbiased and consistent 50,54 , the variance and ﬁnite sample convergence rates are unknown.”
Limit theory for point processes in manifolds (2011, methodology): “The authors of the papers [16, 24, 29, 41, 44] consider estimators of Iρ(κ) in terms of the k nearest-neighbor graph; here we restrict to k = 1, but this is for presentational purposes only.”
On Estimating $L_2^2$ Divergence (2014, methodology): “This line of work has primarily focused on Kullback-Leibler, Renyi-α, and Csiszar f -divergences [11, 13, 14].”
On the Kozachenko-Leonenko entropy estimator (2016, background): “In [10], a consistency result is proved (for HN defined by (2)), in any dimension, under rather weak conditions on f and this is generalized to other notions of entropies by Leonenko, Pronzato and Savani in [11].”
On Estimating L 22 Divergence (2014, methodology): “This line of work has primarily focused on KullbackLeibler, Renyi-↵, and Csiszar f -divergences [12, 16, 17].”
Support Distribution Machines (2012, methodology): “Using the tools that have been applied for Rényi entropy (Leonenko et al., 2008), Shannon entropy (Goria et al.”