Efficient multivariate entropy estimation via $k$-nearest neighbour distances

By Thomas B. Berrett, Richard J. Samworth, Ming Yuan. 2016

140 citations recorded by the source corpus.

Many statistical procedures, including goodness-of-fit tests and methods for independent component analysis, rely critically on the estimation of the entropy of a distribution. In this paper, we seek entropy estimators that are efficient and achieve the local asymptotic minimax lower bound with respect to squared error loss. To this end, we study weighted averages of the estimators originally proposed by Kozachenko and Leonenko (1987), based on the $k$-nearest neighbour distances of a sample of $n$ independent and identically distributed random vectors in $\mathbb{R}^d$. A careful choice of weights enables us to obtain an efficient estimator in arbitrary dimensions, given sufficient smoothness, while the original unweighted estimator is typically only efficient when $d \leq 3$. In addition to the new estimator proposed and theoretical understanding provided, our results facilitate the construction of asymptotically valid confidence intervals for the entropy of asymptotically minimal width.

Subfields: Computational Mechanics, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Statistics and Probability

Citations in the corpus

Measuring Association on Topological Spaces Using Kernels and Geometric Graphs (2020, unknown): “[11] Berrett, T.”
On boosting the power of Chatterjee’s rank correlation (2021, background): “Notably speaking, similar ideas were already pursued in Deb et al. (2020, Equations (3.3) and (8.5)); see also Berrett and Samworth (2019) for a related proposal that approximates the mutual information (Berrett et al., 2019) using M -NN-based statistics.”
Breaking the Bandwidth Barrier: Geometrical Adaptive Entropy Estimation (2016, methodology): “In general d-dimensions, [8, 33] prove bounds on the convergence rate of the bias for finite k = O(1), and [24, 1] for k = Ω(log n).”
On Azadkia–Chatterjee’s conditional dependence coefficient (2021, background): “Berrett et al. (2020, Section 2.2) argue that in many cases the unlabeled data, i.e., data on (X, Y ) but without the Z component, are plentiful, but labeled data on (X, Y,Z) jointly are scarce. In such cases, it is natural to assume that one not only knows (or may very accurately estimate) the need”
Estimating Mutual Information for Discrete-Continuous Mixtures (2017, background): “Recent progress involves an inspired mixture of an ensemble of kernel and k-NN estimators [46, 4].”
Estimating Information Flow in Neural Networks (2018, background): “The second work (Berrett et al., 2019) studies a weighted-kNN estimator in the high smoothness regime and proved its asymptotic efficiency.”