Divergence Estimation for Multidimensional Densities Via $k$-Nearest-Neighbor Distances

By Qing Wang, Sanjeev R. Kulkarni, Sergio Verdú. 2009

414 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Nonparametric Divergence Estimation with Applications to Machine Learning on Distributions (2011, methodology): “Wang et al. (2009b) 1 and Pérez-Cruz (2008) 2 provided an estimator for kl -divergence, and Póczos and Schneider (2011) developed nonparametric methods for the Rényi and Tsallis divergence estimation.”
Linear-Time Learning on Distributions with Approximate Kernel Embeddings (2015, methodology): “We ﬁnally try the full Gram matrix approach of P´oczos et al. (2012b) with the KL estimator of Wang, Kulkarni, and Verd´u (2009) in an RBF kernel ( KL ), as did Ntampaka et al. (2015).”
Minimax Estimation of Functionals of Discrete Distributions (2014, background): “For nonparametric estimation of differential entropy, the readers are referred to [80], [81], [82], [83], and [84].”
Proposal : Scalable , Active and Flexible Learning on Distributions (2015, neutral): “For mixtures specifically of Gaussians, F. Wang et al. (2009) obtain the quadratic ( 2 ) entropy, which allows the computation of Jensen-Rényi divergences for α = 2.”
Breaking the Bandwidth Barrier: Geometrical Adaptive Entropy Estimation (2016, background): “As noted in [28] this is common in the literature on consistency of k-NN estimators, where the same assumptions have been implicitly made without explicitly stating as such, in existing analyses of entropy estimators including [15, 9, 18, 39].”
Support Distribution Machines (2012, methodology): “Using the tools that have been applied for Rényi entropy (Leonenko et al., 2008), Shannon entropy (Goria et al., 2005), KL divergence (Wang et al., 2009), and Rényi divergence estimation (Póczos and Schneider, 2011), we show how to estimate Dα,β(p∥q) in an efficient, nonparametric, and…”
Deep Learning-Based End-to-End Wireless Communication Systems With Conditional GANs as Unknown Channels (2019, methodology): “5 shows the Kullback-Leibler (KL) divergence [36] of the generated distribution and the real distribution, KL(G(z)||pdata), which is estimated based on the k-nearest neighbor estimation approach in [37].”
Estimating Mutual Information for Discrete-Continuous Mixtures (2017, background): “Estimator of the average of Radon-Nikodym derivative dP/dQ has been studied in [46, 47].”
Estimation of Renyi Entropy and Mutual Information Based on Generalized Nearest-Neighbor Graphs (2010, background): “For further applications, see Leonenko et al. (2008); Wang et al. (2009a).”
Nonparametric Estimation of Renyi Divergence and Friends (2014, methodology): “Many of the approaches are based on nearest-neighbor graphs (Hero & Michel, 1999; Wang et al., 2009; Póczos & Schneider, 2011; Källberg & Seleznjev, 2012).”
Machine Learning Methods for Attack Detection in the Smart Grid (2015, background): “, si(k)}, is constructed by computing the Euclidean distances between the samples [18], where i(1), i(2), .”
Tighter Variational Representations of f-Divergences via Restriction to Probability Measures (2012, methodology): “We include the following recent estimators for comparison: • Wang et al (Wang et al., 2009): This estimator is based on nearest neighbour estimates of the two densities and does not make use of a variational representation.”
Point Cloud GAN (2018, background): “However, the estimator with finite samples are either biased or with high variance (Peyré et al., 2017; Wang et al., 2009; Póczos et al., 2012; Weed and Bach, 2017).”