Nonparametric Estimation of Renyi Divergence and Friends

By Akshay Krishnamurthy, Kirthevasan Kandasamy, Barnab'as P'oczos, Larry A. Wasserman. 2014

96 citations recorded by the source corpus.

We consider nonparametric estimation of L2, Renyi-α and Tsallis-α divergences between continuous distributions. Our approach is to construct estimators for particular integral functionals of two densities and translate them into divergence estimators. For the integral functionals, our estimators are based on corrections of a preliminary plug-in estimator. We show that these estimators achieve the parametric convergence rate of n-1/2 when the densities' smoothness, s, are both at least d/4 where d is the dimension. We also derive minimax lower bounds for this problem which confirm that s > d/4 is necessary to achieve the n-1/2 rate of convergence. We validate our theoretical guarantees with a number of simulations.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A Kernelized Stein Discrepancy for Goodness-of-fit Tests (2016, background): “We should point out that similar difﬁcult “constant” terms appear in other common discrepancy measures such as KL divergence and α -divergence (e.g., Krishnamurthy et al., 2014).”
Scalable Mutual Information Estimation Using Dependence Graphs (2018, methodology): “Certain KDE and KNN based estimators can achieve the optimal parametric MSE rate [12, 23, 25], however, implementation of the KDE and KNN methods respectively have O(N(2)) and O(kN logN) computational complexity, where N is the number of samples.”
Alleviating Semantics Distortion in Unsupervised Low-Level Image-to-Image Translation via Structure Consistency Constraint (2022, methodology): “For example, early works studied Non-parametric models based on Kernel Density Estimator (KDE) [21,22,29,50,51], K-nearest Neighbor Method (KNN) [27, 28], and likelihoodratio estimator [55] for MI estimation.”
On Estimating $L_2^2$ Divergence (2014, background): “extend their proof to a class of divergences [9].”
On Estimating L 22 Divergence (2014, background): “extend their proof to a class of divergences [9].”
Practical and Consistent Estimation of f-Divergences (2019, background): “While the majority of this focuses on ↵-divergences and closely related Rényi-↵ divergences [35, 37, 22], many works address specifically the KL-divergence [34, 39] with fewer considering f -divergences in full generality [28, 20, 26, 27].”
Demystifying fixed k-nearest neighbor information estimators (2016, methodology): “Recent works [29, 23] have studied the performance of the leave-one-out (LOO) approach where all but the sample of resubstitution are used for KDE, involving techniques such as von Mises expansion methods.”
Linear-Time Learning on Distributions with Approximate Kernel Embeddings (2015, background): “Another approach estimates a distance between distributions, often the L2 distance or Kullback-Leibler (KL) divergence, parametrically (Jaakkola and Haussler 1998; Moreno, Ho, and Vasconcelos 2003; Jebara, Kondor, and Howard 2004) or nonparametrically (Sricharan, Wei, and Hero 2013; Krishnamurthy et”
Proposal : Scalable , Active and Flexible Learning on Distributions (2015, unknown): “Krishnamurthy et al. (2014) correct a plug-in estimator for L2 andrα divergences by estimating higher order terms in the vonMises expansion; one of their estimators is computationally attractive and optimal for smooth distributions, while another is optimal for a broader range of distributions but r”
Plugin estimation of smooth optimal transport maps (2021, background): “We note that such periodicity constraints are commonly imposed in nonparametric estimation problems to mitigate boundary issues (Efromovich, 1999; Krishnamurthy et al., 2014; Han et al., 2019).”