On the chi square and higher-order chi distances for approximating f-divergences

By Frank Nielsen, Richard Nock. 2013

136 citations recorded by the source corpus.

We report closed-form formula for calculating the Chi square and higher-order Chi distances between statistical distributions belonging to the same exponential family with affine natural space, and instantiate those formula for the Poisson and isotropic Gaussian families. We then describe an analytic formula for the f-divergences based on Taylor expansions and relying on an extended class of Chi-type distances.

Subfields: Applied Mathematics, Statistical and Nonlinear Physics, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Mathematics

In Statistical and Nonlinear Physics

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization (2016, background): “Part of the list of divergences and their generators is based on [26].”
Subsampled Rényi Differential Privacy and Analytical Moments Accountant (2018, background): “For even j , the χ j -divergence can be written in an analytical form as a Rényi divergence (Nielsen & Nock, 2014) using a binomial expansion.”
Poission Subsampled Rényi Differential Privacy (2019, methodology): “…is presented in Appendix A In the theorem, M(X)M(X′) − 1 is a linearized version of the privacy random variable log M(X)M(X′) and Dχℓ(M(X)∥M(X ′)) is the Pearson-Vajda χℓ pseudodivergence (Vajda, 1973), which has more recently been used to approximate any f -divergence in (Nielsen & Nock, 2014).”
Operator Variational Inference (2016, methodology): “to the approximating family Q and maximizing the objective with respect to the function class F We index the family Q with variational parameters (cid:21) and require that it satisﬁes properties typically assumed by black box methods [19]: the variational distribution q. z I (cid:21)/ has a known…”
Multinomial Adversarial Networks for Multi-Domain Text Classification (2018, methodology): “When D is trained to its optimality: where χ 2 Neyman ( ) is the Neyman χ 2 divergence (Nielsen and Nock, 2014).”
Practical and Consistent Estimation of f-Divergences (2019, background): “The closed form expression for the χ 2 -divergence between two d -variate normal distributions can be found in Lemma 1 of [29]: As a corollary, the following also holds: Corollary 1.”