Information geometry of divergence functions

By Shun-ichi Amari, Andrzej S. Cichocki. 2010

189 citations recorded by the source corpus.

Measures of divergence between two points play a key role in many engineering problems. One such measure is a distance function, but there are many important measures which do not satisfy the properties of the distance. The Bregman divergence, KullbackLeibler divergence and f -divergence are such measures. In the present article, we study the differential-geometrical structure of a manifold induced by a divergence function. It consists of a Riemannian metric, and a pair of dually coupled affine connections, which are studied in information geometry. The class of Bregman divergences are characterized by a dually flat structure, which is originated from the Legendre duality. A dually flat space admits a generalized Pythagorean theorem. The class of f -divergences, defined on a manifold of probability distributions, is characterized by information monotonicity, and the Kullback-Leibler divergence belongs to the intersection of both classes. The f -divergence always gives the α-geometry, which consists of the Fisher information metric and a dual pair of ±α-connections. The α-divergence is a special class of f -divergences. This is unique, sitting at the intersection of the f -divergence and Bregman divergence classes in a manifold of positive measures. The geometry derived from the Tsallis q-entropy and related divergences are also addressed.

Subfields: Geometry and Topology, Statistical and Nonlinear Physics, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/8381ed2160e2f0821678caab615f1b57b2c12bb0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Geometry and Topology

In Statistical and Nonlinear Physics

In Statistics and Probability

Citations in the corpus

The geometry of relative arbitrage (2014, methodology): “qjp) := log 1 + ˝ ˇ(p) p ;q p ˛ (log( q) log( p)); p;q2 (n): We use this terminology because the expression (22) can be regarded as a logarithmic version of the widely used Bregman divergence (see [AC10]). It is clear from (15) that T(qjp) 0 and is strictly positive unless is ane on the straight li”
Medians and Means in Riemannian Geometry: Existence, Uniqueness and Computation (2011, methodology): “A similar potential function was used by Amari in [3] to derive the Riemannian metric of multi-variate Gaussian distributions by means of divergence functions.”
Policy Optimization via Importance Sampling (2018, background): “Thanks to the invariance property, there is a tight connection between the geometry induced by the Rényi divergence and the Fisher information metric [3].”
Generative Adversarial Nets from a Density Ratio Estimation Perspective (2016, methodology): “Note that in the space of positive measures, i.e., not satisfying normalized conditions, f-divergence must satisfy f (cid:48) (1) = 0 due to its invariance [3].”
Families of Alpha- Beta- and Gamma- Divergences: Flexible and Robust Measures of Similarities (2010, background): “An insight using information geometry may further elucidates the fundamental structures of such divergences and geometry [1,3,7].”