Families of Alpha- Beta- and Gamma- Divergences: Flexible and Robust Measures of Similarities

By Andrzej S. Cichocki, Shun-ichi Amari. Entropy, 2010

520 citations recorded by the source corpus.

In this paper, we extend and overview wide families of Alpha-, Beta- and Gamma-divergences and discuss their fundamental properties. In literature usually only one single asymmetric (Alpha, Beta or Gamma) divergence is considered. We show in this paper that there exist families of such divergences with the same consistent properties. Moreover, we establish links and correspondences among these divergences by applying suitable nonlinear transformations. For example, we can generate the Beta-divergences directly from Alpha-divergences and vice versa. Furthermore, we show that a new wide class of Gamma-divergences can be generated not only from the family of Beta-divergences but also from a family of Alpha-divergences. The paper bridges these divergences and shows also their links to Tsallis and Renyi entropies. Most of these divergences have a natural information theoretic interpretation.

Subfields: Cognitive Neuroscience, Statistical and Nonlinear Physics, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/97b2d25fede4b5b8dae910b54924e274d5cb8cd6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Statistical and Nonlinear Physics

In Statistics and Probability

Citations in the corpus

Entropy-Based Approach for Uncertainty Propagation of Nonlinear Dynamical Systems (2013, background): “connects the Itakura–Saito distance and the L2 distance and passes through the Kullback–Leibler divergence [25,26].”
f-Divergence constrained policy improvement (2017, background): “The particular choice of the family of functions fα is motivated by generalization of the natural logarithm (Cichocki and Amari, 2010).”
Type 1 and 2 mixtures of Kullback-Leibler divergences as cost functions in dimensionality reduction based on similarity preservation (2013, unknown): “Since BðriÞpHðriÞ for SNE and all its considered variants, we have bðsijÞpsij lnðsijÞ and @bðsijÞ @sij plnðsijÞþ1 : ð20Þ For the initialization, we suggest pð0Þi ¼ d 2 ik , where k is the index of the Kth shortest distance and K is the integer closest to the specified perplexity.”
Robust Bi-Tempered Logistic Loss Based on Bregman Divergences (2019, background): “The second form may be recognized as β-divergence [5] with parameter β = 2 – t.”
Algorithms for Nonnegative Matrix Factorization with the β-Divergence (2010, background): “Cichocki and Amari (2010) give a general Bregman generating function of the β - divergence, deﬁned for all β ∈ R , in the form of φ β 6 =0 , 1 ( y ) = ( y β − βy + β − 1) / ( β ( β − 1)), φ β =0 ( y ) = y − log y − 1 and φ β =1 ( y ) = y log y − y + 1.”
On f-Divergences: Integral Representations, Local Behavior, and Inequalities (2018, background): “Note, however, that the Beta and Gammadivergences in [7], as well as the generalized divergences in [8] and [9], are not f -divergences in general.”
Nonnegative Matrix Factorization: A Comprehensive Review (2013, background): “Other cost functions consist of Minkowski family of metrics known as ‘p-norm, Earth Mover’s distance metric [18], -divergence [17], -divergence [49], -divergence [50],”