On integral probability metrics, φ-divergences and binary classification

By Bharath K. Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, Gert R. G. Lanckriet. 2009

226 citations recorded by the source corpus.

A class of distance measures on probabilities -- the integral probability metrics (IPMs) -- is addressed: these include the Wasserstein distance, Dudley metric, and Maximum Mean Discrepancy. IPMs have thus far mostly been used in more abstract settings, for instance as theoretical tools in mass transportation problems, and in metrizing the weak topology on the set of all Borel probability measures defined on a metric space. Practical applications of IPMs are less common, with some exceptions in the kernel machines literature. The present work contributes a number of novel properties of IPMs, which should contribute to making IPMs more widely used in practice, for instance in areas where $\phi$-divergences are currently popular. First, to understand the relation between IPMs and $\phi$-divergences, the necessary and sufficient conditions under which these classes intersect are derived: the total variation distance is shown to be the only non-trivial $\phi$-divergence that is also an IPM. This shows that IPMs are essentially different from $\phi$-divergences. Second, empirical estimates of several IPMs from finite i.i.d. samples are obtained, and their consistency and convergence rates are analyzed. These estimators are shown to be easily computable, with better rates of convergence than estimators of $\phi$-divergences. Third, a novel interpretation is provided for IPMs by relating them to binary classification, where it is shown that the IPM between class-conditional distributions is the negative of the optimal risk associated with a binary classifier. In addition, the smoothness of an appropriate binary classifier is proved to be inversely related to the distance between the class-conditional distributions, measured in terms of an IPM.

Subfields: Statistical and Nonlinear Physics, Statistics and Probability, Probabilistic & Bayesian Methods, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistical and Nonlinear Physics

In Statistics and Probability

In Probabilistic & Bayesian Methods

In Statistical Learning Theory

Citations in the corpus

Demystifying MMD GANs (2018, background): “But the population divergence the generator attempts to minimize is actually a distance previously studied by Sriperumbudur et al. (2009a) as well as Li et al. (2017b).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “We refer the readers to Sriperumbudur et al. (2009) for a detailed discussion on IPMs.”
MOPO: Model-based Offline Policy Optimization (2020, background): “IPMs are quite general and contain several other distance measures as special cases [61].”
GenDICE: Generalized Offline Estimation of Stationary Values (2020, methodology): “Based on the definition of the IPM, these divergences directly lead to min-max optimizations similar to Equation 13 with the identity function as φ∗ (·) and different feasible sets for the dual functions.”
Hilbert Space Embeddings and Metrics on Probability Measures (2009, background): “In Section 3.1, we present the simple characterization that integrally strictly positive definite (pd) kernels (see Section 1.2 for the definition) are characteristic, i.e., the induced RKHS is characteristic (also see Sriperumbudur et al., 2009a, Theorem 4).”
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations (2017, background): “Replacing KL(qφ(z)‖p(z)) with another suitable distance between qφ(z) and p(z) (e.g., integral probability metrics like Wasserstein distance (Sriperumbudur et al., 2009)) might alleviate some of these issues (Arjovsky et al., 2017) but will still involve complicating the optimization to a saddle…”
Learning in Implicit Generative Models (2016, methodology): “…Dudley and Wasserstein distances can easily be considered, establishing even further connections to other growing areas, such as optimal transport (Frogner et al., 2015; Sriperumbudur et al., 2009); Arjovsky et al. (2017) has since demonstrated the effectiveness of using the Wasserstein distance.”
Stabilizing Training of Generative Adversarial Networks through Regularization (2017, methodology): “As a remedy, it has been suggested to use an alternative family of distance functions known as integral probability metrics Müller (1997); Sriperumbudur et al. (2009).”
Computational Optimal Transport (2018, background): “The total variation distance is the only non-trivial divergence that is also a dual norm, see [Sriperumbudur et al., 2009].”
McGan: Mean and Covariance Feature Matching GAN (2017, methodology): “Intuitively each IPM ﬁnds a “critic” f (Arjovsky et al., 2017) which maximally discriminates between the distributions.”
Connections between Support Vector Machines, Wasserstein distance and gradient-penalty GANs (2019, background): “Of note, certain connections between IPMs and SVMs have been identified in Sriperumbudur et al. (2009).”
On the Discrimination-Generalization Tradeoff in GANs (2017, background): “However, φ -divergence has substantially different properties from IPM (see e.g., Sriperumbudur et al. (2009)), and is not deﬁned as the intuitive moment matching form as IPM.”
How Generative Adversarial Networks and Its Variants Work : An Overview of GAN (2017, background): “Because total variance is the only intersection between IPM and f-divergence [107], it inherits some disadvantages for estimating f-divergence as discussed by Arjovsky et al.”
Representation Balancing Offline Model-based Reinforcement Learning (2021, background): “…bound their difference in expectations, which is the Integral Probability Metric (IPM) (M ¨ uller, 1997): ( where particular choices of G make the IPM equivalent to different well-known distances of distributions, e.g. total variation distance or Wasserstein distance (Sriperumbudur et al., 2009).”