Hilbert Space Embeddings and Metrics on Probability Measures

By Bharath K. Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Bernhard Schölkopf, Gert R. G. Lanckriet. 2009

850 citations recorded by the source corpus.

A Hilbert space embedding for probability measures has recently been proposed, with applications including dimensionality reduction, homogeneity testing, and independence testing. This embedding represents any probability measure as a mean element in a reproducing kernel Hilbert space (RKHS). A pseudometric on the space of probability measures can be defined as the distance between distribution embeddings: we denote this as γk, indexed by the kernel function k that defines the inner product in the RKHS. We present three theoretical properties of γk. First, we consider the question of determining the conditions on the kernel k for which γk is a metric: such k are denoted characteristic kernels. Unlike pseudometrics, a metric is zero only when two distributions coincide, thus ensuring the RKHS embedding maps all distributions uniquely (i.e., the embedding is injective). While previously published conditions may apply only in restricted circumstances (e.g., on compact domains), and are difficult to check, our conditions are straightforward and intuitive: integrally strictly positive definite kernels are characteristic. Alternatively, if a bounded continuous kernel is translation-invariant on ℜd, then it is characteristic if and only if the support of its Fourier transform is the entire ℜd. Second, we show that the distance between distributions under γk results from an interplay between the properties of the kernel and the distributions, by demonstrating that distributions are close in the embedding space when their differences occur at higher frequencies. Third, to understand the nature of the topology induced by γk, we relate γk to other popular metrics on probability measures, and present conditions on the kernel k under which γk metrizes the weak topology.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Radiology, Nuclear Medicine and Imaging

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization (2016, background): “MMD is a particular instance of a larger class of probability metrics [32] which all take the form D(P,Q) = supT∈T |Ex∼P [T (x)]− Ex∼Q[T (x)]|, where the function class T is chosen in a manner specific to the divergence.”
Sequential Kernel Herding: Frank-Wolfe Optimization for Particle Filtering (2015, methodology): “It turns out that the worst-case error of estimators of the form (2) can be analyzed in terms of their approximation distance to the mean element µ ( p ) := E p [Φ] ∈ H (Smola et al., 2007; Sriperumbudur et al., 2010).”
Demystifying MMD GANs (2018, methodology): “…P , and Y = { y j } nj =1 drawn i.i.d. from Q , the empirical witness function is and an unbiased estimator of the squared MMD is (Gretton et al., 2012, Lemma 6) When the kernel is characteristic (Sriperumbudur et al., 2010; 2011), the embedding µ is injective (i.e., associated uniquely with P ).”
Geometric GAN (2017, methodology): “Then, the integral probability metric (IPM) between P and Q on the function space F is deﬁned as follows [8, 15, 16]: where E x ∼ P [ · ] denotes the expectation with respect to the probability distribution P ( x ) .”
Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “Moreover, when k is translation invariant, i.e., k(x,x′) = ψ(x − x′), we have MMD(H ,P,Q) = ∫Rd |ϕP(ω) − ϕQ(ω)|2F−1ψ(ω) dω where F−1 denotes the inverse Fourier transform and ϕP, ϕQ are characteristic functions of P,Q, respectively, (Sriperumbudur et al. 2010; Corollary 4).”
Domain Generalization via Conditional Invariant Representations (2018, methodology): “Before introducing the proposed method, we briefly review the kernel mean embedding of distributions, which is an important mathematical tool to represent and compare distributions (Song, Fukumizu, and Gretton 2013; Sriperumbudur et al. 2010).”
Prototypal Analysis and Prototypal Regression (2017, background): “The Gaussian, Laplacian and B2n+1-spline kernels are shown to induce injective embeddings in Sriperumbudur et al. (2010).”
Consistent Kernel Mean Estimation for Functions of Random Variables (2016, methodology): “…(i) it works for any input type, (ii) the sample size controls the time-accuracy trade-off, and (iii) applying functions to random variables reduces to applying the functions pointwise 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain. ar X iv :1 61 0.”
Convergence guarantees for kernel-based quadrature rules in misspecified settings (2016, background): “We refer the reader to [21, 11] for details of these derivations.”
Interpretable Distribution Features with Maximum Testing Power (2016, background): “For the same reason, SCF-full does much better than the quadratic-time MMD across most sample sizes, as the latter represents a weighted distance between characteristic functions integrated across the entire frequency domain (Sriperumbudur et al., 2010, Corollary 4).”
Minimax Estimation of Kernel Mean Embeddings (2016, methodology): “Since ψ is characteristic, i.e., P 7→ µk(P ) ∈ Hk is injective, which is guaranteed if and only if supp(ψ∧) = Rd (Sriperumbudur et al., 2010, Theorem 9), it follows from (13) that P 7→ µk(P ) ∈ L1(Rd) ∩ L2(Rd) is injective.”
A Kernel Test of Goodness of Fit (2016, methodology): “For this choice of smoothing function, their statistic is identical to the maximum mean discrepancy (MMD) with the exponentiated quadratic kernel, which can be shown using the Bochner representation of the kernel (Sriperumbudur et al., 2010, Corollary 4).”
A Test of Relative Similarity For Model Selection in Generative Models (2015, background): “The maximum mean discrepancy is a metric on the space of probability distirbutions when a characteristic kernel is used (Fukumizu et al., 2008; Sriperumbudur et al., 2010), meaning that the distribution embeddings are unique for each probability measure.”
Fast Two-Sample Testing with Analytic Representations of Probability Measures (2015, methodology): “a kernel is called characteristic when MMD(P;Q) = 0 iff P= Q: (5) Any bounded, continuous, translation-invariant kernel whose inverse Fourier transform is almost everywhere non-zero is characteristic [26]. By representation (2), it is clear that MMD with a characteristic kernel is a metric. Pseudome”
On the Equivalence between Kernel Quadrature Rules and Random Feature Expansions (2015, background): “(7) corresponds to a particular metric between the distribution dρ and its corresponding empirical distribution (Sriperumbudur et al., 2010).”
K2-ABC: Approximate Bayesian Computation with Infinite Dimensional Summary Statistics via Kernel Embeddings (2015, methodology): “The problem is equivalent to learning a regression function in the RKHSs of s(y) and θ induced by their respective kernels (Grünewälder et al., 2012).”
Computing functions of random variables via reproducing kernel Hilbert space representations (2015, neutral): “One of the advantages of this approach is that it allows us to apply RKHS methods to probability distributions, often with strong theoretical guarantees (Sriperumbudur et al. 2008, 2010).”
Non-Linear Regression for Bag-of-Words Data via Gaussian Process Latent Variable Set Model (2015, background): “It is known that kernel embeddingm(P) preserves the properties of probability distribution P such as mean, covariance and higher-order moments by using characteristic kernels (e.g. Gaussian RBF kernel) (Sriperumbudur and Gretton 2010).”
Distinguishing Cause from Effect Using Observational Data: Methods and Benchmarks (2014, background): “For more details on the notion of characteristic kernel, see (Sriperumbudur et al., 2010).”
Learning Theory for Distribution Regression (2014, background): “For example, if k is universal, then µ metrizes the weak topology τw (Sriperumbudur et al., 2010, Theorem 23, page 1552), hence µ is continuous.”