Equivalence of distance-based and RKHS-based statistics in hypothesis testing

By Dino Sejdinovic, Bharath K. Sriperumbudur, Arthur Gretton, Kenji Fukumizu. 2012

794 citations recorded by the source corpus.

We provide a unifying framework linking two classes of statistics used in two-sample and independence testing: on the one hand, the energy distances and distance covariances from the statistics literature; on the other, maximum mean discrepancies (MMD), that is, distances between embeddings of distributions to reproducing kernel Hilbert spaces (RKHS), as established in machine learning. In the case where the energy distance is computed with a semimetric of negative type, a positive definite kernel, termed distance kernel, may be defined such that the MMD corresponds exactly to the energy distance. Conversely, for any positive definite kernel, we can interpret the MMD as energy distance with respect to some negative-type semimetric. This equivalence readily extends to distance covariance using kernels on the product space. We determine the class of probability distributions for which the test statistics are consistent against all alternatives. Finally, we investigate the performance of the family of distance kernels in two-sample and independence tests: we show in particular that the energy distance most commonly employed in statistics is just one member of a parametric family of kernels, and that other choices from this family can yield more powerful tests.

Subfields: Artificial Intelligence, Statistics and Probability, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Statistical Learning Theory

Citations in the corpus

DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, background): “Sejdinovic et al. (2013) showed the connection between the energy distance and MMD metrics.”
Demystifying MMD GANs (2018, methodology): “It is characteristic (when β 2 ), and the MMD is well-deﬁned for a class of distributions P that satisfy a moment condition (Sejdinovic et al., 2013, Remark 21).”
Improving GANs Using Optimal Transport (2018, background): “Using individual samples x , y instead of minibatches X , Y , Sejdinovic et al. (2013) showed that such generalizations of the energy distance can equivalently be viewed as a form of maximum mean discrepancy , where the MMD kernel k is related to the distance function d by d ( x , x 0 ) ≡ k ( x , x…”
Learning Transferable Features with Deep Adaptation Networks (2015, background): “On the one hand, the energy distances are derived from the statistical literature, and on the other hand, the maximum mean discrepancies (MMD) are established in the machine learning literature (Sejdinovic et al., 2013).”
Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “Bounliphone et al. (2015) constructs a consistent test for relative dependency which—unlike the Lancaster test of Sejdinovic et al. (2013a)—measures dependency between a source variable and two candidate target variables.”
NoPeek: Information leakage reduction to share activations in distributed deep learning (2020, methodology): “Our approach is based on minimizing a statistical dependency measure called distance correlation [10, 11, 12, 13, 14] between raw data and any intermediary communications across the clients or server participating in distributed deep learning.”
Interpretable Distribution Features with Maximum Testing Power (2016, background): “…on RKHS distances (Eric et al., 2008; Fromont et al., 2012; Gretton et al., 2012a) or energy distances (Székely and Rizzo, 2004; Baringhaus and Franz, 2004) have as their test statistic an integral probability metric, the Maximum Mean Discrepancy (Gretton et al., 2012a; Sejdinovic et al., 2013).”
Minimax Estimation of Kernel Mean Embeddings (2016, background): “d/2e−k·−xk22,x∈ Rd respectively reduces µkto these notions. The mean embedding µkis closely related to the generalized energy distance (Sz´ekely et al., 2007; Lyons, 2013), since the latter is shown (Sejdinovic et al., 2013, Theorem 22) to be equivalent to the RKHS distance between the mean embeddin”
A Kernel Test of Goodness of Fit (2016, methodology): “A different smoothing scheme in the frequency domain results in an energy distance statistic (this likewise being an MMD with a particular choice of kernel; see Sejdinovic et al., 2013), which can be used in a test of normality (Székely & Rizzo, 2005).”
Adaptivity and Computation-Statistics Tradeoffs for Kernel and Distance based High Dimensional Two Sample Testing (2015, background): “Sejdinovic et al. [2013] describes connections between kernel and distance based tests for independence testing.”
Fast Two-Sample Testing with Analytic Representations of Probability Measures (2015, background): “…distance (sup-norm between cumulative distribution functions); the Maximum Mean Discrepancy (MMD) [8], an RKHS norm of the difference between mean embeddings, and the N -distance (also known as energy distance) [33, 30, 3], which is an MMD-based test for a particular family of kernels [24] .”
K2-ABC: Approximate Bayesian Computation with Infinite Dimensional Summary Statistics via Kernel Embeddings (2015, background): “An embedding exists for any Fx whenever the kernel k is bounded, or if Fx satisfies a suitable moment condition w.r.t. an unbounded kernel k (Sejdinovic et al., 2013).”
A low variance consistent test of relative dependency (2014, background): “, strings, and graphs). The statistics for such tests are diverse, and include kernel measures of covariance [5, 19] and correlation [2, 3], distance covariances (which are instances of kernel tests) [18, 14], rankings [7], and space partitioning approaches [6, 12, 9]. For many problems in data anal”
On the Decreasing Power of Kernel and Distance Based Nonparametric Hypothesis Tests in High Dimensions (2014, methodology): “Informally, for every positive-deﬁnite kernel, there exists a negative-deﬁnite metric, and vice-versa, such that these quantities are equal; see (Sejdinovic et al. 2013) for more formal statements.”
FastMMD: Ensemble of Circular Discrepancy for Efficient Two-Sample Test (2014, background): “…much attention in recent twosample test research due to its solid theoretical fundament (Smola et al., 2008; Sriperumbudur et al., 2010; 2011; Sejdinovic et al., 2013) and successful applications including biological data test (Borgwardt et al., 2006), data integration and attribute matching…”
On the optimal estimation of probability measures in weak and strong topologies (2013, background): “n shown [30, Theorem 22] that choosing K = ˆ k(x,y) = 1 2 (ρ(x,z) +ρ(y,z)−ρ(x,y)),x,y,z∈ X ˙ yields the generalized energy distance [38, 24] where ρis a semimetric of negative type on X (see [24] and [30] for details) and zis somearbitraryelement in X. Speciﬁcally, the choice ofρ(x,y) = kx−yk2,x,y∈ ”
A Kernel Test for Three-Variable Interactions (2013, methodology): “Much of the work in this area focuses on measuring or testing pairwise interaction: for instance, the Hilbert-Schmidt Independence Criterion (HSIC) or Distance Covariance [1, 2, 3], kernel canonical correlation [4, 5, 6], and mutual information [7].”
A Spectral Energy Distance for Parallel Speech Synthesis (2020, positive): “[32], the kernel k(·, ·) of a RKHS H induces a distance metric d(·, ·) via”