Kernel Mean Embedding of Distributions: A Review and Beyonds

By Krikamol Muandet, Kenji Fukumizu, Bharath K. Sriperumbudur, Bernhard Schölkopf. Found. Trends Mach. Learn., 2016

901 citations recorded by the source corpus.

A Hilbert space embedding of a distribution---in short, a kernel mean embedding---has recently emerged as a powerful tool for machine learning and inference. The basic idea behind this framework is to map distributions into a reproducing kernel Hilbert space (RKHS) in which the whole arsenal of kernel methods can be extended to probability measures. It can be viewed as a generalization of the original "feature map" common to support vector machines (SVMs) and other kernel methods. While initially closely associated with the latter, it has meanwhile found application in fields ranging from kernel machines and probabilistic modeling to statistical inference, causal discovery, and deep learning. The goal of this survey is to give a comprehensive review of existing work and recent advances in this research area, and to discuss the most challenging issues and open problems that could lead to new research directions. The survey begins with a brief introduction to the RKHS and positive definite kernels which forms the backbone of this survey, followed by a thorough discussion of the Hilbert space embedding of marginal distributions, theoretical guarantees, and a review of its applications. The embedding of distributions enables us to apply RKHS methods to probability measures which prompts a wide range of applications such as kernel two-sample testing, independent testing, and learning on distributional data. Next, we discuss the Hilbert space embedding for conditional distributions, give theoretical insights, and review some applications. The conditional mean embedding enables us to perform sum, product, and Bayes' rules---which are ubiquitous in graphical model, probabilistic inference, and reinforcement learning---in a non-parametric way. We then discuss relationships between this framework and other related areas. Lastly, we give some suggestions on future research directions.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods, Statistical Learning Theory

View Paper: https://arxiv.org/abs/1605.09522

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

In Statistical Learning Theory

Citations in the corpus

Prototypal Analysis and Prototypal Regression (2017, background): “Probability distributions or samples thereof can also be mapped to a reproducing kernel Hilbert space via kernel embedding (see Berlinet and Thomas-Agnan, 2004; Gretton et al., 2006; Smola et al., 2007; Sriperumbudur et al., 2010; Sejdinovic et al., 2012; Muandet et al., 2017).”
Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning (2020, methodology): “Here, we describe one such algorithm based on Maximum Mean Discrepancy (MMD) [26].”
MMD GAN: Towards Deeper Understanding of Moment Matching Network (2017, positive): “The unified view shows more connections between moment matching and GAN, which can potentially inspire new algorithms based on well-developed tools in statistics [15].”
In Search of Lost Domain Generalization (2020, background): “A.2 Sharing parameters Blanchard et al. [2011] build classiﬁers f ( x d , µ d ) , where µ d is a kernel mean embedding [Muandet et al., 2017] that summarizes the dataset associated to the example x d .”
Pros and Cons of GAN Evaluation Measures (2018, methodology): “This metric measures the dissimilarity between two probability distributions2 Pr and Pg using samples drawn independently from each [25, 66, 88, 8].”
McGan: Mean and Covariance Feature Matching GAN (2017, methodology): “In this paper, inspired by the MMD distance and the kernel mean embedding of distributions (Muandet et al., 2016) we propose to embed distributions in a finite dimensional feature space and to match them based on their mean and covariance feature statistics.”
A Kernel Theory of Modern Data Augmentation (2018, methodology): “This feature map is exactly the embedding of the distribution of transformed points around x into the reproducing kernel Hilbert space [26, 30].”
Local Group Invariant Representations via Orbit Embeddings (2016, methodology): “(3) and consider HK, the RKHS induced by the kernel kq,G(x, y) = ∫ G ∫ G k(gx, g′x′) q(g)q(g′) dν(g)dν(g′), as introduced in Sec.”
Measuring Association on Topological Spaces Using Kernels and Geometric Graphs (2020, background): “Deb, Ghosal and Sen/A Kernel Measure of Association 32 [73] Muandet, K.”
Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings (2018, methodology): “This is analogous to the way MMD over RKHS kernels expresses the distance between two probability distributions (Muandet et al., 2017; Li et al., 2015).”
Automated Machine Learning: Methods, Systems, Challenges (2019, neutral): “It comes back to solve a constrained nonlinear ICA problem, that can be achieved by minimizing I (X, N̂Y ; θ), the mutual information between X and N̂Y [34] with respect to the parameter of the model θ .”
Group Invariance, Stability to Deformations, and Complexity of Deep Convolutional Representations (2017, neutral): “where the integral is a Bochner integral (see, [15, 26]).”
Sinkhorn Barycenters with Free Support via Frank-Wolfe Algorithm (2019, methodology): “For sufficiently large s, the Sobolev norm is in duality with the MMD [35] and allows us to derive the required Lipschitz continuity.”
Bayesian Optimization over Sets (2019, background): “…− 1 (cid:80) | X | i =1 φ ( x i ) where φ ( ) is a basis function R d → d and d (cid:48) is a dimension of projected space by φ , a set kernel [11, 21] is deﬁned as (cid:104) (cid:105) Here, k : X × X → R is a positive-deﬁnite kernel deﬁned to measure the covariance between the d -dimensional…”