Bayesian Learning of Kernel Embeddings

By Seth R. Flaxman, Dino Sejdinovic, John P. Cunningham, Sarah Filippi. 2016

48 citations recorded by the source corpus.

Kernel methods are one of the mainstays of machine learning, but the problem of kernel learning remains challenging, with only a few heuristics and very little theory. This is of particular importance in methods based on estimation of kernel mean embeddings of probability measures. For characteristic kernels, which include most commonly used ones, the kernel mean embedding uniquely determines its probability measure, so it can be used to design a powerful statistical testing framework, which includes nonparametric two-sample and independence tests. In practice, however, the performance of these tests can be very sensitive to the choice of kernel and its lengthscale parameters. To address this central issue, we propose a new probabilistic model for kernel mean embeddings, the Bayesian Kernel Embedding model, combining a Gaussian process prior over the Reproducing Kernel Hilbert Space containing the mean embedding with a conjugate likelihood function, thus yielding a closed form posterior over the mean embedding. The posterior mean of our model is closely related to recently proposed shrinkage estimators for kernel mean embeddings, while the posterior uncertainty is a new, interesting feature with various possible applications. Critically for the purposes of kernel learning, our model gives a simple, closed form marginal pseudolikelihood of the observed data given the kernel hyperparameters. This marginal pseudolikelihood can either be optimized to inform the hyperparameter choice or fully Bayesian inference can be used.

Subfields: Artificial Intelligence, Statistics and Probability, Probabilistic & Bayesian Methods, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Probabilistic & Bayesian Methods

In Statistical Learning Theory

Citations in the corpus

Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, background): “preserves angles locally—given by k̃(x,y) := f(x)k(x,y)f(y) for any bounded continuous function f such that f(x) > 0 for all x ∈ X and k(x,x)|f(x)|2 is bounded (Fukumizu et al. 2009b; Lemma 2), see, also Wu and Amari (2002). The richness of RKHS was previously studied through the so-called universal”
Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences (2018, methodology): “e familiar with GPs but not with RKHSs, and vice versa. We also review a shrinkage estimator for kernel means proposed by Muandet et al. [2016] and the corresponding GP-based Bayesian interpretation [Flaxman et al., 2016]. We then discuss the problem of sampling or numerical integration, for which G”
Information Constraints on Auto-Encoding Variational Bayes (2018, background): “Second, the problem of choosing the best kernel is known to be difficult [46].”