Learning deep kernels for exponential family densities

By Wenliang Li, Danica J. Sutherland, Heiko Strathmann, Arthur Gretton. 2018

79 citations recorded by the source corpus.

The kernel exponential family is a rich class of distributions,which can be fit efficiently and with statistical guarantees by score matching. Being required to choose a priori a simple kernel such as the Gaussian, however, limits its practical applicability. We provide a scheme for learning a kernel parameterized by a deep network, which can find complex location-dependent local features of the data geometry. This gives a very rich class of density models, capable of fitting complex structures on moderate-dimensional problems. Compared to deep density models fit via maximum likelihood, our approach provides a complementary set of strengths and tradeoffs: in empirical studies, the former can yield higher likelihoods, whereas the latter gives better estimates of the gradient of the log density, the score, which describes the distribution's shape.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Generative Modeling by Estimating Gradients of the Data Distribution (2019, methodology): “Our analyses of this are largely inspired by [63], which analyzed the same phenomenon in the context of density estimation with score matching.”
Sliced Score Matching: A Scalable Approach to Density and Score Estimation (2019, methodology): “For density estimation, experiments on deep kernel exponential families (Wenliang et al., 2019) and NICE ﬂow models (Dinh et al., 2015) show that our method is either more scalable or more accurate than existing score matching variants.”
Learning Gradient Fields for Shape Generation (2020, neutral): “For inference, we set T = 10 and α = 2 × 10 − 4 .”
Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefits (2019, result): “The above theorem formalizes the widely-acknowledged intuition in practice (Wenliang et al., 2018) that neural networks has the advantage of learning task-specific, data-dependent adaptive basis, compared to the conventional nonparametric fixed basis approach.”
Efficient Learning of Generative Models via Finite-Difference Score Matching (2020, background): “Deep kernel exponential family (DKEF) [68] is another unnormalized density estimator in the form of log p̃(x) = f(x)+ log p0(x), with p0 be the base measure, f(x) defined as ∑N i=1 ∑Nj j=1 ki(x, zj), where N is the number of kernels, k(·, ·) is the Gaussian kernel function, and zj (j = 0, · · · , Nj”
Learning Deep Kernels for Non-Parametric Two-Sample Tests (2020, methodology): “To model these complex functions, we adopt a deep kernel approach (Wilson et al., 2016; Sutherland et al., 2017; Li et al., 2017; Jean et al., 2018; Li et al., 2019), building a kernel with a deep network.”
Generalized Energy Based Models (2020, methodology): “Score Matching (Hyvärinen, 2005) calculates an alternative objective (the score) to the log-likelihood which is independent of the partition function, and was recently used in the context non-parametric energy functions to provide estimators of the energy that are provably consistent (Sriperumbudur ”
Bridging the Gap Between $f$-GANs and Wasserstein GANs (2019, methodology): “We first demonstrate the effectiveness of KL-WGAN on synthetic and UCI benchmark datasets (Asuncion & Newman, 2007) considered in (Wenliang et al., 2018).”
Robust generalised Bayesian inference for intractable likelihoods (2021, unknown): “For the kernel exponential family we use q ( x ) =  ( 0 , 3 2 ) and the kernel 𝜅 ( x , y ) = exp (−( x − y ) 2 ∕ 2 ) , which ensures that (16) is normalisable due to Propo-sition 2 of Wenliang et al. (2019).”
KALE: When Energy-Based Learning Meets Adversarial Training (2020, methodology): “All numerical results for the remaining methods were provided by authors of (Wenliang et al., 2019) upon request.”
How to Train Your Energy-Based Models (2021, background): “Score Matching has diﬃculty in estimating the relative weights of two modes separated by large low-density regions (Wenliang et al., 2019; Song and Ermon, 2019), which can have which does not depend on the weight π .”