Density Estimation in Infinite Dimensional Exponential Families

By Bharath K. Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Aapo Hyvärinen, Revanth Kumar, Revant Kumar. 2013

145 citations recorded by the source corpus.

In this paper, we consider an infinite dimensional exponential family, $\mathcal{P}$ of probability densities, which are parametrized by functions in a reproducing kernel Hilbert space, $H$ and show it to be quite rich in the sense that a broad class of densities on $\mathbb{R}^d$ can be approximated arbitrarily well in Kullback-Leibler (KL) divergence by elements in $\mathcal{P}$. The main goal of the paper is to estimate an unknown density, $p_0$ through an element in $\mathcal{P}$. Standard techniques like maximum likelihood estimation (MLE) or pseudo MLE (based on the method of sieves), which are based on minimizing the KL divergence between $p_0$ and $\mathcal{P}$, do not yield practically useful estimators because of their inability to efficiently handle the log-partition function. Instead, we propose an estimator, $\hat{p}_n$ based on minimizing the \emph{Fisher divergence}, $J(p_0\Vert p)$ between $p_0$ and $p\in \mathcal{P}$, which involves solving a simple finite-dimensional linear system. When $p_0\in\mathcal{P}$, we show that the proposed estimator is consistent, and provide a convergence rate of $n^{-\min\left\{\frac{2}{3},\frac{2\beta+1}{2\beta+2}\right\}}$ in Fisher divergence under the smoothness assumption that $\log p_0\in\mathcal{R}(C^\beta)$ for some $\beta\ge 0$, where $C$ is a certain Hilbert-Schmidt operator on $H$ and $\mathcal{R}(C^\beta)$ denotes the image of $C^\beta$. We also investigate the misspecified case of $p_0\notin\mathcal{P}$ and show that $J(p_0\Vert\hat{p}_n)\rightarrow \inf_{p\in\mathcal{P}}J(p_0\Vert p)$ as $n\rightarrow\infty$, and provide a rate for this convergence under a similar smoothness condition as above. Through numerical simulations we demonstrate that the proposed estimator outperforms the non-parametric kernel density estimator, and that the advantage with the proposed estimator grows as $d$ increases.

Subfields: Artificial Intelligence, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “They use an infinite dimensional exponential family model in an RKHS as a surrogate model and learn it via score matching (Hyvärinen 2005, Sriperumbudur et al. 2013).”
A Kernelized Stein Discrepancy for Goodness-of-fit Tests (2016, methodology): “…divergence has been widely used for parameter estimation, ﬁnding the optimal q ( x ) that best ﬁts a sample { x i } by minimizing F ( p, q ) ; this yields the score matching methods developed in both parametric (Hyv¨arinen, 2005; Lyu, 2009) and non-parametric (Sriperumbudur et al., 2013) settings.”
A Kernel Test of Goodness of Fit (2016, methodology): “We evaluate two density models: the infinite dimensional exponential family (Sriperumbudur et al., 2014), and a recent approximation to this model using random Fourier features (Strathmann et al.”
Kernel Sequential Monte Carlo (2015, methodology): “Secondly, we use an infinite dimensional exponential family model [21] to estimate target gradients as in Strathmann et al.”
Gradient-free Hamiltonian Monte Carlo with Efficient Kernel Exponential Families (2015, methodology): “The original estimator of f in (2) takes a dual form in a RKHS sub-space spanned by nd + 1 kernel derivatives, [13, Thm. 4].”
Optimal Rates for Random Fourier Features (2015, positive): “To this end, we considerk as in (1) and defineha := cos(πa2 + ·), a ∈ N (in other wordsh0 = cos, h1 = − sin, h2 = − cos, h3 = sin andha = hamod4).”
Variational Autoencoders and Nonlinear ICA: A Unifying Framework (2019, background): “Note that exponential families have universal approximation capabilities, so this assumption is not very restrictive [25].”
Sliced Score Matching: A Scalable Approach to Density and Score Estimation (2019, methodology): “Here, q 0 is a ﬁxed function and f belongs to a reproducing kernel Hilbert space H , with kernel k (Canu & Smola, 2006; Sriperumbudur et al., 2017).”
Learning structured densities via infinite dimensional exponential families (2015, unknown): “1 (Lemma 5 in [8] or Theorem 5 in [26] ) (Properties of C) 1.”
KALE: When Energy-Based Learning Meets Adversarial Training (2020, methodology): “Examples of non-parametric spaces are Reproducing Kernel Hilbert Spaces (RKHS) as considered in (Canu & Smola, 2006; Fukumizu, 2009; Sriperumbudur et al., 2017).”
Efficient Learning of Generative Models via Finite-Difference Score Matching (2020, background): “Different variants of SM have been proposed, including approximate back-propagation [25], curvature propagation [39], denoising score matching (DSM) [65], a bi-level formulation for latent variable models [3] and nonparametric estimators [35, 55, 59, 62, 74], but they may suffer from high computatio”
Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods (2017, background): “Moreover, there have been recent advances in non-parametric score matching methods (Sriperumbudur et al., 2013) for density estimation in infinite dimensional exponential families with guaranteed convergence rates.”
Kernel Exponential Family Estimation via Doubly Dual Embedding (2018, methodology): “However, as Sriperumbudur et al. (2017) note, even with the ﬁnite-dimension approximation, these algorithms are still expensive as every update requires Monte-Carlo sampling from a current model to compute the partition function.”
Generalized Energy Based Models (2020, methodology): “Score Matching (Hyvärinen, 2005) calculates an alternative objective (the score) to the log-likelihood which is independent of the partition function, and was recently used in the context non-parametric energy functions to provide estimators of the energy that are provably consistent (Sriperumbudur ”
Exponential Family Estimation via Adversarial Dynamics Embedding (2019, background): “, 2006) as a special case, by setting f 0 (x) = $ (x) with k = 1, which has been generalized to the infinite dimensional case (Sriperumbudur et al., 2017).”
A Spectral Approach to Gradient Estimation for Implicit Distributions (2018, background): “it, a kernel exponential family (Sriperumbudur et al., 2017) is fit to samples along the Markov Chain trajectory by score matching (Hyvärinen, 2005), and then serves as a surrogate of the target distribution to provide gradient estimates for HMC.”