Kernel Exponential Family Estimation via Doubly Dual Embedding

By Bo Dai, Hanjun Dai, Arthur Gretton, Le Song, Dale Schuurmans, Niao He. 2018

36 citations recorded by the source corpus.

We investigate penalized maximum log-likelihood estimation for exponential family distributions whose natural parameter resides in a reproducing kernel Hilbert space. Key to our approach is a novel technique, doubly dual embedding, that avoids computation of the partition function. This technique also allows the development of a flexible sampling strategy that amortizes the cost of Monte-Carlo sampling in the inference stage. The resulting estimator can be easily generalized to kernel conditional exponential families. We establish a connection between kernel exponential family estimation and MMD-GANs, revealing a new perspective for understanding GANs. Compared to the score matching based estimators, the proposed method improves both memory and time efficiency while enjoying stronger statistical properties, such as fully capturing smoothness in its statistical convergence rate while the score matching estimator appears to saturate. Finally, we show that the proposed estimator empirically outperforms state-of-the-art

Subfields: Artificial Intelligence, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, background): “Currently, most attempts to ﬁx the mismatch lie in designing better sampling algorithms ( e.g. , Hamiltonian Monte Carlo [53], annealed importance sampling [52]) or exploiting variational techniques [34, 15, 14] to reduce the inference approximation error.”
Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methods (2019, background): “Examples of problems that are formulated under this framework include generative adversarial networks (GANs) [51], reinforcement learning [11], adversarial learning [53], learning exponential families [10], fair statistical inference [17, 56, 52, 37], generative adversarial imitation learning [6, 27”
Exponential Family Estimation via Adversarial Dynamics Embedding (2019, methodology): “Our development starts with the primal-dual view of the MLE (Dai et al., 2019) that provides a natural objective for jointly learning both a sampler and a model, as a remedy for the expensive and biased MCMC steps in the CD algorithm.”
KALE: When Energy-Based Learning Meets Adversarial Training (2020, methodology): “(Dai et al., 2019a;b) consider the task of MLE using an inﬁnite-dimensional set of energies.”
Training Deep Energy-Based Models with f-Divergence Minimization (2020, methodology): “Based on the primal-dual view of MLE, doubly dual embedding (Dai et al., 2018) and adversarial dynamics embedding (Dai et al.”
Generalized Energy Based Models (2020, methodology): “Thus, various methods have been proposed to address this (Hinton, 2002; Hyvärinen, 2005; Gutmann and Hyvärinen, 2012; Dai et al., 2019a;b).”
Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models (2020, unknown): “Similar to the Hessian-vector products (Song et al., 2019), only two backpropagations are required in the calculation (since z ∂ ∇ v log ˜ p θ ( v , h i ) ∂ θ = ∂ z (cid:62) ∇ v log ˜ p θ ( v , h i ) ∂ θ ).”
On Fenchel Mini-Max Learning (2019, background): “In the case of MLE, let h ( θ ; ω ) be an unbiased stochastic gradient estimator for ˆ L ( θ ) , i.e. , E ω ∼ p ( ω ) [ h ( θ ; ω )] = ∇ ˆ L ( θ ) .”
How to Train Your Energy-Based Models (2021, unknown): “Dai et al. (2019a) notice that when deﬁning p θ ( x ) = p 0 ( x ) e − E θ ( x ) /Z θ , with p 0 ( x ) being a ﬁxed base distribution, the entropy term − H ( q φ ( x )) in Eq.”