Training Deep Energy-Based Models with f-Divergence Minimization

By Lantao Yu, Yang Song, Jiaming Song, Stefano Ermon. 2020

51 citations recorded by the source corpus.

Deep energy-based models (EBMs) are very flexible in distribution parametrization but computationally challenging because of the intractable partition function. They are typically trained via maximum likelihood, using contrastive divergence to approximate the gradient of the KL divergence between data and model distribution. While KL divergence has many desirable properties, other f-divergences have shown advantages in training implicit density generative models such as generative adversarial networks. In this paper, we propose a general variational framework termed f-EBM to train EBMs using any desired f-divergence. We introduce a corresponding optimization algorithm and prove its local convergence property with non-linear dynamical systems theory. Experimental results demonstrate the superiority of f-EBM over contrastive divergence, as well as the benefits of training EBMs using f-divergences other than KL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, methodology): “Besides fundamental maximum likelihood training, other techniques to train EBMs exist, such as minimizing F-divergence (Yu et al., 2020a) or Stein discrepancy (Grathwohl et al., 2020b), contrastive estimation (Gutmann & Hyvärinen, 2010; Gao et al., 2020) and denoising score matching (Li et al.,…”
Unpaired Image-to-Image Translation via Latent Energy Transport (2020, background): “Energy-based Model Recently, energy-based generative models have drawn significant attention [40, 39, 9, 29, 7, 11, 43, 41].”
Generalized Energy Based Models (2020, methodology): “Yu et al. (2020) explore methods for using general f-divergences, such as Jensen-Shannon, to train EBMs.”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “Training EBMs with arbitrary f-divergences is possible, yielding improved FID scores [250].”
How to Train Your Energy-Based Models (2021, methodology): “When using both adversarial training and MCMC sampling, Yu et al. (2020) observe that EBMs can be trained with an arbitrary f -divergence, including KL, reverse KL, total variation, Hellinger, etc .”
Improved Contrastive Divergence Training of Energy Based Models (2020, background): “, 2020b), and learned F divergences (Yu et al., 2020).”