Flow Contrastive Estimation of Energy-Based Models

By Ruiqi Gao, Erik Nijkamp, Diederik P. Kingma, Zhen Xu, Andrew M. Dai, Ying Nian Wu. 2019

129 citations recorded by the source corpus.

This paper studies a training method to jointly estimate an energy-based model and a flow-based model, in which the two models are iteratively updated based on a shared adversarial value function. This joint training method has the following traits. (1) The update of the energy-based model is based on noise contrastive estimation, with the flow model serving as a strong noise distribution. (2) The update of the flow model approximately minimizes the Jensen-Shannon divergence between the flow model and the data distribution. (3) Unlike generative adversarial networks (GAN) which estimates an implicit probability distribution defined by a generator model, our method estimates two explicit probabilistic distributions on the data. Using the proposed method we demonstrate a significant improvement on the synthesis quality of the flow model, and show the effectiveness of unsupervised feature learning by the learned energy-based model. Furthermore, the proposed training method can be easily adapted to semi-supervised learning. We achieve competitive results to the state-of-the-art semi-supervised learning methods.

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

In Signal Processing

Citations in the corpus

Denoising Diffusion Probabilistic Models (2020, neutral): “By the known connection between score matching and energy-based modeling, our work could have implications for other recent work on energy-based models [67–69, 12, 70, 13, 11, 41, 17, 8].”
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, background): “…fundamental maximum likelihood training, other techniques to train EBMs exist, such as minimizing F-divergence (Yu et al., 2020a) or Stein discrepancy (Grathwohl et al., 2020b), contrastive estimation (Gutmann & Hyvärinen, 2010; Gao et al., 2020) and denoising score matching (Li et al., 2019).”
Learning Energy-Based Models by Diffusion Recovery Likelihood (2020, background): “EBMs [27, 33, 21, 47, 10, 43, 6, 7, 25, 35, 5, 11, 4, 8, 2, 12] are an appealing class of probabilistic models, which can be viewed as generative versions of discriminators [18, 26, 28, 12], yet can be learned from unlabeled data.”
How to Train Your Energy-Based Model for Regression (2020, methodology): “EBMs have therefore become increasingly popular within computer vision in recent years, commonly being applied for various generative image modeling tasks [9, 11, 12, 13, 39, 40, 55].”
NCP-VAE: Variational Autoencoders with Noise Contrastive Priors (2020, methodology): “NCE (Gutmann & Hyvärinen, 2010) has recently been used for training a normalizing flow on data distributions (Gao et al., 2020).”
How to Train Your Energy-Based Models (2021, methodology): “Many methods have been proposed to automatically tune the noise distribution, such as Adversarial Contrastive Estimation (Bose et al., 2018), Conditional NCE (Ceylan and Gutmann, 2018) and Flow Contrastive Estimation (Gao et al., 2020).”
Learning Energy-based Model with Flow-based Backbone by Neural Transport MCMC (2020, background): “Even if we can learn the model by other means without resorting to MCMC sampling, e.g., by noise contrastive estimation (NCE) [19, 14], it is still necessary to be able to draw fair examples from the learned model for the purpose of model checking or some downstream applications based on the…”
Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling (2020, methodology): “These include improvements to MCMC-based training (Nijkamp et al., 2019a), Score Matching (Song & Ermon, 2019), and noise-contrastive approaches (Gao et al., 2019).”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, methodology): “This approach can be used to train a correction via exponential tilting [164], but can also be used to directly train an EBM and normalizing flow [54].”
Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models (2020, unknown): “Speciﬁcally, a Langevin dynamics step updates h by In practice, the standard deviation of (cid:15) is often smaller than √ α to allow a faster convergence (Du & Mordatch, 2019; Nijkamp et al., 2020; 2019; Grathwohl et al., 2020a).”
Unpaired Image-to-Image Translation via Latent Energy Transport (2020, background): “Energy-based Model Recently, energy-based generative models have drawn significant attention [40, 39, 9, 29, 7, 11, 43, 41].”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, methodology): “Energy-based Models (EBMs) [26, 48, 58, 64, 67, 68, 75, 78, 79, 120, 129, 132, 165, 170, 182, 196, 254, 281] can be viewed as one generative version of discriminators [79, 104, 131, 134], while can be learned from unlabeled input data.”
Improved Contrastive Divergence Training of Energy Based Models (2020, background): “Additional objectives include noise contrastive estimation (Gao et al., 2020), learned Steins discrepancy (Grathwohl et al.”