Learning Latent Space Energy-Based Prior Model

By Bo Pang, Tian Han, Erik Nijkamp, Song-Chun Zhu, Ying Nian Wu. 2020

146 citations recorded by the source corpus.

The generator model assumes that the observed example is generated by a low-dimensional latent vector via a top-down network, and the latent vector follows a simple and known prior distribution, such as uniform or Gaussian white noise distribution. While we can learn an expressive top-down network to map the prior distribution to the data distribution, we can also learn an expressive prior model instead of assuming a given prior distribution. This follows the philosophy of empirical Bayes where the prior model is learned from the observed data. We propose to learn an energy-based prior model for the latent vector, where the energy function is parametrized by a very simple multi-layer perceptron. Due to the low-dimensionality of the latent space, learning a latent space energy-based prior model proves to be both feasible and desirable. In this paper, we develop the maximum likelihood learning algorithm and its variation based on short-run Markov chain Monte Carlo sampling from the prior and the posterior distributions of the latent vector, and we show that the learned model exhibits strong performance in terms of image and text generation and anomaly detection.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, background): “Energy-based models [25–33] have gained signiﬁcant attention in recent years.”
Score-based Generative Modeling in Latent Space (2021, background): “For example, Normalizing flows and hierarchical distributions [23, 24, 47, 48, 20, 21], as well as energy-based models [49, 50, 51, 52, 53] have been proposed as VAE priors.”
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, methodology): “Pang et al. (2020) and Vahdat et al. (2018b;a; 2020) use EBMs for the prior distribution, and (Han et al., 2020; 2019) jointly learn a VAE and an EBM with independent sets of parameters by an adversarial game.”
COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics (2022, unknown): “, 2021), latent features (Pang et al., 2020), and audio sequences (Jayaram & Thickstun, 2021).”
Unpaired Image-to-Image Translation via Latent Energy Transport (2020, methodology): “[30] also provides a promising method to learn a generator and the latent EBM cooperatively.”
Controllable and Compositional Generation with Latent-Space Energy-Based Models (2021, methodology): “More similar to our work, [60, 40] considers using the conditional EBMs as a latent prior (conditioning on labels or attributes) of a generator for controllable generation.”
NCP-VAE: Variational Autoencoders with Noise Contrastive Priors (2020, methodology): “Note how reconstruction FID is much lower than our FID, indicating that our (Vahdat & Kautz, 2020) 13.48 RAE (Ghosh et al., 2020) 40.95 2s-VAE (Dai & Wipf, 2018) 44.4 WAE (Tolstikhin et al., 2018) 35 Perceptial AE(Zhang et al., 2020) 13.8 Latent EBM (Pang et al., 2020) 37.87 COCO-GAN (Lin et al., 20”