VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models

By Zhisheng Xiao, Karsten Kreis, Jan Kautz, Arash Vahdat. 2020

142 citations recorded by the source corpus.

Energy-based models (EBMs) have recently been successful in representing complex distributions of small images. However, sampling from them requires expensive Markov chain Monte Carlo (MCMC) iterations that mix slowly in high dimensional pixel space. Unlike EBMs, variational autoencoders (VAEs) generate samples quickly and are equipped with a latent space that enables fast traversal of the data manifold. However, VAEs tend to assign high probability density to regions in data space outside the actual data distribution and often fail at generating sharp images. In this paper, we propose VAEBM, a symbiotic composition of a VAE and an EBM that offers the best of both worlds. VAEBM captures the overall mode structure of the data distribution using a state-of-the-art VAE and it relies on its EBM component to explicitly exclude non-data-like regions from the model and refine the image samples. Moreover, the VAE component in VAEBM allows us to speed up MCMC updates by reparameterizing them in the VAE's latent space. Our experimental results show that VAEBM outperforms state-of-the-art VAEs and EBMs in generative quality on several benchmark image datasets by a large margin. It can generate high-quality images as large as 256$\times$256 pixels with short MCMC chains. We also demonstrate that VAEBM provides complete mode coverage and performs well in out-of-distribution detection.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, background): “CelebA-HQ results reproduced from [39, 57, 93], FFHQ from [38, 39].”
Score-based Generative Modeling in Latent Space (2021, background): “[76] Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat.”
Symbolic Music Generation with Diffusion Models (2021, methodology): “A related approach has been to refine the output of a VAE (Kingma & Welling, 2014) with an energy-based model (Xiao et al., 2020) or use an additional model to con-”
Taming Transformers for High-Resolution Image Synthesis (2020, neutral): “CelebAHQ results reproduced from [1, 47, 67, 22], FFHQ from [58, 28].”
Energy-based Latent Aligner for Incremental Learning (2022, methodology): “EBMs have also been used for synthesizing images [3,53,57,58].”
Diff-TTS: A Denoising Diffusion Model for Text-to-Speech (2021, methodology): “Diff-TTS was trained with 400 time-step and synthesized speech samples with the decimation factor γ ∈ [1, 7, 21, 57].”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “This limits the freedom of the model, however, leading some to jointly sample in latent and data space [3], [245].”
Tackling the Generative Learning Trilemma with Denoising Diffusion GANs (2021, unknown): “Table 4: Generative results on CelebA-HQ-256 Model FID↓ Denoising Diffusion GAN (ours) 7.64 Score SDE (Song et al., 2021c) 7.23 LSGM (Vahdat et al., 2021) 7.22 UDM (Kim et al., 2021) 7.16 NVAE (Vahdat & Kautz, 2020) 29.7 VAEBM (Xiao et al., 2021) 20.4 NCP-VAE (Aneja et al., 2021) 24.8 PGGAN (Karras ”
Score-Based Generative Modeling with Critically-Damped Langevin Diffusion (2021, background): “…δt′ := δt2 and β′(t) := β(t)2, we obtain xn+1 = xn − β′(t)M−1xnδt′ + √ 2β′(t)δt′M−1N (0, 1), (20) which corresponds to the high-friction overdamped Langevin dynamics that are frequently run, for example, to train energy-based generative models (Du & Mordatch, 2019; Xiao et al., 2021).”
Unpaired Image-to-Image Translation via Latent Energy Transport (2020, result): “This approach has proved successful in the recent proposed VAEBM [38].”
Controllable and Compositional Generation with Latent-Space Energy-Based Models (2021, methodology): “Also, many other works have built EBMs or score-based models in the latent space of generative models, most of which, however, focus on improving generation quality with EBMs as a structural prior [2, 4, 19, 56, 55, 58, 53].”
Perception Prioritized Training of Diffusion Models (2022, unknown): “…VQGAN CVPR’21 [11] GAN+AR 9.6 D2C NeurIPS’21 [38] Diff [20] GAN 64.70 MSG-GAN CVPR’20 [16] [22] Flow 68.93 ALAE CVPR’20 [33] GAN 19.21 NVAE NeurIPS’20 [44] VAE 29.76 VAEBM ICLR’21 [51] VAE+EM 20.38 VQGAN CVPR’21 [11] GAN+AR 10.70 LSGM NeurIPS’21 [45] VAE+Diff 7.22 P2 (500step) Diff 6.91 Table 2.”