DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors

By Arash Vahdat, Evgeny Andriyash, William G. Macready. 2018

56 citations recorded by the source corpus.

Boltzmann machines are powerful distributions that have been shown to be an effective prior over binary latent variables in variational autoencoders (VAEs). However, previous methods for training discrete VAEs have used the evidence lower bound and not the tighter importance-weighted bound. We propose two approaches for relaxing Boltzmann machines to continuous distributions that permit training with importance-weighted bounds. These relaxations are based on generalized overlapping transformations and the Gaussian integral trick. Experiments on the MNIST and OMNIGLOT datasets show that these relaxations outperform previous discrete VAEs with Boltzmann priors. An implementation which reproduces these results is available at this https URL .

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Score-based Generative Modeling in Latent Space (2021, background): “For example, Normalizing flows and hierarchical distributions [23, 24, 47, 48, 20, 21], as well as energy-based models [49, 50, 51, 52, 53] have been proposed as VAE priors.”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “…as reducing the gap between approximate and true posterior distributions [3, 4, 5, 6, 7, 8, 9, 10], formulating tighter bounds [11, 12, 13, 14], reducing the gradient noise [15, 16], extending VAEs to discrete variables [17, 18, 19, 20, 21, 22, 23], or tackling posterior collapse [24, 25, 26, 27].”
PixelVAE++: Improved PixelVAE with Discrete Prior (2019, methodology): “Using RBM as a prior in a generative model has been pioneered by [42] and later improved in [40, 43, 44, 45].”
Undirected Graphical Models as Approximate Posteriors (2019, methodology): “…recent methods for discrete variables have focused on REINFORCE with control variates (Mnih & Gregor, 2014; Gu et al., 2015; Mnih & Rezende, 2016; Tucker et al., 2017; Grathwohl et al., 2018) or continuous relaxations (Maddison et al., 2016; Jang et al., 2016; Rolfe, 2016; Vahdat et al., 2018b;a).”
NCP-VAE: Variational Autoencoders with Noise Contrastive Priors (2020, unknown): “These EBMs have proven effective for representing the prior in discrete VAEs (Rolfe, 2016; Vahdat et al., 2018a;b; 2020).”
Controllable and Compositional Generation with Latent-Space Energy-Based Models (2021, background): “Also, many other works have built EBMs or score-based models in the latent space of generative models, most of which, however, focus on improving generation quality with EBMs as a structural prior [2, 4, 19, 56, 55, 58, 53].”
Direct Optimization through arg max for Discrete Variational Auto-Encoder (2019, background): “Rolfe et al. [29] and Vahdat et al. [35, 34] represent the marginal distribution per binary latent variable with a continuous variable in the unit interval.”
QARV: Quantization-Aware ResNet VAE for Lossy Image Compression (2023, background): “Discrete VAEs (DVAEs) [40], [41], [42] assume Bernoulli latent variables, and Vector Quantized VAEs (VQ-VAEs) [43], [44], [45] assume categorical latent variables.”