DVAE++: Discrete Variational Autoencoders with Overlapping Transformations

By Arash Vahdat, William G. Macready, Zhengbing Bian, Amir Khoshaman. 2018

81 citations recorded by the source corpus.

Training of discrete latent variable models remains challenging because passing gradient information through discrete units is difficult. We propose a new class of smoothing transformations based on a mixture of two overlapping distributions, and show that the proposed transformation can be used for training binary latent models with either directed or undirected priors. We derive a new variational bound to efficiently train with Boltzmann machine priors. Using this bound, we develop DVAE++, a generative model with a global discrete prior and a hierarchy of convolutional continuous variables. Experiments on several benchmarks show that overlapping transformations outperform other recent continuous relaxations of discrete latent variables including Gumbel-Softmax (Maddison et al., 2016; Jang et al., 2016), and discrete variational autoencoders (Rolfe 2016).

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

In Signal Processing

Citations in the corpus

BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, neutral): “49 D ISCRETE VAE++ [54] ≤ 3 .”
Score-based Generative Modeling in Latent Space (2021, background): “For example, Normalizing flows and hierarchical distributions [23, 24, 47, 48, 20, 21], as well as energy-based models [49, 50, 51, 52, 53] have been proposed as VAE priors.”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “…as reducing the gap between approximate and true posterior distributions [3, 4, 5, 6, 7, 8, 9, 10], formulating tighter bounds [11, 12, 13, 14], reducing the gradient noise [15, 16], extending VAEs to discrete variables [17, 18, 19, 20, 21, 22, 23], or tackling posterior collapse [24, 25, 26, 27].”
PixelVAE++: Improved PixelVAE with Discrete Prior (2019, methodology): “Using RBM as a prior in a generative model has been pioneered by [42] and later improved in [40, 43, 44, 45].”
Undirected Graphical Models as Approximate Posteriors (2019, methodology): “…recent methods for discrete variables have focused on REINFORCE with control variates (Mnih & Gregor, 2014; Gu et al., 2015; Mnih & Rezende, 2016; Tucker et al., 2017; Grathwohl et al., 2018) or continuous relaxations (Maddison et al., 2016; Jang et al., 2016; Rolfe, 2016; Vahdat et al., 2018b;a).”
Handling Incomplete Heterogeneous Data using VAEs (2018, background): “(2018), DVAE++ (Vahdat et al., 2018), Shape Variational Autoencoder (Nash and Williams, 2017) and GAN-style objective functions (f-GAN (Nowozin et al.”
NCP-VAE: Variational Autoencoders with Noise Contrastive Priors (2020, methodology): “These EBMs have proven effective for representing the prior in discrete VAEs (Rolfe, 2016; Vahdat et al., 2018a;b; 2020).”
Controllable and Compositional Generation with Latent-Space Energy-Based Models (2021, background): “Also, many other works have built EBMs or score-based models in the latent space of generative models, most of which, however, focus on improving generation quality with EBMs as a structural prior [2, 4, 19, 56, 55, 58, 53].”
QARV: Quantization-Aware ResNet VAE for Lossy Image Compression (2023, background): “Discrete VAEs (DVAEs) [40], [41], [42] assume Bernoulli latent variables, and Vector Quantized VAEs (VQ-VAEs) [43], [44], [45] assume categorical latent variables.”
DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors (2018, methodology): “Unfortunately, the overlapping transformations introduced in DVAE++ [20] are limited to mixtures of exponential or logistic distributions where the inverse CDF can be computed analytically.”
Direct Optimization through arg max for Discrete Variational Auto-Encoder (2019, background): “Rolfe et al. [29] and Vahdat et al. [35, 34] represent the marginal distribution per binary latent variable with a continuous variable in the unit interval.”