Discrete Variational Autoencoders

By Jason Tyler Rolfe. 2016

291 citations recorded by the source corpus.

Probabilistic models with discrete latent variables naturally capture datasets composed of discrete classes. However, they are difficult to train efficiently, since backpropagation through discrete variables is generally not possible. We present a novel method to train a class of probabilistic models with discrete latent variables using the variational autoencoder framework, including backpropagation through the discrete latent variables. The associated class of probabilistic models comprises an undirected discrete component and a directed hierarchical continuous component. The discrete component captures the distribution over the disconnected smooth manifolds induced by the continuous component. As a result, this class of models efficiently learns both the class of objects in an image, and their specific realization in pixels, from unsupervised data, and outperforms state-of-the-art methods on the permutation-invariant MNIST, Omniglot, and Caltech-101 Silhouettes datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, background): “03 Without autoregressive components DISCRETE VAE [42] ≤ 81.”
Score-based Generative Modeling in Latent Space (2021, background): “For example, Normalizing flows and hierarchical distributions [23, 24, 47, 48, 20, 21], as well as energy-based models [49, 50, 51, 52, 53] have been proposed as VAE priors.”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “…as reducing the gap between approximate and true posterior distributions [3, 4, 5, 6, 7, 8, 9, 10], formulating tighter bounds [11, 12, 13, 14], reducing the gradient noise [15, 16], extending VAEs to discrete variables [17, 18, 19, 20, 21, 22, 23], or tackling posterior collapse [24, 25, 26, 27].”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “We use pre-trained DALL-E [27] as our dVAE to generate discrete visual tokens for MVM.”
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, methodology): “Early work like discrete Variational Auto-Encoders (dVAEs) [5] optimizes a probabilistic model with discrete latent variables to capture datasets composed of discrete classes.”
Variational Lossy Autoencoder (2016, positive): “Hence we use the best performing model from Kingma et al. (Rezende & Mohamed, 2015) 85.10 DRAW (Gregor et al., 2015) < 80.97 Discrete VAE (Rolfe, 2016) 81.01 PixelRNN (van den Oord et al., 2016a) 79.20 IAF VAE (Kingma et al., 2016) 79 In addition, we hypothesize that the separation of different type”
Learning Sparse Neural Networks through L0 Regularization (2017, methodology): “Another technique that allows for gradient based optimization of discrete r.v.s are the smoothing transformations proposed by Rolfe (2016).”
DVAE++: Discrete Variational Autoencoders with Overlapping Transformations (2018, methodology): “For a fair comparison, we apply only those techniques that were also used in (Rolfe, 2016).”
PixelVAE++: Improved PixelVAE with Discrete Prior (2019, methodology): “Using RBM as a prior in a generative model has been pioneered by [42] and later improved in [40, 43, 44, 45].”
QARV: Quantization-Aware ResNet VAE for Lossy Image Compression (2023, background): “Discrete VAEs (DVAEs) [40], [41], [42] assume Bernoulli latent variables, and Vector Quantized VAEs (VQ-VAEs) [43], [44], [45] assume categorical latent variables.”
Simple and Effective VAE Training with Calibrated Decoders (2020, unknown): “Additionally, different choices for representing such variance exist, including diagonal covariance (Kingma & Welling, 2014; Sønderby et al., 2016; Rolfe, 2016), or a single shared parameter (Kingma et al., 2016; Dai & Wipf, 2019; Edwards & Storkey, 2016; Rezende & Viola, 2018).”
Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training (2022, methodology): “Inspired by this, BEiT [4] proposes to match image patches with discrete tokens via dVAE [40] and pre-train a standard vision transformer [14, 60] by masked image modeling.”
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers (2021, methodology): “To acquire discrete image tokens, iGPT [11] quantized the pixel values using k-means, ViT [23] divide the image into non-overlapping fixed-size patches, and BEiT [2] tokenize the image to discrete visual tokens via discrete VAE [51, 57].”
An Image is Worth 32 Tokens for Reconstruction and Generation (2024, unknown): “Therefore, our method is significantly different from theirs, since the proposed TiTok aims to reconstruct both the high-level and low-level details of an image, same as typical VQ-VAE tokenizers [34, 54, 19].”