Neural Discrete Representation Learning
By Aäron van den Oord, Oriol Vinyals, Koray Kavukcuoglu. 2017
7,634 citations recorded by the source corpus.
Learning useful representations without supervision remains a key challenge in machine learning. In this paper, we propose a simple yet powerful generative model that learns such discrete representations. Our model, the Vector Quantised-Variational AutoEncoder (VQ-VAE), differs from VAEs in two key ways: the encoder network outputs discrete, rather than continuous, codes; and the prior is learnt rather than static. In order to learn a discrete latent representation, we incorporate ideas from vector quantisation (VQ). Using the VQ method allows the model to circumvent issues of "posterior collapse" -- where the latents are ignored when they are paired with a powerful autoregressive decoder -- typically observed in the VAE framework. Pairing these representations with an autoregressive prior, the model can generate high quality images, videos, and speech as well as doing high quality speaker conversion and unsupervised learning of phonemes, providing further evidence of the utility of the learnt representations.
Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Jianfeng Gao
- Quoc V. Le
- Sergey Levine
- Percy Liang
- Luke S. Zettlemoyer
- Ruslan Salakhutdinov
In Representation & Self-Supervised Learning
- Yoshua Bengio
- James R. Glass
- Trevor Darrell
- Dacheng Tao
- Sergey Levine
- Yu-An Chung
- Wei-Ning Hsu
- Ruslan Salakhutdinov
In Signal Processing
Citations in the corpus
- Masked Autoencoders Are Scalable Vision Learners (2021, background): “Most recently, BEiT [2] proposes to predict discrete tokens [37, 43].”
- High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “Following common practice [11, 23, 64, 65, 93], we separate training into two distinct phases: First, we train an autoencoder which provides a lower-dimensional (and thereby efficient) representational space which is perceptually equivalent to the data space.”
- Jukebox: A Generative Model for Music (2020, methodology): “This approach has demonstrated some success in generating short instrumental pieces restricted to a set of a few instruments (Oord et al., 2017; Dieleman et al., 2018).”
- Zero-Shot Text-to-Image Generation (2021, methodology): “Oord et al. (2017); Razavi et al. (2019) address this using an online cluster assignment procedure coupled with the straight-through estimator (Bengio et al., 2013).”
- VQ-DRAW: A Sequential Discrete VAE (2020, methodology): “Like VQ-VAE, VQ-DRAW produces deterministic discrete latent codes and assumes a uniform prior.”
- Predictive Sampling with Forecasting Autoregressive Models (2020, methodology): “Following van den Oord et al. (2017), we separate the training of autoencoder and ARM.”
- Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, methodology): “The VQ-VAE model [41] can be better understood as a communication system.”
- Autoregressive Quantile Networks for Generative Modeling (2018, methodology): “Second, the VQ-VAE model (van den Oord et al., 2017), learns a PixelCNN model of the (discretized) latent space.”
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “Consequently, recent approaches either compress the image to a discrete representation [13, 59] via Vector Quantized (VQ) VAEs [59], or down-sample the image resolution [11, 21].”
- Towards Language-Free Training for Text-to-Image Generation (2021, methodology): “The basic idea is to encode images into discrete latent tokens using VQ-VAE [39, 45], and pre-train a huge-size auto-regressive Transformers [46] to predict these discrete tokens based on paired text sequences.”
- Vector-quantized Image Modeling with Improved VQGAN (2021, methodology): “Remarkable image generation results have been achieved by pre-quantizing images into discrete latent variables and modeling them autoregressively, including VQVAE (Oord et al., 2017), DALLE (Ramesh et al.”
- CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders (2021, neutral): “…have enabled considerable progress towards photorealistic image synthesis, strong autoregressive models have achieved similar quality results (Oord et al., 2017; Chen et al., 2020), with the recent DALL-E model (Ramesh et al., 2021) showcasing the benefit of scaling text-to-image synthesis…”
- D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Other methods adopt a “two-stage” approach [23], which fits a generative model over the latent space of autoencoders [96, 75, 26, 74].”
- Score-based Generative Modeling in Latent Space (2021, methodology): “In two-stage training, a separate generative model is trained in latent space as a new prior after training the VAE itself [60, 61, 62, 63, 64, 10].”
- Cascaded Diffusion Models for High Fidelity Image Generation (2021, background): “Diffusion models [24] have recently been shown to synthesize high quality images and audio [3, 10, 14, 27], which have long been led by other classes of generative models such as autoregressive models, GANs, VAEs, and flows [2, 5, 7, 9, 12, 13, 18, 29, 30, 32].”
- CogView: Mastering Text-to-Image Generation via Transformers (2021, methodology): “The framework of Vector Quantized Variational AutoEncoders (VQ-VAE) [46] alleviates this problem.”