Theory and Experiments on Vector Quantized Autoencoders

By Aurko Roy, Ashish Vaswani, Arvind Neelakantan, Niki Parmar. 2018

100 citations recorded by the source corpus.

Deep neural networks with discrete latent variables offer the promise of better symbolic reasoning, and learning abstractions that are more useful to new tasks. There has been a surge in interest in discrete latent variable models, however, despite several recent improvements, the training of discrete latent variable models has remained challenging and their performance has mostly failed to match their continuous counterparts. Recent work on vector quantized autoencoders (VQ-VAE) has made substantial progress in this direction, with its perplexity almost matching that of a VAE on datasets such as CIFAR-10. In this work, we investigate an alternate training technique for VQ-VAE, inspired by its connection to the Expectation Maximization (EM) algorithm. Training the discrete bottleneck with EM helps us achieve better image generation results on CIFAR-10, and together with knowledge distillation, allows us to develop a non-autoregressive machine translation model whose accuracy almost matches a strong greedy autoregressive baseline Transformer, while being 3.3 times faster at inference.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Orhan Firat
2. Graham Neubig
3. Tao Qin
4. Tie-Yan Liu
5. Ankur Bapna
6. Yu Zhang
7. Yonghui Wu
8. Xu Tan

In Computer Vision and Pattern Recognition

1. Orhan Firat
2. Luc Van Gool
3. Dacheng Tao
4. Quoc V. Le
5. Ankur Bapna
6. Trevor Darrell
7. Ming-Hsuan Yang
8. Yi Yang

In Representation & Self-Supervised Learning

1. Andros Tjandra
2. Aäron van den Oord
3. Sakriani Watiasri Sakti
4. Satoshi Nakamura
5. Ali Razavi
6. Ben Poole
7. Wei Ping
8. Kainan Peng

In Probabilistic & Bayesian Methods

1. Dustin Tran
2. Rif A. Saurous
3. Srinivas Vasudevan
4. Alexey Radul
5. Matthew D. Hoffman
6. David M. Blei
7. Sakriani Watiasri Sakti
8. Satoshi Nakamura

Citations in the corpus

• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Finally, while scaling up existing approaches is one way to improve model quality, some approaches or architectures are more effi-cient to train (Shazeer et al., 2017; Vaswani et al., 2017; Wu et al., 2019), more sample efficient, or faster at inference (Gu et al., 2017; Roy et al., 2018).”
• ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech (2018, methodology): “Similar techniques are applied in non-autoregressive models for machine translation (Gu et al., 2018; Kaiser et al., 2018; Lee et al., 2018; Roy et al., 2018).”
• Preventing Posterior Collapse with delta-VAEs (2019, methodology): “To address this limitation, we adopt the approaches of van den Oord et al. (2017); Roy et al. (2018) and train an auxiliary prior over the course of learning to match the aggregate posterior, but that does not influence the training of the encoder or decoder.”
• Transformer VQ-VAE for Unsupervised Unit Discovery and Speech Synthesis: ZeroSpeech 2020 Challenge (2020, background): “, learning rate, momentum) and also avoids posterior collapse problem [18].”
• Simple, Distributed, and Accelerated Probabilistic Programming (2018, methodology): “For 64x64 ImageNet, we use a vector-quantized variational auto-encoder trained with soft EM [37].”