Fast Decoding in Sequence Models using Discrete Latent Variables

By Łukasz Kaiser, Aurko Roy, Ashish Vaswani, Niki Parmar, Samy Bengio, Jakob Uszkoreit, Noam Shazeer. 2018

255 citations recorded by the source corpus.

Autoregressive sequence models based on deep neural networks, such as RNNs, Wavenet and the Transformer attain state-of-the-art results on many tasks. However, they are difficult to parallelize and are thus slow at processing long sequences. RNNs lack parallelism both during training and decoding, while architectures like WaveNet and Transformer are much more parallelizable during training, yet still operate sequentially during decoding. Inspired by [arXiv:1711.00937], we present a method to extend sequence models using discrete latent variables that makes decoding much more parallelizable. We first auto-encode the target sequence into a shorter sequence of discrete latent variables, which at inference time is generated autoregressively, and finally decode the output sequence from this shorter latent sequence in parallel. To this end, we introduce a novel method for constructing a sequence of discrete latent variables and compare it with previously introduced methods. Finally, we evaluate our model end-to-end on the task of neural machine translation, where it is an order of magnitude faster at decoding than comparable autoregressive models. While lower in BLEU than purely autoregressive models, our model achieves higher scores than previously proposed non-autoregressive translation models.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “…have pointed to natural language as a means for controlling these multi-task systems (McCann et al., 2018; Radford et al., 2019; Keskar et al., 2019), and several point to the beneﬁts of a code book either speciﬁed explicitly (Wu et al., 2016) or learned in a latent space (Kaiser et al., 2018).”
Blockwise Parallel Decoding for Deep Autoregressive Models (2018, neutral): “Our code is publicly available in the open-source Tensor2Tensor library (Vaswani et al., 2018).”
ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech (2018, methodology): “Similar techniques are applied in non-autoregressive models for machine translation (Gu et al., 2018; Kaiser et al., 2018; Lee et al., 2018; Roy et al., 2018).”
Speech-to-Speech Translation Between Untranscribed Unknown Languages (2019, methodology): “With an EMA update rule for training codebook E, the model has a more stable result during the training process and avoids the posterior collapse issue [19].”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, methodology): “To make the decoding parallelizable, non-autoregressive models based on Transformer have been proposed to generate all the tokens simultaneously [55, 59].”
Mixture Models for Diverse Machine Translation: Tricks of the Trade (2019, methodology): “Kaiser et al. (2018) proposed a similar model, but with groups of discrete multinomial latent variables.”
Levenshtein Transformer (2019, background): “If not better, recent nonautoregressive approaches (Gu et al., 2018; Kaiser et al., 2018; Lee et al., 2018) have proved it possible to perform generation within a much smaller number of decoding iterations.”
Understanding Knowledge Distillation in Non-autoregressive Machine Translation (2019, background): “…outputs (Lee et al., 2018; Ghazvininejad et al., 2019; Wang et al., 2018; Stern et al., 2018; 2019; Gu et al., 2019); (2) using latent variables (Kaiser et al., 2018; Ma et al., 2019; Shu et al., 2019) or structured information such as syntax trees (Akoury et al., 2019) to capture translation…”
Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade (2020, methodology): “…are typically predicted from the inputs x 1: T (cid:48) : where z can be pre-deﬁned by external library (e.g. fertility in Gu et al. (2018a)), or jointly optimized with the NAT model using normalizing ﬂow (Ma et al., 2019) or variational auto-encoders (VAEs) (Kaiser et al., 2018; Shu et al., 2020).”
Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge (2020, background): “Another common issue with vector quantization is codebook collapse – where only a few codes are ever selected [26, 31].”
Parallel Iterative Edit Models for Local Sequence Transduction (2019, methodology): “lel models have also been explored in tasks like translation (Stern et al., 2018; Lee et al., 2018; Gu et al., 2018; Kaiser et al., 2018) and speech synthesis (van den Oord et al.”
You Look Twice: GaterNet for Dynamic Filter Selection in CNNs (2018, methodology): “In this paper, we adopt a method called Improved SemHash [17, 18].”
Latent Normalizing Flows for Discrete Sequences (2019, methodology): “In an alternative approach for faster neural machine translation, Kaiser et al. (2018) propose to use a discrete latent space of variable but reduced size (e.g. 8x fewer tokens than the length of the sentence).”
Unsupervised Acoustic Unit Representation Learning for Voice Conversion using WaveNet Auto-encoders (2020, methodology): “More specifically, we propose to use sliced vector quantization module [26], aiming to increase the complexity of the discrete latent space.”
Discrete Flows: Invertible Generative Models of Discrete Data (2019, background): “Gu et al. (2018) and Kaiser et al. (2018) apply latent variable models for fast translation, where the prior is autoregressive and the decoder is conditionally independent.”