Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement

By Jason Lee, Elman Mansimov, Kyunghyun Cho. 2018

498 citations recorded by the source corpus.

We propose a conditional non-autoregressive neural sequence model based on iterative refinement. The proposed model is designed based on the principles of latent variable models and denoising autoencoders, and is generally applicable to any sequence generation task. We extensively evaluate the proposed model on machine translation (En-De and En-Ro) and image caption generation, and observe that it significantly speeds up decoding while maintaining the generation quality comparable to the autoregressive counterpart.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

KERMIT: Generative Insertion-Based Modeling for Sequences (2019, methodology): “We also follow prior work (Gu et al., 2018; Stern et al., 2018, 2019; Lee et al., 2018) in using distillation (Hinton et al., 2015; Kim and Rush, 2016) to train our models.”
Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, background): “, 2019c) or a constant number of generation steps (Lee et al., 2018).”
Mask-Predict: Parallel Decoding of Conditional Masked Language Models (2019, methodology): “Mask-predict repeatedly masks out and re-predicts the subset of words in the current translation that the model is least conﬁdent about, in contrast to recent parallel decoding translation approaches that repeatedly predict the entire sequence (Lee et al., 2018).”
WaveGrad: Estimating Gradients for Waveform Generation (2020, methodology): “Edit-based models can also iteratively refine the outputs during inference (Lee et al., 2018; Ghazvininejad et al., 2019; Chan et al., 2020), while they do not rely on a (continuous) gradient-based sampler, they rely on a (discrete) edit-based sampler.”
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020, background): “Recent advances (Lee et al., 2018; Stern et al., 2019) in conditional sentence generation have an emerging interest on (edit-based) non-autoregressive generation models, which iteratively reﬁne generated sequences.”
ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech (2018, methodology): “Similar techniques are applied in non-autoregressive models for machine translation (Gu et al., 2018; Kaiser et al., 2018; Lee et al., 2018; Roy et al., 2018).”
Going deeper with Image Transformers (2021, background): “Note, transforming only the class token with frozen trunk embeddings in CaiT is reminiscent of nonautoregressive encoder-decoders [25, 41], where a whole sequence (we have only one prediction) is produced at once by iterative refinements.”
Insertion Transformer: Flexible Sequence Generation via Insertion Operations (2019, methodology): “Lee et al. (2018) adds an iterative reﬁnement stage to the decoder in which a new hypothesis is produced conditioning on the input and the previous output.”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, methodology): “onditioned on the words that are previously generated. To make the decoding parallelizable, non-autoregressive models based on Transformer have been proposed to generate all the tokens simultaneously [55, 59]. Nonautoregressive modeling factorizes the distribution over a target sequence given a quer”
Learning to Efficiently Sample from Diffusion Probabilistic Models (2021, background): “…very closely related to DDPMs, there has also been work on non-autoregressive modeling of text sequences that can be regarded as discrete-space DDPMs with a forward process that masks or remove tokens [Lee et al., 2018, Gu et al., 2019, Stern et al., 2019, Chan et al., 2020, Saharia et al., 2020].”
Insertion-based Decoding with Automatically Inferred Generation Order (2019, methodology): “For the NL2Code task, We use the Django dataset (Oda et al., 2015)5 and the MS COCO (Lin et al., 2014) with the standard split (Karpathy and Fei-Fei, 2015) for the NL2Code task and image captioning, respectively.”
Parallel Tacotron: Non-Autoregressive and Controllable TTS (2020, methodology): “This type of iterative loss has previously been used in [32, 34].”
Levenshtein Transformer (2019, methodology): “We follow closely a standard pipeline established by Lee et al. (2018). Table 2b shows this comparison.”
Blank Language Models (2020, background): “Another stream of work focuses on generating sequences in a non-autoregressive fashion for fast decoding in machine translation (Gu et al., 2017; Lee et al., 2018; Stern et al., 2019; Gu et al., 2019b).”