Non-Autoregressive Neural Machine Translation

By Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, Richard Socher. 2017

925 citations recorded by the source corpus.

Existing approaches to neural machine translation condition each output word on previously generated outputs. We introduce a model that avoids this autoregressive property and produces its outputs in parallel, allowing an order of magnitude lower latency during inference. Through knowledge distillation, the use of input token fertilities as a latent variable, and policy gradient fine-tuning, we achieve this at a cost of as little as 2.0 BLEU points relative to the autoregressive Transformer network used as a teacher. We demonstrate substantial cumulative improvements associated with each of the three aspects of our training strategy, and validate our approach on IWSLT 2016 English-German and two WMT language pairs. By sampling fertilities in parallel at inference time, our non-autoregressive model achieves near-state-of-the-art performance of 29.8 BLEU on WMT 2016 English-Romanian.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

End-to-End Object Detection with Transformers (2020, background): “s.1. Compared to most previous work on direct set prediction, the main features of DETR are the conjunction of the bipartite matching loss and transformers with (non-autoregressive) parallel decoding [29,12,10,8]. In contrast, previous work focused on autoregressive decoding with RNNs [43,41,30,36,4”
Meta-Learning for Low-Resource Neural Machine Translation (2018, methodology): “We implement Transformer in this paper based on (Gu et al., 2018a) 8 and modify it to use the universal lexical representation from §3.3.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Finally, while scaling up existing approaches is one way to improve model quality, some approaches or architectures are more efﬁ-cient to train (Shazeer et al., 2017; Vaswani et al., 2017; Wu et al., 2019), more sample efﬁcient, or faster at inference (Gu et al., 2017; Roy et al., 2018).”
KERMIT: Generative Insertion-Based Modeling for Sequences (2019, methodology): “We also follow prior work (Gu et al., 2018; Stern et al., 2018, 2019; Lee et al., 2018) in using distillation (Hinton et al., 2015; Kim and Rush, 2016) to train our models.”
Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, methodology): “Non-Autoregressive Transformer (NAT) (Gu et al., 2018) and Mask-Predict (Ghazvininejad et al., 2019) are closely related to our Imputer model.”
Mask-Predict: Parallel Decoding of Conditional Masked Language Models (2019, methodology): “This problem, known as the multi-modality problem (Gu et al., 2018), often manifest as token repetitions in the output when the model has multiple hypotheses that predict the same word w with high conﬁdence, but at different positions.”
Parallel WaveNet: Fast High-Fidelity Speech Synthesis (2017, background): “Recently (Gu et al., 2017) has presented a related idea to train feed-forward networks for neural machine translation.”
Efficient Neural Audio Synthesis (2018, background): “Multiple recent approaches have the goal of making sampling from sequential models more parallel (Reed et al., 2017; Gu et al., 2017; van den Oord et al., 2017).”
Blockwise Parallel Decoding for Deep Autoregressive Models (2018, neutral): “den_] Step 5 2 tokens [Weltraum, _] Step 6 3 tokens [gestartet_, ._, <EOS>] 8 Wall-Clock Model Source BLEU Speedup Transformer (beam size 4) Vaswani et al. (2017) 28.4 Transformer (beam size 1) Gu et al. (2018) 22.71 Transformer (beam size 4) Gu et al. (2018) 23.45 Non-autoregressive Transform”
MaskGIT: Masked Generative Image Transformer (2022, methodology): “Our work is inspired by bi-directional machine translation [16, 19, 20] in NLP, and our novelty lies in the proposed new masking strategy and decoding algorithm which, as substantiated by our experiments, are essential for image generation.”
SoundStorm: Efficient Parallel Audio Generation (2023, neutral): “In order to improve the inference time and to allow bidirectional non-causal attention on the input sequence, parallel decoding schemes have been proposed for text (Gu et al., 2017; Ghazvininejad et al., 2019), im-SoundStorm Figure 1.”
ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech (2018, methodology): “Similar techniques are applied in non-autoregressive models for machine translation (Gu et al., 2018; Kaiser et al., 2018; Lee et al., 2018; Roy et al., 2018).”
Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search (2020, background): “In natural language processing, Gu et al. (2017) tackle the challenges by using an external aligner to segment output tokens according to each input token.”
Going deeper with Image Transformers (2021, background): “Note, transforming only the class token with frozen trunk embeddings in CaiT is reminiscent of non-autoregressive encoder-decoders [25, 41], where a whole sequence (we have only one prediction) is produced at once by iterative refinements.”
Lifelong Language Knowledge Distillation (2020, methodology): “…output sequence ˆ x from the teacher model as the hard target, just like normal language modeling: Seq-KD is usually applied for improving weak non-autoregressive translation (NAT) models (Zhou et al., 2020) by reducing the multi-modality problem in machine translation datasets (Gu et al., 2018).”
Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement (2018, methodology): “Similarly to two recent works (Oord et al., 2017; Gu et al., 2017), we introduce latent variables to implicitly capture the dependencies among target variables.”
Insertion Transformer: Flexible Sequence Generation via Insertion Operations (2019, methodology): “More recently, there has been work on non-autoregressive sequence models such as the Non-Autoregressive Trans-former (NAT) (Gu et al., 2018) and the Iterative Reﬁnement model (Lee et al., 2018).”
SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation (2018, neutral): “(w.r.t. Luong and Manning (2015); Gu et al. (2018); Vaswani et al. (2017)).”
Fast Decoding in Sequence Models using Discrete Latent Variables (2018, methodology): “For comparison, we include the baselines from (Gu et al., 2017).”