Non-Autoregressive Machine Translation with Latent Alignments

By Chitwan Saharia, William Chan, Saurabh Saxena, Mohammad Norouzi. 2020

169 citations recorded by the source corpus.

This paper investigates two latent alignment models for non-autoregressive machine translation, namely CTC and Imputer. CTC generates outputs in a single step, makes strong conditional independence assumptions about output variables, and marginalizes out latent alignments using dynamic programming. Imputer generates outputs in a constant number of steps, and approximately marginalizes out possible generation orders and latent alignments for training. These models are simpler than existing non-autoregressive methods, since they do not require output length prediction as a pre-process. In addition, our architecture is simpler than typical encoder-decoder architectures, since input-output cross attention is not used. On the competitive WMT'14 En$\rightarrow$De task, our CTC model achieves 25.7 BLEU with a single generation step, while Imputer achieves 27.5 BLEU with 2 generation steps, and 28.0 BLEU with 4 generation steps. This compares favourably to the baseline autoregressive Transformer with 27.8 BLEU.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

WaveGrad: Estimating Gradients for Waveform Generation (2020, methodology): “Their work also relies heavily on a noise distribution, for example Bernoulli (Devlin et al., 2019), uniform (Saharia et al., 2020), or even hand-crafted (Chan et al., 2020) to provide support for learning the edit distribution, while in our work we rely on a Markov chain from the diffusion…”
Learning to Efficiently Sample from Diffusion Probabilistic Models (2021, background): “…very closely related to DDPMs, there has also been work on non-autoregressive modeling of text sequences that can be regarded as discrete-space DDPMs with a forward process that masks or remove tokens [Lee et al., 2018, Gu et al., 2019, Stern et al., 2019, Chan et al., 2020, Saharia et al., 2020].”
Diffusion-LM Improves Controllable Text Generation (2022, background): “There are other non-autoregressive LMs that have been developed for machine translation and speech-to-text tasks [12, 43].”
Structured Denoising Diffusion Models in Discrete State-Spaces (2021, background): “Other works have considered non-autoregressive translation and speech transcription via insertion and deletion [16, 33], masking [13], and iteratively-refined sequence alignments [5, 34].”
Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade (2020, methodology): “To ease the modeling difficulty, recent state-ofthe-art NAT systems (Lee et al., 2018; Stern et al., 2019; Ghazvininejad et al., 2019; Gu et al., 2019; Kasai et al., 2020a; Shu et al., 2020; Saharia et al., 2020) trade accuracy with latency by incorporating”
Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation (2022, background): “…to improve the performance of NAR models, including training with alignment-based objectives (Libovický and Helcl, 2018; Ghazvininejad et al., 2020; Saharia et al., 2020; Gu and Kong, 2021; Shao and Feng, 2022), modeling dependencies between target to-kens (Ghazvininejad et al., 2019; Shu et al.,…”
Step-unrolled Denoising Autoencoders for Text Generation (2021, methodology): “Finally, a recent line of work called Imputer (Chan et al., 2020; Saharia et al., 2020) achieves strong results in machine translation by aligning target to source via dynamic programming.”
Continuous diffusion for categorical data (2022, background): “More recently, Imputer (Chan et al., 2020; Saharia et al., 2020) achieved good results by optimizing alignment between source and target.”
ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation (2020, methodology): “finements (Lee et al., 2018; Ghazvininejad et al., 2019; Saharia et al., 2020) and rescoring with autoregressive models (Kaiser et al.”
SmBoP: Semi-autoregressive Bottom-up Semantic Parsing (2020, methodology): “For each example, we ﬁnd the ﬁrst gold tree that is dropped from the beam (if there is more than one, we choose one randomly).”
Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation (2020, methodology): “Overall, our AR deep-shallow models outperform most NAR models, with the only exception being EN → RO where it underperforms Imputer by 0.6 BLEU points.”
Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation (2020, methodology): “Distillation Following previous work on non-autoregressive translation (e.g., Ghazvininejad et al., 2019; Kasai et al., 2020; Saharia et al., 2020), we apply sequence-level knowledge distillation (Kim and Rush, 2016) by training every model in all directions on translations produced by a standard…”
Lossless Acceleration for Seq2seq Generation with Aggressive Decoding (2022, methodology): “Recent work mainly focused on improving the quality while maintaining competitive speedups, including applying various training objectives (Ghazvininejad et al., 2020a; Saharia et al., 2020; Du et al., 2021; Huang et al., 2021), modeling dependencies between target tokens (Ghazvininejad et al.”