Iterative Refinement for Machine Translation

By Roman Novak, Michael Auli, David Grangier. 2016

29 citations recorded by the source corpus.

Existing machine translation decoding algorithms generate translations in a strictly monotonic fashion and never revisit previous decisions. As a result, earlier mistakes cannot be corrected at a later stage. In this paper, we present a translation scheme that starts from an initial guess and then makes iterative improvements that may revisit previous decisions. We parameterize our model as a convolutional neural network that predicts discrete substitutions to an existing translation based on an attention mechanism over both the source sentence as well as the current translation output. By making less than one modification per sentence, we improve the output of a phrase-based translation system by up to 0.4 BLEU on WMT15 German-English translation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Retrieve and Refine: Improved Sequence Generation Models For Dialogue (2018, methodology): “…have been used in MT and summarization either for refinement of initial predictions (Junczys-Dowmunt and Grundkiewicz, 2017; Niehues et al., 2016; Novak et al., 2016; Xia et al., 2017; Grangier and Auli, 2017) or combining with retrieval (Gu et al., 2017; Cao et al., 2018), as well as for…”
Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement (2018, background): “Non-Autoregressive Neural Machine Translation Schwenk (2012) proposed a continuous-space translation model to estimate the conditional distribution over a target phrase given a source phrase, while dropping the conditional dependencies among target tokens.”
Levenshtein Transformer (2019, neutral): “For instance, Novak et al. (2016) predict and apply token substitutions iteratively on phase-based MT system outputs using convolutional neural network.”
Neural Networks for Modeling Source Code Edits (2018, background): “The structured objects could be images (Gregor et al., 2015; Denton et al., 2015; Karras et al., 2018), translations (Li et al., 2017; Novak et al., 2016; Niehues et al., 2016; Lee et al., 2018), or source code (Gupta et al., 2017; Shin et al., 2018).”
Latent Alignment and Variational Attention (2018, methodology): “While q is not used directly in left-to-right NMT decoding, it could be employed for other applications such as in an iterative refinement approach [56, 42].”
QuickEdit: Editing Text & Translations by Crossing Words Out (2017, background): “Dual attention has been introduced recently in the context of automatic post-editing (Novak et al., 2016; Libovickỳ and Helcl, 2017).”
Progressive Generation of Long Text with Pretrained Language Models (2021, background): “, 2020), or refinement (Lee et al., 2018; Novak et al., 2016; Mansimov et al., 2019; Kasai et al., 2020) that differs from the restricted left-toright generation in conventional LMs.”