An Empirical Study of Generation Order for Machine Translation

By William Chan, Mitchell Stern, Jamie Ryan Kiros, Jakob Uszkoreit. 2019

10 citations recorded by the source corpus.

In this work, we present an empirical study of generation order for machine translation. Building on recent advances in insertion-based modeling, we first introduce a soft order-reward framework that enables us to train models to follow arbitrary oracle generation policies. We then make use of this framework to explore a large variety of generation orders, including uninformed orders, location-based orders, frequency-based orders, content-based orders, and model-based orders. Curiously, we find that for the WMT'14 English $\to$ German translation task, order does not have a substantial impact on output quality, with unintuitive orderings such as alphabetical and shortest-first matching the performance of a standard Transformer. This demonstrates that traditional left-to-right generation is not strictly necessary to achieve high performance. On the other hand, results on the WMT'18 English $\to$ Chinese task tend to vary more widely, suggesting that translation for less well-aligned language pairs may be more sensitive to generation order.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yu Zhang
2. Mohammad Norouzi
3. William Chan
4. Quoc V. Le
5. Ron J. Weiss
6. Yoshua Bengio
7. Shinji Watanabe
8. Jinyu Li

In Natural Language Processing

1. Yu Zhang
2. William Chan
3. Ron J. Weiss
4. Mohammad Norouzi
5. Navdeep Jaitly
6. Heiga Zen
7. Dong Yu
8. Jinyu Li

Citations in the corpus

• Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, methodology): “Such models sacrifice almost no performance, while requiring a logarithmic (Chan et al., 2019c) or a constant number of generation steps (Lee et al., 2018).”
• WaveGrad: Estimating Gradients for Waveform Generation (2020, background): “…2019; Lee et al., 2018; Ghazvininejad et al., 2019; Chan et al., 2020; Saharia et al., 2020), insertion-based generative models (Stern et al., 2019; Chan et al., 2019b;a;c; Li & Chan, 2019) and edit-based generative models (Sabour et al., 2019; Gu et al., 2019; Ruis et al., 2019) found in the…”
• Non-Autoregressive Machine Translation with Latent Alignments (2020, methodology): “…non-autoregressive iterative methods for machine translation (Gu et al., 2018), some of which are: iterative refinement (Lee et al., 2018), insertion-based methods (Stern et al., 2019; Chan et al., 2019a; Li and Chan, 2019), and conditional masked language models (Ghazvininejad et al., 2019, 2020b).”
• Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation (2020, unknown): “Non-autoregressive Machine Translation In addition to the work already discussed, several other works proposed to iteratively refine (or insert) output predictions (Mansimov et al., 2019; Stern et al., 2019; Gu et al., 2019a; Chan et al., 2019; 2020; Li et al., 2020; Guo et al., 2020).”
• Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation (2020, unknown): “Non-autoregressive Translation In addition to the work already discussed, several other works proposed to iteratively refine (or insert) output predictions (Mansimov et al., 2019; Stern et al., 2019; Gu et al., 2019a; Chan et al., 2019a,b; Li et al., 2020).”