Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade

By Jiatao Gu, X. Kong. 2020

155 citations recorded by the source corpus.

Fully non-autoregressive neural machine translation (NAT) is proposed to simultaneously predict tokens with single forward of neural networks, which significantly reduces the inference latency at the expense of quality drop compared to the Transformer baseline. In this work, we target on closing the performance gap while maintaining the latency advantage. We first inspect the fundamental issues of fully NAT models, and adopt dependency reduction in the learning space of output tokens as the basic guidance. Then, we revisit methods in four different aspects that have been proven effective for improving NAT models, and carefully combine these techniques with necessary modifications. Our extensive experiments on three translation benchmarks show that the proposed system achieves the new state-of-the-art results for fully NAT models, and obtains comparable performance with the autoregressive and iterative NAT systems. For instance, one of the proposed models achieves 27.49 BLEU points on WMT14 En-De with approximately 16.5X speed up at inference time.

Subfields: Artificial Intelligence, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Furu Wei
2. Tao Ge
3. Xin Sun
4. Heming Xia
5. Zhifang Sui
6. Houfeng Wang
7. Ce Liu
8. Peiyi Wang

In Machine Translation

1. José Lezama
2. Xin Sun
3. Houfeng Wang
4. Han Zhang
5. Irfan Essa
6. Huiwen Chang
7. Lu Jiang
8. Furu Wei

Citations in the corpus

• Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding (2021, methodology): “…the decoder layer, we see a drop in the performance of 10+2 and 11+1 despite the improved efficiency because it becomes difficult to train the Transformer with extremely imbalanced encoder and decoder well, as indicated 9 by the previous work (Kasai et al., 2020; Li et al., 2021; Gu and Kong, 2020).”
• MaskGIT: Masked Generative Image Transformer (2022, methodology): “Our work is inspired by bi-directional machine translation [16, 19, 20] in NLP, and our novelty lies in the proposed new masking strategy and decoding algorithm which, as substantiated by our experiments, are essential for image generation.”
• Lossless Acceleration for Seq2seq Generation with Aggressive Decoding (2022, unknown): “…Du et al., 2021; Huang et al., 2021), modeling dependencies between target tokens (Ghazvininejad et al., 2019; Qian et al., 2021; Song et al., 2021; Gu & Kong, 2021) and refining the translation outputs with multi-pass iterations (Ghazvininejad et al., 2020b; Kasai et al., 2020; Hao et al., 2021;…”
• Improved Masked Image Generation with Token-Critic (2022, methodology): “During inference, the model adopts a non-autoregressive decoding method to synthesize an image in a constant number of steps (typically 8-16 steps) [15].”
• Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation (2022, background): “…NAR models, including training with alignment-based objectives (Libovický and Helcl, 2018; Ghazvininejad et al., 2020; Saharia et al., 2020; Gu and Kong, 2021; Shao and Feng, 2022), modeling dependencies between target to-kens (Ghazvininejad et al., 2019; Shu et al., 2020; Qian et al., 2021; Bao…”