Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input

By Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, Tie-Yan Liu. 2018

133 citations recorded by the source corpus.

Non-autoregressive translation (NAT) models, which remove the dependence on previous target tokens from the inputs of the decoder, achieve significantly inference speedup but at the cost of inferior accuracy compared to autoregressive translation (AT) models. Previous work shows that the quality of the inputs of the decoder is important and largely impacts the model accuracy. In this paper, we propose two methods to enhance the decoder inputs so as to improve NAT models. The first one directly leverages a phrase table generated by conventional SMT approaches to translate source tokens to target tokens, which are then fed into the decoder as inputs. The second one transforms source-side word embeddings to target-side word embeddings through sentence-level alignment and word-level adversary learning, and then feeds the transformed word embeddings into the decoder as inputs. Experimental results show our method largely outperforms the NAT baseline (Gu et al. 2017) by 5.11 BLEU scores on WMT14 English-German task and 4.72 BLEU scores on WMT16 English-Romanian task.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Tie-Yan Liu
3. Xu Tan
4. Sheng Zhao
5. Yingce Xia
6. Lijun Wu
7. Di He
8. Fei Tian

Citations in the corpus

• Multilingual Neural Machine Translation with Knowledge Distillation (2019, neutral): “…rapid development in recent years (Bahdanau et al., 2015; Luong et al., 2015b; Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017; Guo et al., 2018; Shen et al., 2018), including advanced model structures (Gehring et al., 2017; Vaswani et al., 2017; He et al., 2018; Xia et al.,…”
• Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion (2019, methodology): “[5] adopted convolutional sequence to sequence model and proposed the non-sequential decoding [22] for G2P conversion, which achieved the previous state-of-theart result on the public CMUDict 0.”
• M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers (2021, background): “Thus, the non-autoregressive generation (NAR) paradigm is proposed to avoid these drawbacks in neural machine translation [19, 20, 29, 16], image captioning [14, 22, 70], and speech synthesis [51, 50].”
• Gradient-based Constrained Sampling from Language Models (2022, background): “…in the literature including predicting the sequence length (Wang et al., 2021), generating multiple outputs with varying lengths and reranking (Guo et al., 2019), continuing generating autoregressively to finish a sentence after a fixed length output is generated (Qin et al., 2022) of all…”
• Neural Machine Translation: A Review (2019, background): “• Non- or partially autoregressive architectures (Wang et al., 2018a; Gu et al., 2017a; Guo et al., 2018; Wang et al., 2019; Libovický & Helcl, 2018; Lee et al., 2018; Akoury et al., 2019)”
• UFC-BERT: Unifying Multi-Modal Controls for Conditional Image Synthesis (2021, background): “Thus, the non-autoregressive generation (NAR) paradigm is proposed to avoid these drawbacks in neural machine translation [19, 20, 29, 16], image captioning [14, 22, 70], and speech synthesis [50, 49].”
• Efficient Bidirectional Neural Machine Translation (2019, background): “We will also investigate whether there exist some better ways to support bidirectional sequence generation in a single model, or even generate the sequence in parallel (Guo et al., 2019).”
• Sentence-wise Smooth Regularization for Sequence to Sequence Learning (2018, methodology): “ern at Microsoft. Copyright c 2019, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. tasks, such as exposure bias (Wiseman and Rush 2016; Wu et al. 2018; Guo et al. 2019), loss-evaluation mismatch (Wiseman and Rush 2016; Vijayakumar et al. 2016), and myo”
• Multi-Agent Dual Learning (2019, methodology): “For example, we can apply our framework to improve the recent non-autoregressive machine translation, which is known to have high inference efficiency yet poor translation quality (Gu et al., 2018; Wang et al., 2019; Guo et al., 2019).”