Bridging the Gap between Training and Inference for Neural Machine Translation

By Wen Zhang, Yang Feng, Fandong Meng, Di You, Qun Liu. 2019

276 citations recorded by the source corpus.

Neural Machine Translation (NMT) generates target words sequentially in the way of predicting the next word conditioned on the context words. At training time, it predicts with the ground truth words as context while at inference it has to generate the entire sequence from scratch. This discrepancy of the fed context leads to error accumulation among the way. Furthermore, word-level training requires strict matching between the generated sequence and the ground truth sequence which leads to overcorrection over different but reasonable translations. In this paper, we address these issues by sampling context words not only from the ground truth sequence but also from the predicted sequence by the model during training, where the predicted sequence is selected with a sentence-level optimum. Experiment results on Chinese->English and WMT’14 English->German translation tasks demonstrate that our approach can achieve significant improvements on multiple datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Discriminative Adversarial Search for Abstractive Summarization (2020, neutral): “Finally, Zhang et al. (2019) proposed to select, among the different beams decoded, the one obtaining the highest BLEU score and then to ﬁne-tune the model on that sequence.”
Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation (2020, background): “Wang and Sennrich (2020) links the exposure bias problem (Ranzato et al., 2016; Shao et al., 2018; Zhang et al., 2019) to the phenomenon of NMT tends to generate hallucinations under domain shift.”
Quark: Controllable Text Generation with Reinforced Unlearning (2022, background): “Recent works have explored applying RL on tasks such as question-answering [84, 85, 47, 83, 84], summarization [58, 53, 70, 60, 17, 51], and machine translation [58, 87, 79, 82, 81, 13, 66, 5, 49].”
Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training (2021, methodology): “We follow BERT to adopt the WordPiece tokenizer [44] to split a sentence into word tokens.”
Relating Neural Text Degeneration to Exposure Bias (2021, methodology): “Since then, there have been many methods proposed to tackle this problem (Bengio et al., 2015; Huszár, 2015; Ranzato et al., 2016; Lamb et al., 2016; Yu et al., 2017; Wiseman and Rush, 2016; Schmidt, 2019; Zhang et al., 2019a; Wang and Sennrich, 2020).”
Classification of Hyperspectral Image Based on Double-Branch Dual-Attention Mechanism Network (2019, background): “A great surge of computer vision tasks have benefited from DL and made significant breakthroughs, such as objection detection [18], natural language processing [19], and Preprints (www.”
Token-level Adaptive Training for Neural Machine Translation (2020, unknown): “…the underlying cause of, among others, the gender-biased output problem (Vanmassenhove et al., 2019a), the inability of MT system to handle morphologically richer language correctly (Passban et al., 2018), or the exposure bias problem (Ranzato et al., 2016; Shao et al., 2018; Zhang et al., 2019).”
KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation (2021, methodology): “Linguistic Embedding For the language D , we ﬁrst tokenize the sentence into a sequence of word tokens using WordPiece (Wu et al., 2016), then encode them into word embeddings W = { w j } Tj =1 where w j ∈ R d w is the feature vector.”
On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation (2020, background): “Previous work has sought to reduce exposure bias in training (Bengio et al., 2015; Ranzato et al., 2016; Shen et al., 2016; Wiseman and Rush, 2016; Zhang et al., 2019).”
Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation (2021, background): “Zhang et al. (2019b) applies curriculum learning to the DA problem.”
Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation (2020, background): “, 2016), has been linked to many of the pathologies of NMT and has motivated modifications or alternatives to MLE aimed at exposing the model to its own predictions already during training (Bengio et al., 2015; Ranzato et al., 2016; Shen et al., 2016; Wiseman and Rush, 2016; Zhang et al., 2019).”
TPLinker: Single-stage Joint Extraction of Entities and Relations Through Token Pair Linking (2020, background): “As a result, the predicted tokens at training and inference are drawn from different distributions, namely, from the data distribution as opposed to the model distribution(Zhang et al., 2019).”
Autoregressive Knowledge Distillation through Imitation Learning (2020, unknown): “, 2011) and has been reported in previous works (Bahdanau et al., 2014; Zhang et al., 2019).”
Natural Language Processing Advancements By Deep Learning: A Survey (2020, background): “[181] is also an insightful recent work where the authors sample context words from the predicted sequence as well as the ground truth to try to reconcile the training and inference processes.”
Modeling Fluency and Faithfulness for Diverse Neural Machine Translation (2019, methodology): “Neural machine translation (NMT) (Kalchbrenner and Blunsom 2013; Sutskever, Vinyals, and Le 2014; Bahdanau, Cho, and Bengio 2014; Gehring et al. 2017; Vaswani et al. 2017; Zhang et al. 2019b) has shown its superiority and drawn much attention recently.”
Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation (2022, background): “Authors have both validated (Xu et al., 2019; Zhang et al., 2019b) and questioned (He et al.”
On the Relation between Position Information and Sentence Length in Neural Machine Translation (2019, background): “Only a few studies do the analysis on the translation quality in terms of sentence length (Elbayad et al., 2018; Zhang et al., 2019).”
BRIO: Bringing Order to Abstractive Summarization (2022, methodology): “, 2019) and other online sampling based methods (Bengio et al., 2015; Norouzi et al., 2016; Zhang et al., 2019) belong to another school of methods used to circumvent the problem of non-differentiability.”