Greedy Search with Probabilistic N-gram Matching for Neural Machine Translation

By Chenze Shao, Yang Feng, Xilin Chen. 2018

29 citations recorded by the source corpus.

Neural machine translation (NMT) models are usually trained with the word-level loss using the teacher forcing algorithm, which not only evaluates the translation improperly but also suffers from exposure bias. Sequence-level training under the reinforcement framework can mitigate the problems of the word-level loss, but its performance is unstable due to the high variance of the gradient estimation. On these grounds, we present a method with a differentiable sequence-level training objective based on probabilistic n-gram matching which can avoid the reinforcement framework. In addition, this method performs greedy search in the training which uses the predicted words as context just as at inference to alleviate the problem of exposure bias. Experiment results on the NIST Chinese-to-English translation tasks show that our method significantly outperforms the reinforcement-based algorithms and achieves an improvement of 1.5 BLEU points on average over a strong baseline system.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yang Feng
2. Shuhao Gu
3. Qun Liu
4. Chenze Shao
5. Wanying Xie
6. Fandong Meng
7. Dong Yu
8. Di You

In Computer Vision and Pattern Recognition

1. Yang Feng
2. Shuhao Gu
3. Chenze Shao
4. Wanying Xie
5. Dong Yu
6. Qun Liu
7. Di You
8. Fandong Meng

Citations in the corpus

• Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation (2020, background): “Wang and Sennrich (2020) links the exposure bias problem (Ranzato et al., 2016; Shao et al., 2018; Zhang et al., 2019) to the phenomenon of NMT tends to generate hallucinations under domain shift.”
• Token-level Adaptive Training for Neural Machine Translation (2020, unknown): “Some work also makes use of word frequency information to help learning, such as in the word segmentation (Sun et al., 2014) and term extraction (Frantzi et al., 1998; Vu et al., 2008).”
• Modeling Fluency and Faithfulness for Diverse Neural Machine Translation (2019, methodology): “Shao, Chen, and Feng (2018) employed a greedy search strategy to generate translation, then used the accuracy of probabilistic n-grams as the training loss.”
• Bridging the Gap between Training and Inference for Neural Machine Translation (2019, background): “Shao et al. (2018) proposed to eliminate the exposure bias through a probabilistic n-gram matching objective, which trains NMT NMT under the greedy decoding strategy.”