Beyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Matter

By Lijun Wu, Xu Tan, Di He, Fei Tian, Tao Qin, J. Lai, Tie-Yan Liu. 2018

56 citations recorded by the source corpus.

Neural machine translation usually adopts autoregressive models and suffers from exposure bias as well as the consequent error propagation problem. Many previous works have discussed the relationship between error propagation and the accuracy drop (i.e., the left part of the translated sentence is often better than its right part in left-to-right decoding models) problem. In this paper, we conduct a series of analyses to deeply understand this problem and get several interesting findings. (1) The role of error propagation on accuracy drop is overstated in the literature, although it indeed contributes to the accuracy drop problem. (2) Characteristics of a language play a more important role in causing the accuracy drop: the left part of the translation result in a right-branching language (e.g., English) is more likely to be more accurate than its right part, while the right part is more accurate for a left-branching language (e.g., Japanese). Our discoveries are confirmed on different model structures including Transformer and RNN, and in other sequence generation tasks such as text summarization.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Tie-Yan Liu
3. Xu Tan
4. Yi Ren
5. Sheng Zhao
6. Zhou Zhao
7. Yingce Xia
8. Frank K. Soong

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, background): “Autoregressive sequence generation usually suffers from exposure bias and error propagation [20, 384].”
• Multilingual Neural Machine Translation with Knowledge Distillation (2019, background): “Neural Machine Translation (NMT) has witnessed rapid development in recent years (Bahdanau et al., 2015; Luong et al., 2015b; Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017; Wu et al., 2018; Song et al., 2018; Shen et al., 2018; Guo et al., 2018; He et al., 2018; Gong et al., 2018), inc”
• Almost Unsupervised Text to Speech and Automatic Speech Recognition (2019, background): “…learning tasks such as neural machine translation, they will suffer more from error propagation (Bengio et al., 2015; Shen et al., 2016; Wu et al., 2018), which refers to the problem of the right part of the generated sequence being usually worse than the left part, especially in…”
• Insertion-based Decoding with Automatically Inferred Generation Order (2019, neutral): “Nevertheless, previous studies on generation orders mostly resort to a fixed set of generation orders, showing particular choices of ordering are helpful (Wu et al., 2018; Ford et al., 2018; Mehri and Sigal, 2018), without providing an efficient algorithm for finding adaptive generation orders, or r”
• Multilingual Neural Machine Translation with Language Clustering (2019, background): “…leveraged to cluster languages (Paul et al., 2009; Comrie, 1989; Bakker et al., 2009; Holman et al., 2008; Liu and Li, 2010; Chen and Gerdes, 2017; Wu et al., 2018; Leng et al., 2019), such as language family, language typology or other language characteristics from the URIEL database (Levin et…”
• Sentence-wise Smooth Regularization for Sequence to Sequence Learning (2018, background): “…sounds reasonable in general, it has several potential issues in the context of sequence to sequence tasks, such as exposure bias (Wiseman and Rush 2016; Wu et al. 2018; Guo et al. 2019), loss-evaluation mismatch (Wiseman and Rush 2016; Vijayakumar et al. 2016), and myopic bias (He et al. 2017).”
• Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input (2018, background): “All those models translate a source sentence in an autoregressive manner, i.e., they generate a target sentence word by word from left to right (Wu et al. 2018) and the generation of t -th token y t depends on previously generated tokens y 1: t − 1 : where E ( · ) and D ( · ) denote the encoder and…”
• Layer-Wise Coordination between Encoder and Decoder for Neural Machine Translation (2018, background): “Neural Machine Translation (NMT) is a challenging task that attracts a lot of attention in recent years [5, 2, 18, 29, 7, 33, 28, 27, 35, 37, 9], and the structure of NMT models has evolved quickly.”
• On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation (2020, background): “However, the relevance of error propagation is under debate: Wu et al. (2018) argue that its role is overstated in literature, and that linguistic features explain some of the accuracy drop at higher time steps.”
• Efficient Bidirectional Neural Machine Translation (2019, background): “The same phenomenon was also observed in Liu et al. (2016a); Wu et al. (2018); Ren et al. (2019).”