Six Challenges for Neural Machine Translation

By Philipp Koehn, Rebecca Knowles. 2017

1,374 citations recorded by the source corpus.

We explore six challenges for neural machine translation: domain mismatch, amount of training data, rare words, long sentences, word alignment, and beam search. We show both deficiencies and improvements over the quality of phrase-based statistical machine translation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

The Curious Case of Neural Text Degeneration (2019, background): “Still, similar issues have been reported when using large beam sizes (Koehn & Knowles, 2017) and more recently with exact inference (Stahlberg & Byrne, 2019), a counter-intuitive observation since more comprehensive search helps maximize probability.”
Meta-Learning for Low-Resource Neural Machine Translation (2018, background): “…Bahdanau et al., 2015; Vaswani et al., 2017), it has been noticed that the vanilla NMT often lags behind conventional machine translation systems, such as statistical phrase-based translation systems (PBMT, Koehn et al., 2003), for low-resource language pairs (see, e.g., Koehn and Knowles, 2017).”
Simple, Scalable Adaptation for Neural Machine Translation (2019, background): “…for high resource languages seems to be dramatically improved (Wu et al., 2016; Hassan et al., 2018), performance on out-of-domain data or low resource languages, still remains pretty poor (Duh et al., 2013; Koehn and Knowles, 2017; Farajian et al., 2017; Dong et al., 2015; Zoph et al., 2016).”
Semantic Noise Matters for Neural Natural Language Generation (2019, background): “…reports on hallucinations (i.e. adding information not grounded in the input), as observed for image captioning (Rohrbach et al., 2018), sports report generation (Wiseman et al., 2017), machine translation (Koehn and Knowles, 2017; Lee et al., 2019), and question answering (Feng et al., 2018).”
Non-Autoregressive Neural Machine Translation (2017, background): “Meanwhile, beam search suffers from diminishing returns with respect to beam size (Koehn & Knowles, 2017) and exhibits limited search parallelism because it introduces computational dependence between beams.”
Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction (2022, background): “In fact, better understanding the behavior and effect of beam search in seq2seq models remains an active research topic (Cohen and Beck, 2019; Koehn and Knowles, 2017; Hokamp and Liu, 2017), and we believe how to perform effective beam search in our setup could be an interesting research question th”
Survey of Hallucination in Natural Language Generation (2022, background): “This is because there is another contributor of hallucinations – training and modeling choices of neural models [81, 145, 150, 181].”
Unsupervised Neural Machine Translation (2017, methodology): “100,000 sentences of in-domain data, which would be insufﬁcient to train a standard NMT system (Koehn & Knowles, 2017), bring an improvement of 4-7 BLEU points, surpassing the comparable NMT system trained in a much larger parallel corpus in all cases but one.”
ToTTo: A Controlled Table-To-Text Generation Dataset (2020, background): “Neural models are known to be prone to hallucination , i.e., generating text that is ﬂuent but not faithful to the source (Vinyals and Le, 2015; Koehn and Knowles, 2017; Lee et al., 2018; Tian et al., 2019) and it is often easier to assess faithfulness of the generated text when the source content…”
Phrase-Based & Neural Unsupervised Machine Translation (2018, background): “While recent advances have reported near human-level performance on several language pairs using neural approaches (Wu et al., 2016; Hassan et al., 2018), other studies have highlighted several open challenges (Koehn and Knowles, 2017; Isabelle et al., 2017; Sennrich, 2017).”
Datasets: A Community Library for Natural Language Processing (2021, background): “…they compete with human performance, many research projects have demonstrated that these same models are fooled when given out-of-domain examples (Koehn and Knowles, 2017), simple adversarial constructions (Belinkov and Bisk, 2018), or examples that spuriously match basic patterns (Poliak et…”
The State and Fate of Linguistic Diversity and Inclusion in the NLP World (2020, background): “For example, while Machine Translation (MT) models require datapoints to the order of millions (Koehn and Knowles, 2017) to perform competitively, competent models in Question Answering require around 100 thousand datapoints (Rajpurkar et al.”
Nearest Neighbor Machine Translation (2020, methodology): “We use the multi-domains dataset (Koehn & Knowles, 2017), re-split by Aharoni & Goldberg (2020) for the domain adaptation experiments.”
Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings (2018, background): “While Neural Machine Translation (NTM) has obtained breakthrough improvements in standard benchmarks, it is known to be particularly sensitive to the size and quality of the training data (Koehn and Knowles, 2017; Khayrallah and Koehn, 2018).”
Sentence-State LSTM for Text Representation (2018, background): “Finally, sequential information flow leads to relatively weaker power in capturing longrange dependencies, which results in lower performance in encoding longer sentences (Koehn and Knowles, 2017).”
Is Attention Interpretable? (2019, background): “In machine translation, prior work has observed that recurrent encoders over a full sequence can “shift” tokens’ signal in ways that cause subsequent attention layers to compute unintuitive off-by-one alignments (Koehn and Knowles, 2017).”
An Analysis of Encoder Representations in Transformer-Based Machine Translation (2018, background): “The most closely related work is by Belinkov et al. (2017b), in which they investigate the representation learned by the encoder of a sequence-tosequence NMT system across different languages.”
Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis (2018, background): “Koehn & Knowles (2017) ﬁnd that when training machine translation models on corpora with fewer than 15 million words (English side), statistical machine translation approaches outperform neural ones.”
Rapid Adaptation of Neural Machine Translation to New Languages (2018, methodology): “Comparing with the phrase-based baseline, as noted by Koehn and Knowles (2017) NMT tends to underperform on low-resource settings when trained only on the data available for these languages.”