Domain Robustness in Neural Machine Translation

By Mathias Müller, Annette Rios Gonzales, Rico Sennrich. 2019

104 citations recorded by the source corpus.

Translating text that diverges from the training domain is a key challenge for neural machine translation (NMT). Domain robustness - the generalization of models to unseen test domains - is low compared to statistical machine translation. In this paper, we investigate the performance of NMT on out-of-domain test sets, and ways to improve it. We observe that hallucination (translations that are fluent but unrelated to the source) is common in out-of-domain settings, and we empirically compare methods that improve adequacy (reconstruction), out-of-domain translation (subword regularization), or robustness against adversarial examples (defensive distillation), as well as noisy channel models. In experiments on German to English OPUS data, and German to Romansh, a low-resource scenario, we find that several methods improve domain robustness, reconstruction standing out as a method that not only improves automatic scores, but also shows improvements in a manual assessments of adequacy, albeit at some loss in fluency. However, out-of-domain performance is still relatively low and domain robustness remains an open problem.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, background): “Other general training methods such as regularization [61, 70, 93] and loss reconstruction [81, 142, 145] have also been proposed to tackle the hallucination problem.”
Detecting Hallucinated Content in Conditional Neural Sequence Generation (2020, background): “Previous work (Wang and Sennrich, 2020; Müller et al., 2019; Koehn and Knowles, 2017) has shown that translation models are particularly prone to hallucination when tested out of domain.”
The Curious Case of Hallucinations in Neural Machine Translation (2021, unknown): “Such hallucinated outputs are characterized as being decoupled from the source sequence, despite being (fully or moderately) ﬂuent in the target language (Müller et al., 2020).”
On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation (2020, methodology): “5 NMT (Müller et al., 2019) 61.”
Unsupervised Domain Clusters in Pretrained Language Models (2020, methodology): “We started from the split of M¨uller et al. (2019) as it included newer versions of some of the datasets.”
SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness (2020, background): “Romansh is a low-resource language with an estimated 40,000 native speakers where OOD robustness is of practical relevance (Müller et al., 2019).”