Analyzing Uncertainty in Neural Machine Translation

By Myle Ott, Michael Auli, David Grangier, Marc'Aurelio Ranzato. 2018

293 citations recorded by the source corpus.

Machine translation is a popular test bed for research in neural sequence-to-sequence models but despite much recent research, there is still a lack of understanding of these models. Practitioners report performance degradation with large beams, the under-estimation of rare words and a lack of diversity in the final translations. Our study relates some of these issues to the inherent uncertainty of the task, due to the existence of multiple valid translations for a single source sentence, and to the extrinsic uncertainty caused by noisy training data. We propose tools and metrics to assess how uncertainty in the data is captured by the model distribution and how it affects search strategies that generate translations. Our results show that search works remarkably well but that models tend to spread too much probability mass over the hypothesis space. Next, we propose tools to assess model calibration and show how to easily fix some shortcomings of current models. As part of this study, we release multiple human reference translations for two popular benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, methodology): “Recent GPUs enable efﬁcient half precision ﬂoating point (FP16) computation.”
Scaling Neural Machine Translation (2018, background): “We also remove pairs for which the source and target are copies (Ott et al., 2018).”
Adaptive Input Representations for Neural Language Modeling (2018, methodology): “We found that large batch training is beneﬁcial for this dataset and we therefore accumulate gradient updates over two batches before committing a parameter update (Ott et al., 2018a).”
Discriminative Adversarial Search for Abstractive Summarization (2020, positive): “Beam search has contributed to performance improvements of state-of-the-art models for many tasks, such as Neural Machine Translation, Summarization, and Question Generation (Ott et al., 2018; Dong et al., 2019).”
BPE-Dropout: Simple and Effective Subword Regularization (2019, background): “(8)This is the result of using beam search: while samples from a model reproduce training data distribution quite well, beam search favors more frequent tokens (Ott et al., 2018).”
FRAGE: Frequency-Agnostic Word Representation (2018, background): “However, the low-frequency sub-word units are still difﬁcult to train: [31] provides a comprehensive study which shows that the rare (sub)words are usually under-estimated in neural machine translation: during inference step, the model tends to choose popular words over their rare alternatives.”
Deep Learning for Symbolic Mathematics (2019, background): “Unlike in machine translation, where increasing the beam size does not necessarily increase the performance (Ott et al., 2018), we always observe signiﬁcant improvements with wider beams.”
Language GANs Falling Short (2018, background): “Work from Ott et al. (2018) thoroughly examines the local beam search strategy for neural machine translation.”
The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English (2019, methodology): “Indeed, modern neural MT systems can achieve near human-level translation performance on language pairs for which sufﬁcient parallel training resources exist (e.g., Chinese–English translation (Hassan et al., 2018) and English–French translation (Gehring et al., 2016; Ott et al., 2018a).”
Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation (2018, background): “The phenomenon is not well understood, but it seems that NMT models have learned to travel a path straight towards their goal; as soon as they get off this path, they get lost, and can no longer function (Ott et al., 2018).”
Mixture Models for Diverse Machine Translation: Tricks of the Trade (2019, methodology): “We validate on new-stest2012+2013, and test on a 500 sentence subset of new-stest2014 with 10 reference translations (Ott et al., 2018a).”
Two New Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English (2019, methodology): “Indeed, modern neural MT systems can achieve near human-level translation performance on language pairs for which sufﬁcient parallel training resources exist (e.g., Chinese–English translation (Hassan et al., 2018) and English–French translation (Gehring et al., 2016; Ott et al., 2018a)).”
On the Impact of Various Types of Noise on Neural Machine Translation (2018, background): “20 See Figure 3 in Ott et al. (2018).”
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing (2020, background): “copies and partial copies, as multiple studies (Ott et al., 2018; Khayrallah and Koehn, 2018) have noted that MT performance degrades substantially when systems are exposed to copies in training.”
Understanding Knowledge Distillation in Non-autoregressive Machine Translation (2019, background): “This ﬁnding is in concert with Ott et al. (2018), who demonstrate that because beam search approximately selects the most probable translation, it effectively reduces diversity in the output translations compared to sampling or the true distribution.”
Empirical Analysis of Beam Search Performance Degradation in Neural Sequence Models (2019, background): “Ott et al. (2018) proposed the existence of training pairs in which the target is a copy of the source as an explanation for the performance degradation in length-normalized machine translation models.”
Mixture Content Selection for Diverse Sequence Generation (2019, methodology): “, 2018) or pairwise- (Ott et al., 2018) metric, this measures the within-distribution similarity.”
Taking Notes on the Fly Helps BERT Pre-training (2020, unknown): “However, the low-frequency sub-word units are still difﬁcult to train (Ott et al., 2018).”
Generalizing Back-Translation in Neural Machine Translation (2019, background): “Label smoothing (LS) (Pereyra et al., 2017) is a common component of state-of-the-art NMT systems (Ott et al., 2018).”
If Beam Search Is the Answer, What Was the Question? (2020, background): “Other authors have ascribed fault to the model architectures, or how they are trained (Cho et al., 2014; Bengio et al., 2015; Sountsov and Sarawagi, 2016; Vinyals et al., 2017; Ott et al., 2018a; Kumar and Sarawagi, 2019).”