Revisiting Low-Resource Neural Machine Translation: A Case Study

By Rico Sennrich, Biao Zhang. 2019

237 citations recorded by the source corpus.

It has been shown that the performance of neural machine translation (NMT) drops starkly in low-resource conditions, underperforming phrase-based statistical machine translation (PBSMT) and requiring large amounts of auxiliary data to achieve competitive results. In this paper, we re-assess the validity of these results, arguing that they are the result of lack of system adaptation to low-resource settings. We discuss some pitfalls to be aware of when training low-resource NMT systems, and recent techniques that have shown to be especially helpful in low-resource settings, resulting in a set of best practices for low-resource NMT. In our experiments on German–English with different amounts of IWSLT14 training data, we show that, without the use of any auxiliary monolingual or multilingual data, an optimized NMT system can outperform PBSMT with far less data than previously claimed. We also apply these techniques to a low-resource Korean–English dataset, surpassing previously reported results by 4 BLEU.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Beyond English-Centric Multilingual Machine Translation (2020, background): “Strong performance for low-resource languages remains a critical area for future improvement (Gu et al., 2018; Sennrich and Zhang, 2019).”
BPE-Dropout: Simple and Effective Subword Regularization (2019, background): “This might be explained by the fact that Kudo (2018) used large vocabulary size (16k, 32k), which has been shown counterproductive for small datasets (Sennrich and Zhang, 2019; Ding et al., 2019).”
PMIndia - A Collection of Parallel Corpora of Languages of India (2020, background): “It has recently been shown that performance on low-resource NMT is sensitive to hyperparameter tuning (Sennrich and Zhang, 2019), so we expect that better results could be obtained by tuning for each language pair individually.”
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation (2020, background): “A growing body of research is dedicated to the analyses of attention mechanisms and the interpretation of weights and connections (Raganato and Tiedemann, 2018; Tang et al., 2018; Mareček and Rosa, 2019; Voita et al., 2019a).”
Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation (2020, methodology): “data from the fine-tuning dataset, and use a smaller vocabulary size of 5000, which empirically performs better for low-resource MT (Guzmán et al., 2019; Sennrich and Zhang, 2019).”
Vocabulary Learning via Optimal Transport for Neural Machine Translation (2020, unknown): “Many previous studies (Sennrich and Zhang, 2019; Ding et al., 2019; Provilkov et al., 2020; Salesky et al., 2020) show that vocabulary size also affects downstream performances, especially on low-resource tasks.”