Two New Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English

By Francisco (Paco) Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, Marc'Aurelio Ranzato. 2019

150 citations recorded by the source corpus.

The vast majority of language pairs in the world are low-resource because they have little, if any, parallel data available. Unfortunately, machine translation (MT) systems do not currently work well in this setting. Besides the technical challenges of learning with limited supervision, there is also another challenge: it is very difficult to evaluate methods trained on low resource language pairs because there are very few freely and publicly available benchmarks. In this work, we take sentences from Wikipedia pages and introduce new evaluation datasets in two very low resource language pairs, Nepali-English and Sinhala-English. These are languages with very different morphology and syntax, for which little out-of-domain parallel data is available and for which relatively large amounts of monolingual data are freely available. We describe our process to collect and cross-check the quality of translations, and we report baseline performance using several learning settings: fully supervised, weakly supervised, semi-supervised, and fully unsupervised. Our experiments demonstrate that current state-of-the-art methods perform rather poorly on this benchmark, posing a challenge to the research community working on low resource MT. Data and code to reproduce our experiments are available at https://github.com/facebookresearch/flores.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Multilingual Denoising Pre-training for Neural Machine Translation (2020, methodology): “For a fair comparison, we use the same monolingual data as (Guzmán et al., 2019) to generate BT data.”
Beyond English-Centric Multilingual Machine Translation (2020, background): “• FLORES — FLORES2 (Guzmán et al., 2019) pairs two low resource languages, Sinhala and Nepali, with English in the Wikipedia domain.”
MURAL: Multimodal, Multitask Retrieval Across Languages (2021, background): “0, Autshumato (Groe-newald and du Plooy, 2010), PMIndia (Haddow and Kirefu, 2020), CVIT (Srivastava et al., 2020), Inuktitut (Hernandez and Nguyen, 2020), NLPC, JESC (Pryzant et al., 2018), KFTT (Neubig, 2011), ASPEC (Nakazawa et al., 2016), Flores (Guzmán et al., 2019).”
Unsupervised Translation of Programming Languages (2020, background): “Since creating parallel corpora for training is not realistic (creating a small parallel corpus for evaluation is already challenging [19]), some approaches have investigated the use of monolingual data to improve existing machine translation systems [17, 20, 41, 49].”
The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation (2021, background): “These often have very low coverage of low-resource languages (Riza et al., 2016; Thu et al., 2016; Guzmán et al., 2019; Barrault et al., 2020a; Nekoto et al., 2020; Ebrahimi et al., 2021; Kuwanto et al., 2021), limiting our understanding of how well methods generalize and scale to a larger number of”
Low-Resource Corpus Filtering Using Multilingual Sentence Embeddings (2019, methodology): “The quality of the resulting subsets is determined by the quality of a statistical machine translation (Moses, phrase-based (Koehn et al., 2007)) and the neural machine translation system fairseq (Ott et al., 2019) trained on this data.”
Baichuan 2: Open Large-scale Language Models (2023, unknown): “We use Flores-101 (Costa-Jussà et al., 2022; Goyal et al., 2021; Guzmán et al., 2019) to evaluate multilingual ability.”
Multilingual Translation from Denoising Pre-Training (2021, unknown): “…2014, 2016, 2017, 2018; Barrault et al., 2019, 2020), IWSLT (Luong and Manning, 2015; Cettolo et al., 2017), TED58 (Qi et al., 2018), OPUS (Tiedemann, 2012), WAT (Nakazawa et al., 2019), LauraMartinus (Abbott and Martinus, 2019), ITB (Kunchukuttan et al., 2018), and FLORES (Guzmán et al., 2019).”
Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models (2020, methodology): “98 , and (cid:15) = 1 e − 8 and we use the same learning rate scheduler as Ott et al. (2019).”
Findings of the 2020 Conference on Machine Translation (WMT20) (2020, methodology): “The development and test sets were created as part of the Flores initiative (Guzmán et al., 2019) by professional translation of Wikipedia content with careful vetting of the translations.”
Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “One reason could be due to the release of the FLoRes Evaluation Datasets [72], which include both Sinhala-English and Nepali-English.”
A Survey of Multilingual Neural Machine Translation (2019, methodology): “In addition, there are other small parallel datasets like the FLORES dataset for English-{Nepali,Sinhala} (Guzmán et al., 2019), the XNLI test set spanning 15 languages (Conneau et al., 2018b) and the Indic parallel corpus (Birch et al., 2011).”
PMIndia - A Collection of Parallel Corpora of Languages of India (2020, background): “…were released for the shared news translation tasks in English-Hindi and EnglishGujarati at the WMT conference (Bojar et al., 2014; Barrault et al., 2019), as well as for the corpus-filtering task (Guzmán et al., 2019; Koehn et al., 2019), which targeted English–Sinhala and English–Nepali.”
Findings of the WMT 2019 Shared Task on Parallel Corpus Filtering for Low-Resource Conditions (2019, methodology): “, 2019) trained on this data, and then measuring their BLEU score on the flores Wikipedia test sets (Guzmán et al., 2019).”