The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English

By Francisco (Paco) Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, Marc'Aurelio Ranzato. 2019

250 citations recorded by the source corpus.

For machine translation, a vast majority of language pairs in the world are considered low-resource because they have little parallel data available. Besides the technical challenges of learning with limited supervision, it is difficult to evaluate methods trained on low-resource language pairs because of the lack of freely and publicly available benchmarks. In this work, we introduce the FLORES evaluation datasets for Nepali–English and Sinhala– English, based on sentences translated from Wikipedia. Compared to English, these are languages with very different morphology and syntax, for which little out-of-domain parallel data is available and for which relatively large amounts of monolingual data are freely available. We describe our process to collect and cross-check the quality of translations, and we report baseline performance using several learning settings: fully supervised, weakly supervised, semi-supervised, and fully unsupervised. Our experiments demonstrate that current state-of-the-art methods perform rather poorly on this benchmark, posing a challenge to the research community working on low-resource MT. Data and code to reproduce our experiments are available at https://github.com/facebookresearch/flores.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Beyond English-Centric Multilingual Machine Translation (2020, background): “• FLORES — FLORES2 (Guzmán et al., 2019) pairs two low resource languages, Sinhala and Nepali, with English in the Wikipedia domain.”
Revisiting Self-Training for Neural Sequence Generation (2019, methodology): “We use the same transformer architecture as in (Guzmán et al., 2019).”
Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, methodology): “For English-French we use WMT’14, for EnglishGerman and English-Romanian we use WMT’16 test data, and for English-Nepali and EnglishSinhala we use Flores test set [21].”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, background): “In most cases, evaluation has been restricted to typologically related languages and similar domains; approaches have been shown to fail in less favorable conditions (Glavaˇs et al., 2019; Vuli´c et al., 2019; Guzm´an et al., 2019).”
Unsupervised Translation of Programming Languages (2020, background): “Since creating parallel corpora for training is not realistic (creating a small parallel corpus for evaluation is already challenging [19]), some approaches have investigated the use of monolingual data to improve existing machine translation systems [17, 20, 41, 49].”
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (2020, background): “Instead of a direct assessment (Graham et al., 2013) often used in benchmark MT evaluations (Barrault et al., 2019; Guzm´an et al., 2019), we opt for post-editing.”
Detecting Hallucinated Content in Conditional Neural Sequence Generation (2020, methodology): “We train two NMT models (described below) on this dataset and test on 150 examples from (Ott et al., 2019) for synthetic labeled data generation.”
Low-Resource Corpus Filtering Using Multilingual Sentence Embeddings (2019, methodology): “The quality of the machine translation system will be measured by BLEU score using SacreBLEU (Post, 2018) on a held-out test set of Wikipedia translations for Sinhala–English and Nepali–English from the flores dataset (Guzmán et al., 2019).”
Data Diversification: A Simple Strategy For Neural Machine Translation (2019, methodology): “5 BLEU improvements over the baseline model [10].”
Generalized Data Augmentation for Low-Resource Translation (2019, methodology): “Applying the UMT framework to low-resource and morphologically rich languages is largely unexplored, with the exception of Neubig and Hu (2018) and Guzmán et al. (2019), showing that UMT performs exceptionally poorly between dissimilar language pairs with BLEU scores lower than 1.”
Imitation Attacks and Defenses for Black-box Machine Translation Systems (2020, methodology): “For Nepali → English, we evaluate using FLoRes devtest (Guzm´an et al., 2019).”
Neural Machine Translation with Byte-Level Subwords (2019, methodology): “• Sinhala-English (Si-En): we use the data from FLoRes (Guzmán et al. 2019).”
Machine Translation into Low-resource Language Varieties (2021, methodology): “Direct unsupervised translation from SRC to TGT performs poorly as well, confirming previously reported results of the ineffectiveness of such methods on unrelated languages (Guzmán et al., 2019).”
Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models (2020, methodology): “English and Nepali-English tasks described in the FLoRes dataset (Guzmán et al., 2019).”
A Survey of Multilingual Neural Machine Translation (2019, methodology): “In addition, there are other small parallel datasets like the FLORES dataset for English-{ Nepali,Sinhala } (Guzm´an et al., 2019), the XNLI test set spanning 15 languages (Conneau et al., 2018b) and the In-dic parallel corpus (Birch et al., 2011).”
Rissanen Data Analysis: Examining Dataset Characteristics via Description Length (2021, background): “Furthermore, a capability’s usefulness to a model in high-data regimes does not necessarily reflect its usefulness in low-data regimes, which have increasingly become of interest (Lake et al., 2017; Guzmán et al., 2019; Brown et al., 2020).”
Croissant: A Metadata Format for ML-Ready Datasets (2024, unknown): “"extract": {"fileProperty": "content"} "@id": "images/hash", 17 "@type": "cr:Field",”