The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation

By Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, S. Krishnan, Marc'Aurelio Ranzato, Francisco (Paco) Guzmán, Angela Fan. 2021

831 citations recorded by the source corpus.

One of the biggest challenges hindering progress in low-resource and multilingual machine translation is the lack of good evaluation benchmarks. Current evaluation benchmarks either lack good coverage of low-resource languages, consider only restricted domains, or are low quality because they are constructed using semi-automatic procedures. In this work, we introduce the Flores-101 evaluation benchmark, consisting of 3001 sentences extracted from English Wikipedia and covering a variety of different topics and domains. These sentences have been translated in 101 languages by professional translators through a carefully controlled process. The resulting dataset enables better assessment of model quality on the long tail of low-resource languages, including the evaluation of many-to-many multilingual translation systems, as all translations are fully aligned. By publicly releasing such a high-quality and high-coverage dataset, we hope to foster progress in the machine translation community and beyond.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “We do not evaluate OPT-66B because of the lack of a similarly-sized BLOOM model. we compare to supervised results from the M2M-124 model (Fan et al., 2021) with 615M parameters, for which scores are computed by Goyal et al. (2022).”
FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech (2022, methodology): “For source transcripts, we reuse the transcripts produced by human annotators from [24].”
mSLAM: Massively multilingual joint pre-training for speech and text (2022, background): “Fleurs-LangID: Fleurs3 is a speech extension of the FLORES massively multilingual benchmark for MT (Goyal et al., 2021).”
Qwen2 Technical Report (2024, neutral): “5-110B Qwen2-72B (Goyal et al., 2022) (5-shot).”
A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity (2023, methodology): “We experiment with 6 languages: French, Chinese, Indonesian, Korean, Javanese, and Sundanese from the FLORES-200 dataset (Team et al., 2022; Goyal et al., 2021).”
MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages (2022, background): “…(Simpson et al., 2008; Strassel and Tracey, 2016; Cruz and Cheng, 2020; Lakew et al., 2020; Marivate et al., 2020; Magueresse et al., 2020; Goyal et al., 2021), while the growing number and 1https://github.com/alexa/massive size of language models (mBERT (Devlin, 2018), RoBERTa (Liu et…”
Building Machine Translation Systems for the Next Thousand Languages (2022, result): “Nonetheless, in Table 15 we provide a comparison between the SPBLEU results from our method (on distilled models) versus the Flores-101 benchmark scores (Goyal et al., 2021) reported for the massively multilingual M2M-124 (Fan et al., 2021) for overlapping languages.”
Speech Recognition via Large-Scale Weak Supervision (positive): “FLEURS (Conneau et al., 2023) is a parallel speech dataset, built on the MT benchmark FLoRes-101 (Goyal et al., 2022), which has 3001 sentences extracted from English Wikipedia and translated into 101 other languages by human translators.”
Exploring AI Ethics of ChatGPT: A Diagnostic Analysis (2023, methodology): “the translation quality, we use sacreBLEU [49] and its variant, spBLEU [50], to measured n-gram word overlaps between language model’s predication and reference.”
Is ChatGPT A Good Translator? A Preliminary Study (2023, methodology): “We compare the three different candidate prompts on the Chinese-to-English (Zh ⇒ En) translation task with the test set from Flores-101.”
Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine (2023, methodology): “By evaluating the translation among four selected languages on the Flores-101 test sets, we find that ChatGPT performs competitively with commercial translation products (e.g., Google Translate) on high-resource European languages but lags behind significantly on low-resource or distant languages.”
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants (2023, background): “The FL O R ES -200 Machine Translation Benchmark (Goyal et al., 2022; Team et al., 2022) is a dataset parallel across 200 languages.”
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2022, methodology): “Built upon N-way text translations from FLoRes (Goyal et al., 2022), FLEURS provides speech for aligned texts and creates speech-to-speech data covering all mined directions.”