XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization

By Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, Melvin Johnson. 2020

1,171 citations recorded by the source corpus.

Much recent progress in applications of machine learning models to NLP has been driven by benchmarks that evaluate models across a wide variety of tasks. However, these broad-coverage benchmarks have been mostly limited to English, and despite an increasing interest in multilingual models, a benchmark that enables the comprehensive evaluation of such methods on a diverse range of languages and tasks is still missing. To this end, we introduce the Cross-lingual TRansfer Evaluation of Multilingual Encoders XTREME benchmark, a multi-task benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models, particularly on syntactic and sentence retrieval tasks. There is also a wide spread of results across languages. We release the benchmark to encourage research on cross-lingual learning methods that transfer linguistic knowledge across a diverse and representative set of languages and tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (2020, background): “A composite score could reﬂect average performance across languages (Hu et al., 2020).”
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, methodology): “To test the effects of moving to byte-level modeling on cross-lingual understanding, we compare parameter-matched ByT5 and mT5 models on tasks from the popular XTREME benchmark suite (Hu et al., 2020).”
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, methodology): “To validate the performance of mT5, we evaluate our models on 6 tasks from the xtreme multilingual benchmark (Hu et al., 2020): the XNLI (Conneau et al., 2018) We cast all tasks into the text-to-text format, i.e. generating the label text (XNLI and PAWS-X), entity tags and labels (WikiAnn NER), or…”
MLSUM: The Multilingual Summarization Corpus (2020, background): “XTREME The Cross-lingual TRansfer Evaluation of Multilingual Encoders benchmark (Hu et al., 2020) covers 40 languages over 9 tasks.”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, background): “A multilingual version of EXMIX could provide a more robust foundation for the analysis of task families in existing works that analyze how multilingual NLP models transfer amongst different languages (Kudugunta et al., 2019; Hu et al., 2020; Wang et al., 2021).”
MLQA: Evaluating Cross-lingual Extractive Question Answering (2019, background): “Two such projects, XGLUE (Liang et al., 2020) and XTREME (Hu et al., 2020) incorporate MLQA as part of their aggregated benchmark.”
Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, methodology): “We use the Tatoeba dataset [2] to evaluate the cross-lingual alignment quality of CRISS model following the evaluation procedure specified in the XTREME benchmark [11].”
mSLAM: Massively multilingual joint pre-training for speech and text (2022, background): “…shown the benefit of cross-lingual transfer for improving representations of low-resource languages: on these languages, multilingual models strongly outperform monolingual pre-trained models on public benchmarks (Conneau et al., 2018; 2019; Lewis et al., 2019; Hu et al., 2020; Ruder et al., 2021).”
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, background): “Furthermore, it is only necessary to maintain a single multilingual model instead of a myriad of monolingual models.”
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation (2021, methodology): “Representative samples of these datasets include ImageNet [13] for computer vision, GLUE [76] and SuperGLUE[77] for natural language understanding, XTREME [29] and XGLUE [43] for cross-lingual natural language processing.”
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, methodology): “The mT5 model pre-trained on mC4 is evaluated on 6 tasks of the XTREME benchmark (Hu et al., 2020) covering a variety of languages and outperforms other multilingual pre-trained language models such as mBERT (Devlin et al.”
Pre-training via Paraphrasing (2020, methodology): “We follow the setup of Hu et al. [2020], using no ﬁne-tuning.”
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (2020, background): “Aside from the problem of education and complexity, existing techniques may not apply due to linguistic and morphological differences in the languages, or the scale, domain, or quality of the data (Hu et al., 2020; Pires et al., 2019).”
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2020, methodology): “Cross-Lingual Transfer Gap Cross-lingual transfer gap (Hu et al., 2020) is the difference between the performance on the English test set and the averaged performance on the test sets of all other languages.”
What Language Model to Train if You Have One Million GPU Hours? (2022, background): “Zero-shot multilingual evaluation is more challenging to setup because it requires writing new prompts for each new language.”
MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer (2020, methodology): “This subsumes both the standard zero-shot cross-lingual transfer setting (Hu et al., 2020) as well as the standard monolingual in-language setting.”
Language Models are Multilingual Chain-of-Thought Reasoners (2022, methodology): “Evaluation of multilingual models has mostly focused on general information extraction tasks such as question answering (Clark et al., 2020; Hu et al., 2020; Kassner et al., 2021; Ruder & Sil, 2021) as well as speciﬁc types of reasoning such as commonsense reasoning (Ponti et al., 2020; Lin et al.,…”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “Publicly accessible large-scale multi-task benchmarks [12, 62, 61, 23, 37] have facilitated recent advances [16, 72, 42, 71, 64, 18, 47] in NLP.”