XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation

By Sebastian Ruder, Noah Constant, Jan A. Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Jun-Jie Hu, Graham Neubig and 2 more. 2021

234 citations recorded by the source corpus.

Machine learning has brought striking advances in multilingual natural language processing capabilities over the past year. For example, the latest techniques have improved the state-of-the-art performance on the XTREME multilingual benchmark by more than 13 points. While a sizeable gap to human-level performance remains, improvements have been easier to achieve in some tasks than in others. This paper analyzes the current state of cross-lingual transfer learning and summarizes some lessons learned. In order to catalyze meaningful progress, we extend XTREME to XTREME-R, which consists of an improved set of ten natural language understanding tasks, including challenging language-agnostic retrieval tasks, and covers 50 typologically diverse languages. In addition, we provide a massively multilingual diagnostic suite and fine-grained multi-dataset evaluation capabilities through an interactive public leaderboard to gain a better understanding of such models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

mSLAM: Massively multilingual joint pre-training for speech and text (2022, methodology): “We evaluate our models under both the zero-shot and translate-train-all settings (Ruder et al., 2021), and compare performance against mT5 (Xue et al., 2021a).”
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, methodology): “Multilingually pretrained NLP models such as mBERT [2], XLM-R [8] or mT5 [9] brought significant improvements in multilingual language understanding [10, 11].”
Language Models are Multilingual Chain-of-Thought Reasoners (2022, methodology): “(2020), whereas the mT5 results are from (Ruder et al., 2021).”
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (2021, background): “Multilingual models Current multilingual models are generally analogues to successful monolingual Transformer models (Ruder et al., 2021).”
SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training (2021, background): “Since GPT (Radford et al., 2018), BERT (Devlin et al., 2019) and their variations (Yang et al., 2019; Conneau & Lample, 2019; Lewis et al., 2020; Raffel et al., 2020; Joshi et al., 2020), performance on natural language understanding downstream tasks (Socher et al., 2013; Rajpurkar et al., 2016; Agi”
mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval (2024, methodology): “We only run XTREME-R (Ruder et al., 2021) in the zero-shot cross-lingual transfer learning setting, where models are fine-tuned on English trainset and tested on multi-and cross-lingual data.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “Recently, as there is less room for improvement in the XTREME benchmark with existing achieving improvements by almost 13 points, Ruder et al. [271] extended XTREME to XTREME-R which consists of ten challenging NLU tasks.”
Few-shot Learning with Multilingual Language Models (2021, background): “Despite being the dominant approach adopted by the community (Ruder et al., 2021), it was previously shown to introduce data artifacts that inflate the measured cross-lingual transfer of models (Artetxe et al.”
TaTa: A Multilingual Table-to-Text Dataset for African Languages (2022, background): “Despite the recent increase in work focusing on creating multilingual and cross-lingual resources for NLP (Nekoto et al., 2020; Ponti et al., 2020; Ruder et al., 2021), data-to-text datasets are mostly limited to English and a small number of other languages.”
IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages (2022, methodology): “Similar to previous text-based, cross-lingual transfer benchmarks, such as XGLUE (Liang et al., 2020) and XTREME (Hu et al., 2020), our V&L benchmark is designed to evaluate zero-shot transfer scenarios, where annotated training data is provided in English, but none in the target language.”
Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation (2022, background): “Cross-lingual language understanding is an important area of ongoing research (Conneau et al., 2020; Hu et al., 2020; Ruder et al., 2021).”
Challenges and Strategies in Cross-Cultural NLP (2022, background): “While the volume of multilingual datasets increases and multilingual NLU benchmarks are becoming available (Hu et al., 2020; Liang et al., 2020; Ruder et al., 2021), this does not guarantee cross-cultural representation.”
Polyglot Prompt: Multilingual Multitask PrompTraining (2022, background): “), we extend the ExplainaBoard (Liu et al., 2021a; Fu et al., 2020; Ruder et al., 2021) to multilingual settings.”
XTREME-S: Evaluating Cross-lingual Speech Representations (2022, methodology): “In the multilingual setting, new evaluation datasets like XNLI [31], MLQA [32] or TyDi QA [33] are grouped in the XTREME benchmarks [9, 10], on which methods like mBERT, XLM-R or mT5 show their generalization capabilities across languages.”
A Primer on Pretrained Multilingual Language Models (2021, unknown): “Of these Mewsli-X and LAReQA XQuAD-R are considered to be more challenging as they involve retrieving a matching sentence in the target language from a multilingual pool of sentences.”
Training Sparse Mixture Of Experts Text Embedding Models (2025, unknown): “We evaluate mNomic-BERT on XTREME-R (Ruder et al., 2021), a comprehensive benchmark consisting of 10 tasks designed to assess multilingual natural language understanding capabilities.”