The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants
By Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Don Husa, Abhinandan Krishnan, Luke S. Zettlemoyer, Madian Khabsa. Annual Meeting of the Association for Computational Linguistics, 2023
336 citations recorded by the source corpus.
We present Belebele, a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. Significantly expanding the language coverage of natural language understanding (NLU) benchmarks, this dataset enables the evaluation of text models in high-, medium-, and low-resource languages. Each question is based on a short passage from the Flores-200 dataset and has four multiple-choice answers. The questions were carefully curated to discriminate between models with different levels of general language comprehension. The English dataset on its own proves difficult enough to challenge state-of-the-art language models. Being fully parallel, this dataset enables direct comparison of model performance across all languages. We use this dataset to evaluate the capabilities of multilingual masked language models (MLMs) and large language models (LLMs). We present extensive results and find that despite significant cross-lingual transfer in English-centric LLMs, much smaller MLMs pretrained on balanced multilingual data still understand far more languages. We also observe that larger vocabulary size and conscious vocabulary construction correlate with better performance on low-resource languages. Overall, Belebele opens up new avenues for evaluating and analyzing the multilingual capabilities of NLP systems.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/fe6670cfc0d0dfe184afc8e003df51333d3a750e
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- Qwen2 Technical Report (2024, neutral): “…MMLU (Chen et al., 2023b) (5-shot on Arabic, Spanish, French, Portuguese, German, Italian, Japanese, and Korean); (b) Understanding: BELEBELE (Bandarkar et al., 2023) (5-shot), XCOPA (Ponti et al., 2020) (5-shot), XWinograd (Muennighoff et al., 2023) (5-shot), XStoryCloze (Lin et al., 2022b)…”
- Qwen3 Technical Report (2025, positive): “To evaluate the performance of Qwen3 across a broader range of languages, we utilize Belebele (Bandarkar et al., 2023), a benchmark for natural language understanding.”
- SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation (2026, unknown): “BelebeleRetrieval (Bandarkar et al., 2024) provides machine reading comprehension across 122 language variants, while WebFAQRe-trieval (Dinzinger et al., 2025) contains question-answer pairs from web FAQ pages across 75 languages.”
- MIMO: Multilingual Information Retrieval via Monolingual Objectives (2026, unknown): “This figure visualizes queries and documents from the Belebele parallel dataset (Bandarkar et al., 2024) in the embedding space, using an xlm-roberta-large encoder fine-tuned exclusively with the InfoNCE objective on monolingual query-positive pairs.”
- MLAIRE: Multilingual Language-Aware Information Retrieval Evaluation Protocal (2026, unknown): “Belebele [29] provides 488 reading-comprehension passages derived from FLORES-200 [30], each translated into 122 language variants, with associated queries.”
- AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages (2026, unknown): “…AfriMGSM (Math), AfriMMLU (Knowledge), AfriXNLI (nat-ural language inference) (Adelani et al., 2024c), Belebele (Reading Comprehension) (Bandarkar et al., 2024), Flores (Translation) (Goyal et al., 2022), Injongo (Intent Classification) (Yu et al., 2025), and SIB (topic classification)…”