MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering

By Shayne Longpre, Yi Lu, Joachim Daiber. 2020

226 citations recorded by the source corpus.

Abstract Progress in cross-lingual modeling depends on challenging, realistic, and diverse evaluation sets. We introduce Multilingual Knowledge Questions and Answers (MKQA), an open- domain question answering evaluation set comprising 10k question-answer pairs aligned across 26 typologically diverse languages (260k question-answer pairs in total). Answers are based on heavily curated, language- independent data representation, making results comparable across languages and independent of language-specific passages. With 26 languages, this dataset supplies the widest range of languages to-date for evaluating question answering. We benchmark a variety of state- of-the-art methods and baselines for generative and extractive question answering, trained on Natural Questions, in zero shot and translation settings. Results indicate this dataset is challenging even in English, but especially in low-resource languages.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “On MKQA, evaluation is performed by lowercasing both queries and documents, we observed that this improves performance.”
MTG: A Benchmark Suite for Multilingual Text Generation (2021, background): “A significant body of works have been committed to the construction of multilingual datasets covering diverse tasks (Hu et al., 2020; Jiang et al., 2020; Longpre et al., 2020).”
One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval (2021, methodology): “T Y D I QA (Asai et al., 2021) and MKQA (Longpre et al., 2020), across 26 typologically diverse languages; CORA achieves gains of 23.4 and 4.7 F1 points in X OR -”
mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval (2024, methodology): “Our models are close to the state-of-the-art large models on Miracl, MKQA and BEIR, while achieve better scores on long-context datasets MLDR and LoCo.”
Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languages (2022, background): “We have recently witnessed efforts in building resources for non-English languages, for example, CLIRMatrix [24], XTREME [7], MKQA [16], mMARCO [3], TyDi QA [4], XOR-TyDi [1], and Mr.”
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024, background): “Instruction Tuning Before multitask finetuning, significant work focused on finetuning pre-trained models on a variety of languages through data augmentation for a single task (Longpre et al., 2021; Asai et al., 2022, 2023; Hu et al., 2020).”
RealTime QA: What's the Answer Right Now? (2022, background): “An open-book QA model conditions answer generation upon newly-retrieved documents (Lewis et al., 2020b), but the extent to which answer generation can be updated based on the retrieved documents is limited (Longpre et al., 2021b).”
A Survey of Large Language Models (2023, unknown): “0 [296], GrailQA [297], KQApro [298], CWQ [299], MKQA [300], ScienceQA [301]”
Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval (2021, background): “Another point of comparison worth discussing is MKQA (?)”
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models (2021, unknown): “There are many multilingual question answering datasets such as XQuAD (Artetxe et al., 2020), TiDy (Clark et al., 2020), MKQA (Longpre et al., 2020) and MLQA (Lewis et al., 2020b).”
M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation (2024, unknown): “We make evaluation for the cross-lingual retrieval performance with the MKQA benchmark (Longpre et al., 2021), which includes queries in 25 non-English languages.”
BERGEN: A Benchmarking Library for Retrieval-Augmented Generation (2024, unknown): “Following [4], we use MKQA [55] and XOR-TyDi QA [2] datasets for multilingual evaluation in our experiments.”