MLQA: Evaluating Cross-lingual Extractive Question Answering

By Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, Holger Schwenk. 2019

613 citations recorded by the source corpus.

Question answering (QA) models have shown rapid progress enabled by the availability of large, high-quality benchmark datasets. Such annotated datasets are difficult and costly to collect, and rarely exist in languages other than English, making building QA systems that work well in other languages challenging. In order to develop such systems, it is crucial to invest in high quality multilingual evaluation benchmarks to measure progress. We present MLQA, a multi-way aligned extractive QA evaluation benchmark intended to spur research in this area. MLQA contains QA instances in 7 languages, English, Arabic, German, Spanish, Hindi, Vietnamese and Simplified Chinese. MLQA has over 12K instances in English and 5K in each other language, with each instance parallel between 4 languages on average. We evaluate state-of-the-art cross-lingual models and machine-translation-based baselines on MLQA. In all cases, transfer results are shown to be significantly behind training-language performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Toolformer: Language Models Can Teach Themselves to Use Tools (2023, methodology): “Multilingual QA We evaluate all models on MLQA (Lewis et al., 2019), a multilingual QA benchmark.”
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, background): “…on short-to-medium length text 2Previous work has also developed token-free approaches for specific tasks: Gillick et al. (2016) for span labeling, Li et al. (2019) for speech recognition and synthesis, and many authors for machine translation (Lee et al., 2017; Costa-jussà et al., 2017; Cherry…”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “For example, XTREME (Hu et al., 2020) introduces a benchmark covering 40 languages across multiple NLU and retrieval tasks, XCOPA (Ponti et al., 2020) is a commonsense reasoning dataset for eleven languages, and MLQA (Lewis et al., 2020b) is a dataset for extractive question answering across seven l”
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, methodology): “Popular models of this type are mBERT (Devlin, 2018), mBART (Liu et al., 2020), and XLM-R (Conneau et al., 2019), which are multilingual variants of BERT (Devlin et al., 2018), BART (Lewis et al., 2019a), and RoBERTa (Liu et al., 2019), respectively.”
MLSUM: The Multilingual Summarization Corpus (2020, methodology): “This methodology could be thus applied on ML-SUM as a ﬁrst step toward a large-scale multilingual Question Answering corpus.”
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, methodology): “For all languages, we see a substantially lower lexical overlap in T Y D I QA as compared to MLQA and XQuAD, corpora whose generation procedures involve seen answers and translation; we also see overall lower lexical overlap in non-English languages.”
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, methodology): “…2018; Howard & Ruder, 2018; Radford et al., 2018; Devlin et al., 2018; Baevski et al., 2019; Song et al., 2019; Dong et al., 2019; Yang et al., 2019; Joshi et al., 2019; Liu et al., 2019; Lewis et al., 2019a; Raffel et al., 2019) has achieved signiﬁcant improvements for a wide range of NLP tasks.”
mSLAM: Massively multilingual joint pre-training for speech and text (2022, background): “…shown the benefit of cross-lingual transfer for improving representations of low-resource languages: on these languages, multilingual models strongly outperform monolingual pre-trained models on public benchmarks (Conneau et al., 2018; 2019; Lewis et al., 2019; Hu et al., 2020; Ruder et al., 2021).”
Unsupervised Cross-lingual Representation Learning at Scale (2019, methodology): “We also obtain new state of the art results on the MLQA cross-lingual question answering benchmark, introduced by Lewis et al. (2019).”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, methodology): “, 2018b), document classification (Schwenk & Li, 2018), and question answering (Artetxe et al., 2020; Lewis et al., 2019).”
On the Cross-lingual Transferability of Monolingual Representations (2019, background): “…work also shows that a shared vocabulary is not important for learning multilingual representations (K et al., 2020; Wu et al., 2019), while Lewis et al. (2019) propose a question answering dataset that is similar in spirit to ours but covers fewer languages and is not parallel across all…”
Pre-training via Paraphrasing (2020, methodology): “Variations on masked language models (MLMs) [Devlin et al., 2019, Liu et al., 2019, Yang et al., 2019b, Conneau et al., 2019, Lewis et al., 2019a, Raffel et al., 2019, Clark et al., 2020] provide highly effective self supervision for pre-training by removing and then reconstructing parts of an…”
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2020, methodology): “We conduct extensive experiments on several cross-lingual understanding tasks, including cross-lingual natural language inference (Con-neau et al., 2018), cross-lingual question answering (Lewis et al., 2020), and cross-lingual sentence retrieval (Artetxe and Schwenk, 2019).”
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (2021, methodology): “Models were pre-trained with a maximum sequence length of 2048 and ﬁne-tuned with a maximum sequence length of 4096 for TyDiQA, XQuAD, and MLQA, and 2048 for XNLI and PAWS-X.”
XGLUE: A New Benchmark Datasetfor Cross-lingual Pre-training, Understanding and Generation (2020, methodology): “XNLI (Conneau et al., 2018), NER (Sang, 2002; Sang and De Meulder, 2003), POS Tagging (Kim et al., 2017), MLQA (Lewis et al., 2019b) and PAWS-X (Yang et al., 2019a) are 5 multilingual datasets built for specific tasks.”
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models (2020, background): “formance of multilingual systems on different natural language processing tasks, including question answering (Artetxe et al., 2020; Lewis et al., 2019; Clark et al., 2020), natural language understanding (Conneau et al.”
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2021, methodology): “The predicted alignments are evaluated by alignment error rate (AER; Och and Ney 2003): where A, S, and P stand for the predicted alignments, the annotated sure alignments, and the annotated possible alignments, respectively.”
Rethinking embedding coupling in pre-trained language models (2020, methodology): “, 2020), MLQA (Lewis et al., 2020), and TyDiQA-GoldP (Clark et al.”
Few-Shot Question Answering by Pretraining Span Selection (2021, background): “…et al., 2016), and has since been adopted by several datasets in various domains (Trischler et al., 2017; Kembhavi et al., 2017) and languages (Lewis et al., 2020; Clark et al., 2020), with some extensions allowing for unanswerable questions (Levy et al., 2017; Rajpurkar et al., 2018) or…”