TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages

By J. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, Jennimaria Palomaki. 2020

759 citations recorded by the source corpus.

Abstract Confidently making progress on multilingual modeling requires challenging, trustworthy evaluations. We present TyDi QA—a question answering dataset covering 11 typologically diverse languages with 204K question-answer pairs. The languages of TyDi QA are diverse with regard to their typology—the set of linguistic features each language expresses—such that we expect models performing well on this set to generalize across a large number of the world’s languages. We present a quantitative analysis of the data quality and example-level qualitative linguistic analyses of observed language phenomena that would not be found in English-only corpora. To provide a realistic information-seeking task and avoid priming effects, questions are written by people who want to know the answer, but don’t know the answer yet, and the data is collected directly in each language without the use of translation.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, methodology): “, tasks from GEM (Gehrmann et al., 2021), and TyDiQA (Clark et al., 2020).”
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, background): “, 2019), and TyDi QA (Clark et al., 2020) reading comprehension benchmarks with 10,”
Scaling Instruction-Finetuned Language Models (2022, methodology): “For TyDiQA (Clark et al., 2020), we use one-shot prompting following the protocol of Chowdhery et al.”
Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “First, we consider Mr. TyDi (Zhang et al., 2021), a multilingual information retrieval benchmark derived from TyDi QA (Clark et al., 2020).”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, methodology): “TyDiQA-GoldP We use the gold passage version of the Typologically Diverse Question Answering (Clark et al., 2020) dataset, a benchmark for information-seeking question answering, which covers nine languages.”
PaLM 2 Technical Report (2023, methodology): “Multilingual QA To demonstrate PaLM 2’s multilingual capabilities, we evaluate on the multilingual QA dataset TyDi QA (Clark et al., 2020) in a one-shot setting.”
Language Models are Multilingual Chain-of-Thought Reasoners (2022, methodology): “Evaluation of multilingual models has mostly focused on general information extraction tasks such as question answering (Clark et al., 2020; Hu et al., 2020; Kassner et al., 2021; Ruder & Sil, 2021) as well as speciﬁc types of reasoning such as commonsense reasoning (Ponti et al., 2020; Lin et al.,…”
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (2021, methodology): “In particular, we evaluate on the question answering tasks TyDiQA-GoldP (Clark et al., 2020), XQuAD (Artetxe et al.”
Transcending Scaling Laws with 0.1% Extra Compute (2022, neutral): “For multilingual question answering, we use the well-established TydiQA [Clark et al., 2020a] benchmark.”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, background): “More specifically we compare mT5-XXL and PaLI-17B, evaluating on the English-only SuperGLUE benchmark (Wang et al., 2019a), and on three multilingual benchmarks from the XTREME (Hu et al., 2020): XNLI (Conneau et al., 2018), which is a textual entailment task covering 14 languages, XQuAD (Artetxe et”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “Multi-lingual models and benchmarks (e.g., Conneau et al. (2018); Conneau and Lample (2019); Hu et al. (2020); Clark et al. (2020)) represent another source of subpopulation shifts with corresponding disparities in model performance: training sets might contain fewer examples in low-resource…”
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020, background): “Unlike Chen and Mooney (2011) and like the TyDiQA multilingual question answering dataset (Clark et al., 2020), RxR’s instructions are not translations: all instructions are created from scratch by native speakers.”
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models (2020, background): “…of multilingual systems on different nat-ural language processing tasks, including question answering (Artetxe et al., 2020; Lewis et al., 2019; Clark et al., 2020), natural language understanding (Conneau et al., 2018; Yang et al., 2019a; Zweigen-baum et al., 2018; Artetxe and Schwenk, 2019),…”
Rethinking embedding coupling in pre-trained language models (2020, methodology): “Despite its empirical success, inefﬁciencies have been observed related to the training duration (Liu et al., 2019b), pre-training objective (Clark et al., 2020b), and training data (Conneau et al., 2020a), among others.”
XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning (2020, background): “On the other hand, TyDiQA was built independently in each language.”
Few-Shot Question Answering by Pretraining Span Selection (2021, background): “…and has since been adopted by several datasets in various domains (Trischler et al., 2017; Kembhavi et al., 2017) and languages (Lewis et al., 2020; Clark et al., 2020), with some extensions allowing for unanswerable questions (Levy et al., 2017; Rajpurkar et al., 2018) or multiple answer spans…”