Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval

By Xinyu Crystina Zhang, Xueguang Ma, Peng Shi, Jimmy J. Lin. 2021

173 citations recorded by the source corpus.

We present Mr. TyDi, a multi-lingual benchmark dataset for mono-lingual retrieval in eleven typologically diverse languages, designed to evaluate ranking with learned dense representations. The goal of this resource is to spur research in dense retrieval techniques in non-English languages, motivated by recent observations that existing techniques for representation learning perform poorly when applied to out-of-distribution data. As a starting point, we provide zero-shot baselines for this new dataset based on a multi-lingual adaptation of DPR that we call “mDPR”. Experiments show that although the effectiveness of mDPR is much lower than BM25, dense representations nevertheless appear to provide valuable relevance signals, improving BM25 results in sparse–dense hybrids. In addition to analyses of our results, we also discuss future challenges and present a research agenda in multi-lingual dense retrieval. Mr. TyDi can be downloaded at https://github.com/castorini/mr.tydi.

Subfields: Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “TyDi (Zhang et al., 2021) we report results from the original paper.”
Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent (2023, background): “(zero-shot) on passage re-ranking benchmarks (TREC, BEIR, and Mr.TyDi), compared with BM25 and previous best-supervised systems (SOTA sup., e.g., monoT5 (Nogueira et al., 2020)).”
LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (2024, methodology): “The dataset consists of ELI5 (sample ratio 0.1) (Fan et al., 2019), HotpotQA (Yang et al., 2018), FEVER (Thorne et al., 2018), MIRACL (Zhang et al., 2023), MS-MARCO passage ranking (sample ratio 0.5) and document ranking (sample ratio 0.2) (), NQ (Karpukhin et al., 2020), NLI (Gao et al., 2021), SQu”
mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval (2024, methodology): “Additionally, we incorporate three multilingual datasets: Mr.TyDi (Zhang et al., 2021), MIR-ACL (Zhang et al., 2023b), and MLDR (Chen et al., 2024).”
Precise Zero-Shot Dense Retrieval without Relevance Labels (2022, unknown): “For non-English retrieval, we consider Swahili, Korean, Japanese, and Bengali from the Mr.Tydi dataset (Zhang et al., 2021).”
RealTime QA: What's the Answer Right Now? (2022, background): “Much prior work proposed datasets for open domain (or open retrieval) QA for English and beyond (Clark et al., 2020; Asai et al., 2021, 2022; Longpre et al., 2021a; Zhang et al., 2021).”
NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (2024, unknown): “…al., 2022), SQuAD (Rajpurkar et al., 2016), ArguAna (Wachsmuth et al., 2018), BioASQ (Tsatsaronis et al., 2015), FiQA (Maia et al., 2018), FEVER (Thorne et al., 2018), HoVer (Jiang et al., 2020), SciFact (Wadden et al., 2022), NFCorpus, MIRACL (Zhang et al., 2023) and Mr.TyDi (Zhang et al., 2021).”
BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models (2021, background): “It is worthwhile to add more multilingual datasets [2, 76] (in consideration of the selection criteria) as a next step for the benchmark.”
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023, background): “However, to our knowledge, none of existing work has conducted large-scale evaluations of ChatGPT for multiple and diverse languages/tasks as we do.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “[320] present the multi-lingual benchmark dataset called Mr.”
Multilingual E5 Text Embeddings: A Technical Report (2024, background): “…models for information retrieval, semantic similarity, and clustering tasks across a diverse range of languages. nDCG@10 R@100 mE5 small mE5 base mE5 large E5 large-instruct mE5 small mE5 base mE5 large E5 2017)(sample at 10% ), MrTyDi (Zhang et al., 2021), and DuReader (Qiu et al., 2022) datasets.”
M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation (2024, unknown): “Another one is the curation of training and evaluation data for multi-lingual text embeddings, e.g., MIRACL (Zhang et al., 2023c), mMARCO (Bonifacio et al., 2021), Mr. TyDi (Zhang et al., 2021b), MKQA (Longpre et al., 2021).”
Improving Text Embeddings with Large Language Models (2023, unknown): “We also intend to investigate ways to improve the inference efficiency and lower the storage cost for LLM based text embeddings.”
MIMO: Multilingual Information Retrieval via Monolingual Objectives (2026, unknown): “Despite MLIR being the most realistic setting for global web search, recent benchmarks predominantly focus on Multi-Monolingual IR, where query and document languages are paired identically (Zhang et al., 2021; Chen et al., 2024; Akram et al., 2026; Eslami et al., 2026).”
GigaEmbeddings: Efficient Russian Language Embedding Model (2025, unknown): “The second stage focuses on high-quality labeled datasets, including MS-MARCO (Bajaj et al., 2018), Natural Questions (NQ) (Kwiatkowski et al., 2019), SQuAD (Rajpurkar et al., 2016), MIRACL (Zhang et al., 2023), and Mr TyDi (Zhang et al., 2021), to refine retrieval-specific capabilities.”
Ruri: Japanese General Text Embeddings (2024, methodology): “For reranker training, we utilized existing Japanese retrieval and QA datasets, including JSQuAD (Kurihara et al., 2022), JaQuAD (So et al., 2022), JQaRA (Tateno, 2024b), MKQA (Long-pre et al., 2021), Mr. TyDi (Zhang et al., 2021b), and MIRACL (Zhang et al., 2022).”