MS MARCO: Benchmarking Ranking Models in the Large-Data Regime

By Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Fernando Campos, Jimmy J. Lin. 2021

85 citations recorded by the source corpus.

Evaluation efforts such as TREC, CLEF, NTCIR and FIRE, alongside public leaderboard such as MS MARCO, are intended to encourage research and track our progress, addressing big questions in our field. However, the goal is not simply to identify which run is "best", achieving the top score. The goal is to move the field forward by developing new robust techniques, that work in many different settings, and are adopted in research and practice. This paper uses the MS MARCO and TREC Deep Learning Track as our case study, comparing it to the case of TREC ad hoc ranking in the 1990s. We show how the design of the evaluation effort can encourage or discourage certain outcomes, and raising questions about internal and external validity of results. We provide some analysis of certain pitfalls, and a statement of best practices for avoiding such pitfalls. We summarize the progress of the effort so far, and describe our desired end state of "robust usefulness", along with steps that might be required to get us there.

Subfields: Artificial Intelligence, Management Science and Operations Research, Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

Human Preferences as Dueling Bandits (2022, methodology): “This over-all approach has been validated through comparisons with Deep Learning Track results in 2019 and 2020 [10].”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “A recent study [70] has summarized the promotion effect of MS MARCO on the progress of dense retrieval: “the MS MARCO datasets have enabled large-data exploration of neural models”.”
Overview of the TREC 2021 Deep Learning Track (2025, unknown): “The MS MARCO passage and document ranking v1 datasets are used in the current MS MARCO leaderboards [Lin et al., 2022, Craswell et al., 2021b, Lin et al., 2021a] and in TREC 2019 and TREC 2020.”
Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languages (2022, methodology): “For retrieval models in English, the MS MARCO datasets [2, 5, 13] have had a transformative impact in advancing the field.”
Shallow pooling for sparse labels (2021, methodology): “Recent work from our research group has proposed other measures for preference judgments [8, 21] but for a single best answer, MRR remains appropriate [11].”
TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime (2021, background): “Both tasks have large training sets based on human relevance assessments, derived from MS MARCO [3, 10].”