Overview of the TREC 2021 Deep Learning Track

By Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Fernando Campos, Jimmy J. Lin. 2025

91 citations recorded by the source corpus.

This is the third year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made hundreds of thousands of human annotated training labels available for both passage and document ranking tasks. In addition, this year we refreshed both the document and the passage collections which also led to a nearly four times increase in the document collection size and nearly $16$ times increase in the size of the passage collection. Deep neural ranking models that employ large scale pretraininig continued to outperform traditional retrieval methods this year. We also found that single stage retrieval can achieve good performance on both tasks although they still do not perform at par with multistage retrieval pipelines. Finally, the increase in the collection size and the general data refresh raised some questions about completeness of NIST judgments and the quality of the training labels that were mapped to the new collections from the old ones which we discuss in this report.

Subfields: Artificial Intelligence, General Social Sciences, Molecular Biology, Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Molecular Biology

Wayne Xin Zhao

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

Human Preferences as Dueling Bandits (2022, positive): “Using this algorithm, we collect over 10,000 crowdsourced preference judgments for the TREC 2021 Deep Learning Track [11], verifying the suitability of the algorithm and suggesting future directions for the track.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “For example, TREC 2019 and 2020 Deep Learning Track [31], [34] release a training set of 0.367 million queries (derived from MS MARCO [32]) for document retrieval task, which is significantly larger than those in previous retrieval tasks in TREC.”
Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems (2026, unknown): “Commonly used collections, such as MS MARCO v1 (Bajaj et al., 2016) and v2 (Craswell et al., 2021), contain at most 12M documents.”
Search for Coverage: Learning Coverage-Aware Retrieval with Augmented Sub-Question Answerability (2026, unknown): “With the advances in representation contrastive learning [20, 23], neural dense retrieval models have achieved great success in relevance ranking tasks, such as MSMARCO passage ranking [1] or TREC DL [8].”
FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents (2025, unknown): “This is used to construct standard test collections, such as BEIR [74] and TREC datasets such as the Deep Learning (DL) track [10, 11, 13].”