TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime

By Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Fernando Campos, Ellen M. Voorhees, Ian Soboroff. 2021

73 citations recorded by the source corpus.

The TREC Deep Learning (DL) Track studies ad hoc search in the large data regime, meaning that a large set of human-labeled training data is available. Results so far indicate that the best models with large data may be deep neural networks. This paper supports the reuse of the TREC DL test collections in three ways. First we describe the data sets in detail, documenting clearly and in one place some details that are otherwise scattered in track guidelines, overview papers and in our associated MS MARCO leaderboard pages. We intend this description to make it easy for newcomers to use the TREC DL data. Second, because there is some risk of iteration and selection bias when reusing a data set, we describe the best practices for writing a paper using TREC DL data, without overfitting. We provide some illustrative analysis. Finally we address a number of issues around the TREC DL data, including an analysis of reusability.

Subfields: Artificial Intelligence, Management Science and Operations Research, Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

MS MARCO: Benchmarking Ranking Models in the Large-Data Regime (2021, background): “I R ] 9 M a y 2021 leader-boards and the TREC Deep Learning Track [13–15].”
Overview of the TREC 2021 Deep Learning Track (2025, unknown): “The MS MARCO passage and document ranking v1 datasets are used in the current MS MARCO leaderboards [Lin et al., 2022, Craswell et al., 2021b, Lin et al., 2021a] and in TREC 2019 and TREC 2020.”
FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents (2025, unknown): “…(1) requires at least 50 queries (50 is the typical minimum number of queries used in TREC, e.g., Deep Learning (DL) track 2020 used 54 queries [12]). that are human-generated for a lower variance in retrieval performance, (2) answers to the queries (preferably human-generated) that can be…”