Overview of the TREC 2019 deep learning track

By Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Fernando Campos, Ellen M. Voorhees. 2020

697 citations recorded by the source corpus.

The Deep Learning Track is a new track for TREC 2019, with the goal of studying ad hoc ranking in a large data regime. It is the first track with large human-labeled training sets, introducing two sets corresponding to two tasks, each with rigorous TREC-style blind evaluation and reusable test sets. The document retrieval task has a corpus of 3.2 million documents with 367 thousand training queries, for which we generate a reusable test set of 43 queries. The passage retrieval task has a corpus of 8.8 million passages with 503 thousand training queries, for which we generate a reusable test set of 43 queries. This year 15 groups submitted a total of 75 runs, using various combinations of deep learning, transfer learning and traditional IR ranking methods. Deep learning runs significantly outperformed traditional IR runs. Possible explanations for this result are that we introduced large training data and we included deep models trained on such data in our judging pools, whereas some past studies did not have such training data or pooling.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, methodology): “In TREC Deep Learning Track’s text retrieval benchmarks [18], ANCE signiﬁcantly boosts the accuracy of dense retrieval.”
Generation-Augmented Retrieval for Open-Domain Question Answering (2020, background): “, 2020), or broad search in information retrieval (Nguyen et al., 2016; Craswell et al., 2020).”
Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation (2020, methodology): “on & Query Sets We use the MSMARCO-Passage [1] collection with sparsely-judged MSMARCO-DEV query set of 49,000 queries as well as the denselyjudged query set of 43 queries derived from TREC-DL’19 [7]. For TREC graded relevance labels we use a binarization point of 2 for MRR and MAP. MSMARCO is b”
Shallow pooling for sparse labels (2021, methodology): “They argue that this concern is addressed by experiments conducted for the TREC Deep Learning Track [9, 10], which follows the typical methodology of a TREC evaluation exercise.”
Deep Reinforced Query Reformulation for Information Retrieval (2020, methodology): “The research by participants in the recent TREC 2019 Deep Learning track [9] exemplifies recent work in this direction.”
Complementing Lexical Retrieval with Semantic Residual Embedding (2020, methodology): “• TREC2019 DL Queries: this evaluation query set is the ofﬁcial evaluation query set used in the TREC 2019 Deep Learning Track (Craswell et al., 2019), a shared passage retrieval task.”
EARL: Speedup Transformer-based Rankers with Pre-computed Representation (2020, methodology): “Deep Learning Track [4], a shared passage retrieval task.”
Distilling Dense Representations for Ranking using Tightly-Coupled Teachers (2020, background): “(b) TREC-2019 DL (Craswell et al., 2019): the or-ganizers of the 2019 Deep Learning track at the Text REtrieval Conference (TREC) released 43 queries with multiple graded relevance labels, where 9k (query, passage) pairs were annotated by NIST assessors.”
Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Document Matching (2020, methodology): “Short-to-long Document Ranking TREC 2019 Deep Learning [3] Given a query and a candidate document set, rank documents according to”
TILDE: Term Independent Likelihood moDEl for Passage Re-ranking (2021, positive): “We conduct our experiments on the MS MARCO dataset 4 and the two recent TREC Deep Learning tracks (DL2019, DL2020) [4, 5].”
Learning Passage Impacts for Inverted Indexes (2021, methodology): “To evaluate query processing effectiveness and efficiency, we compare with existing methods using the MSMARCO DevQueries,1 and we test all methods on the TREC 2019 [3] and TREC 2020 [2] queries from the TREC Deep Learning passage ranking track.”
Conformer-Kernel with Query Term Independence for Document Retrieval (2020, methodology): “In the inaugural year of the TREC Deep Learning track [Craswell et al., 2019], Transformer-based [Vaswani et al., 2017] ranking models demonstrated substantial improvements over traditional information retrieval (IR) methods.”
Overview of the TREC 2021 Deep Learning Track (2025, methodology): “As in previous years [Craswell et al., 2020, 2021a], we leverage the MS MARCO datasets [Bajaj et al., 2016] that made hundreds of thousands of human annotated training labels available for both passage and document ranking tasks.”
Modularized Transfomer-based Ranking Framework (2020, methodology): “Neural rankers based on Transformer architectures (Vaswani et al., 2017) ﬁne-tuned from BERT (Devlin et al., 2019) achieve current state-of-the-art (SOTA) ranking effectiveness (Nogueira and Cho, 2019; Craswell et al., 2019).”
Fine-Tuning LLaMA for Multi-Stage Text Retrieval (2023, methodology): “We evaluate the effectiveness of our models using the development split of the MS MARCO passage ranking task and the TREC DL19/DL20 passage ranking tasks [3, 4].”
A Modern Perspective on Query Likelihood with Deep Generative Retrieval Models (2021, positive): “To approach the second question, using the query-document pairs of the TREC-19 results, we average the term-level uncertainty values over all query terms that appear in a specific position of the queries.”
Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction (2022, methodology): “For training and in-domain evaluation we use the MSMARCOPassage (V1) collection [3] with the sparsely-judged MSMARCODEV query set of 6,980 queries (used in the leaderboard) as well as the densely-judged 97 query set of combined TREC-DL ’19 [7] and ’20 [8].”
On the Reliability of Test Collections for Evaluating Systems of Different Types (2020, methodology): “be a requirement based on the experience in other fields, and 2) The lack of test collections to evaluate the quality of neural models in a fair and reliable manner. The TREC 2019 Deep Learning Track [6] addressed these problems by releasing large-scale training data, as well as by developing reliab”