Overview of the TREC 2020 Deep Learning Track

By Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Fernando Campos, Ellen M. Voorhees. Text Retrieval Conference, 2021

531 citations recorded by the source corpus.

This is the second year of the TREC Deep Learning Track, with the goal of studying ad hoc ranking in the large training data regime. We again have a document retrieval task and a passage retrieval task, each with hundreds of thousands of human-labeled training queries. We evaluate using single-shot TREC-style evaluation, to give us a picture of which ranking methods work best when large data is available, with much more comprehensive relevance labeling on the small number of test queries. This year we have further evidence that rankers with BERT-style pretraining outperform other rankers in the large data regime.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/494e730f73035376736b36cecf2023ea6deb3cbe

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling (2021, methodology): “We use the MSMARCO-Passage [1] collection with the sparselyjudged MSMARCO-DEV query set of 6,980 queries (used in the leaderboard) as well as the densely-judged query sets of 43 and 54 queries derived from TREC-DL ’19 [6] and ’20 [7].”
Sparse, Dense, and Attentional Representations for Text Retrieval (2020, methodology): “To evaluate the performance of our best models in comparison to state-of-the-art works on large-scale retrieval and ranking, in § 9 we report results on a third group of tasks focusing on passage/document ranking: the passage and document-level MS MARCO retrieval datasets (Nguyen et al., 2016) also ”
Expansion via Prediction of Importance with Contextualization (2020, background): “mputing Machinery. https://doi.org/10.1145/1122445.1122456 Query | | Passage Importance Importance & Expansion Size of Vocabulary ( , ) Relevance Score Figure 1: Overview of EPIC. characteristics [1, 6]. However, these approaches come at a substantial computational burden, which can make them im”
TILDE: Term Independent Likelihood moDEl for Passage Re-ranking (2021, unknown): “We conduct our experiments on the MS MARCO dataset 4 and the two recent TREC Deep Learning tracks (DL2019, DL2020) [4, 5].”
Shallow pooling for sparse labels (2021, methodology): “They argue that this concern is addressed by experiments conducted for the TREC Deep Learning Track [9, 10], which follows the typical methodology of a TREC evaluation exercise.”
Overview of the TREC 2021 Deep Learning Track (2025, unknown): “The MS MARCO passage and document ranking v1 datasets are used in the current MS MARCO leaderboards [Lin et al., 2022, Craswell et al., 2021b, Lin et al., 2021a] and in TREC 2019 and TREC 2020.”
Understanding BERT Rankers Under Distillation (2020, methodology): “y. Following [6], we used MRR@10 to evaluate the ranking accuracy on this query set. TREC2019 DL Queries: this evaluation query set is the official evaluationquerysetusedintheTREC2019DeepLearningTrack[1]. It contains 43 queries with multiple relevant documents manually judged by NIST assessors with ”
Fine-Tuning LLaMA for Multi-Stage Text Retrieval (2023, methodology): “We evaluate the effectiveness of our models using the development split of the MS MARCO passage ranking task and the TREC DL19/DL20 passage ranking tasks [3, 4].”
Long Document Ranking with Query-Directed Sparse Transformer (2020, methodology): “This BERT ranker can be fine-tuned using relevance labels on (q, d) pairs, as simple as a classification task, and has achieved strong performances in various text ranking benchmarks (Bajaj et al., 2016; Craswell et al., 2020).”
TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime (2021, background): “We note that NIST makes the runs available, so they become part of the DL Track resources.”
Precise Zero-Shot Dense Retrieval without Relevance Labels (2022, unknown): “(To remind our readers, Contriever FT is in-domain supervisedly fine-tuned for TREC DL19/20).”
Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction (2022, methodology): “For training and in-domain evaluation we use the MSMARCOPassage (V1) collection [3] with the sparsely-judged MSMARCODEV query set of 6,980 queries (used in the leaderboard) as well as the densely-judged 97 query set of combined TREC-DL ’19 [7] and ’20 [8].”
Training Curricula for Open Domain Answer Re-Ranking (2020, background): “ing training sample difficulty, based on the rank or score of an unsupervised ranking model. 4 EXPERIMENTS Weconductexperimentsonthreelarge-scaleanswerrankingdatasets — namely TREC Deep Learning (DL) [6] (Section 4.1), TREC Complex Answer Retrieval (CAR) [10] (Section 4.2), and ANTIQUE [13] (Section”
MS MARCO: Benchmarking Ranking Models in the Large-Data Regime (2021, background): “[13, 14] have reported, several pretraining-based deep models finetuned on the MS MARCO training data achieve significant improvements over traditional IR methods in this setting.”
Optimizing Dense Retrieval Model Training with Hard Negatives (2021, methodology): “We list several representative results according to the TREC overview paper [4] and runs onMSMARCO [1] leaderboard, such as BM25 [31], the best traditional retrievalmethod, BERT weighted BM25 (DeepCT) [5].”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “For example, TREC 2019 and 2020 Deep Learning Track [31], [34] release a training set of 0.367 million queries (derived from MS MARCO [32]) for document retrieval task, which is significantly larger than those in previous retrieval tasks in TREC.”
SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking (2021, methodology): “The development set contains 6980 queries with similar labels, while the TREC DL 2019 evaluation set provides fine-grained annotations from human assessors for a set of 43 queries [3].”
Boosting Search Engines with Interactive Agents (2021, unknown): “As the external benchmark we use DPR (Karpukhin et al., 2020), a popular neural retriever based on dual encoders, the dominant architecture for deep learning-based ad hoc retrieval (Craswell et al., 2020).”