TREC-COVID: rationale and structure of an information retrieval shared task for COVID-19

By Kirk Roberts, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, Kyle Lo, Ian Soboroff, Ellen M. Voorhees, Lucy Lu Wang, William R. Hersh. 2020

130 citations recorded by the source corpus.

Abstract TREC-COVID is an information retrieval (IR) shared task initiated to support clinicians and clinical research during the COVID-19 pandemic. IR for pandemics breaks many normal assumptions, which can be seen by examining 9 important basic IR research questions related to pandemic situations. TREC-COVID differs from traditional IR shared task evaluations with special considerations for the expected users, IR modality considerations, topic development, participant requirements, assessment process, relevance criteria, evaluation metrics, iteration process, projected timeline, and the implications of data use as a post-task test collection. This article describes how all these were addressed for the particular requirements of developing IR systems under a pandemic situation. Finally, initial participation numbers are also provided, which demonstrate the tremendous interest the IR community has in this effort.

Subfields: Molecular Biology, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

Citations in the corpus

MS MARCO: Benchmarking Ranking Models in the Large-Data Regime (2021, methodology): “This prompted the creation of the Covid-19 Open Research Dataset (CORD-19) [81] and the TREC-COVID [60, 73] benchmarking effort”
SLEDGE: A Simple Yet Effective Baseline for Coronavirus Scientific Knowledge Search (2020, methodology): “We evaluate the performance of S LEDGE using the TREC-COVID Information Retrieval Challenge dataset (round 1) (Roberts et al., 2020).”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “Information retrieval Web MS MARCO [32] 502,939 532,761 6,980 6,837 8,841,823 Web mMARCO [44] 808,731 – 101,093 – 8,841,823 News TREC-NEWS [43] – – – 57 594,977 Biomedical TREC-COVID [45] – – – 50 171,332 Biomedical NFCorpus [46] 5,922 110,575 324 323 3,633 Twitter Signal-1M [47] – – – 97 2,866,316 ”
CORD-19: The Covid-19 Open Research Dataset (2020, background): “Each paper can have unique identiﬁers such as DOIs, PubMed Central IDs, PubMed IDs, the WHO Covidence #, 4 or MAG identiﬁers (Shen et al., 2018).”
Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Dataset (2020, background): “The TREC-COVID challenge, which ran from midApril to late-August 2020, provided an opportunity for researchers to study methods for quickly standing up information access systems, both in response to the current pandemic and to prepare for similar future events (Voorhees et al., 2020; Roberts et al.”