How reliable are the results of large-scale information retrieval experiments?

By Justin Zobel. 1998

Two stages in measurement of techniques for informationretrieval are gathering of documents for relevance assessment anduse of the assessments to numerically evaluate effectiveness. Weconsider both of these stages in the context of the TRECexperiments, to determine whether they lead to measurements thatare trustworthy and fair. Our detailed empirical investigation ofthe TREC results shows that the measured relative performance ofsystems appears to be reliable, but that recall is overestimated:it is likely that many relevant documents have not been found. Wepropose a new pooling strategy that can significantly in- creasethe number of relevant documents found for given effort, withoutcompromising fairness.

Subfields: Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Retrieval & Search

Citations in the corpus

Multi-Task Retrieval-Augmented Text Generation with Relevance Sampling (2022, background): “The coverage sparsity of relevance judgements of large collections and the resulting reliability issues are well studied, yet still a timely problem in the retrieval community (Zobel, 1998; Voorhees, 2001; Craswell et al., 2021; Hofstätter et al., 2021).”
Liberal relevance criteria of TREC -: counting on negligible documents? (2002, methodology): “In TREC, for example, the pooling d to guarantee that the list of documents is as comprehensive as possible [13, 14].”
TU Wien at TREC DL and Podcast 2021: Simple Compression for Dense Retrieval (2021, background): “Our first participation at the TREC-Podcast track gives us an excellent opportunity to study the effects of pool bias [13, 14, 20, 24, 27], and discuss its impact on take away messages: RQ-P-4 What can we learn about out-of-pool evaluation for DR by comparing 2020 (out-of-pool) with 2021 (in-pool)…”
Improvements that don't add up: ad-hoc retrieval results since 1998 (2009, methodology): “The validity of the new technique is tested by comparing its score to the baseline, and any apparent improvement is then tested with a statistical signiﬁcance test [Zobel, 1998, Sanderson and Zobel, 2005, Smucker et al., 2007].”
Overview of TREC 2021 (2021, unknown): “A variant of the ”Leave-Out-Uniques” test [10] that accommodates the use of CAL shows that some Document Ranking task submissions experience large swings in evaluated effectiveness if their team’s uniquely-retrieved relevant documents are removed from the top-10 pools.”
Methods for Identifying Versioned and Plagiarized Documents (2003, background): “tems [19]; however, they are not well-suited to this task.”
User performance versus precision measures for simple search tasks (2006, background): “While some relevant documents may thus remain unidentiﬁed, this approach has been demonstrated to work eﬀectively for the comparison of diﬀerent retrieval systems [20].”
Learning a Health Knowledge Graph from Electronic Medical Records (2017, methodology): “…the efficacy of information retrieval models, Zobel found that although the pooling method is overly optimistic in its evaluation of recall, it provides a reliable measure of precision and does not unjustly bias one method over another, providing a “fair basis of measurement of new systems” 37 .”
Overview of the TREC 2019 deep learning track (2020, background): “Leave-Out-Uniques (LOU) tests [Buckley et al., 2007, Zobel, 1998] are a way of analyzing the reusability of a collection.”
Concept-Based Information Retrieval Using Explicit Semantic Analysis (2011, methodology): “However, research has shown that the use of pooling could discriminate against a new method that is based on novel principles [Zobel 1998], and it has been recommended that researchers consider the number of unjudged documents being fetched as an indication that performance is probably being underes”
On the Reliability of Test Collections for Evaluating Systems of Different Types (2020, background): “[14] argued that test collections constructed using depth-k pooling [11] tend to be reasonably reusable and tend to be fair towards evaluating the quality of new systems.”
Information retrieval system evaluation: effort, sensitivity, and reliability (2005, methodology): “This we took to be evidence that the dependency between large topic sets has a substantial influence on the error rate measured by the methodology used by Zobel (1998), by Voorhees and Buckley (2002), and used so far in this paper.”
Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling (2021, background): “Naturally, most queries have fewer relevant passages than non-relevant ones [45].”