Report on the need for and provision of an 'ideal' information retrieval test collection

By Karen Sparck Jones, C. J. V. Rijsbergen. 1975

Subfields: Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

Learning a Health Knowledge Graph from Electronic Medical Records (2017, methodology): “This method, termed ‘pooling’, is frequently used in information retrieval settings 36 .”
Evaluating sentence-level relevance feedback for high-recall information retrieval (2018, methodology): “The dominant methodology of studies reported in the literature derives from Sparck Jones and Van Rijsbergen’s “ideal” test collection (Sparck Jones and Van Rijsbergen, 1975), in which the results of ad hoc searches for each of a set of topics within a dataset are compared to relevance labels for a s”
Unbiased Learning-to-Rank with Biased Feedback (2016, background): “First, since it is clearly impossible to get explicit judgments for all documents, pooling techniques [23] are used such that only the most promising documents are judged.”
Learning to model relatedness for news recommendation (2011, methodology): “To generate relatedness judgments for each seed article efficiently, we used the pooling technique [15] and asked assessors to judge only a pool of top-ranked documents returned by various runs (i.”
On the Reliability of Test Collections for Evaluating Systems of Different Types (2020, background): “[14] argued that test collections constructed using depth-k pooling [11] tend to be reasonably reusable and tend to be fair towards evaluating the quality of new systems.”
The MIR flickr retrieval evaluation (2008, background): “In this approach (e.g. [11]), only images are annotated that appear in the top N of most relevant images in the ranking of at least one approach participating in the benchmark.”
On Building Fair and Reusable Test Collections using Bandit Techniques (2018, background): “However, the TREC-8 collection was built using pooling [13] and contains about 85,000 human relevance judgments across 50 topics, more than twice the number of judgments that the Common Core track budget would afford.”
Information Retrieval Evaluation (2011, background): “The van Rijsbergen book [180]and the papers by Robertson [133] and Sanderson [147] are excellent reviews of the metrics in this period and are strongly recommended reading for both derivations of the metrics and comparisons across the various metrics that were proposed (including many not discussed ”
On the history of evaluation in IR (2008, background): “Bruce Croft, originally from Australia, and then taking a PhD under Keith van Rijsbergen in Cambridge in the late 1970s, moved on to the USA and joined the University of Massachusetts, where he has been ever since.”
Information retrieval system evaluation: effort, sensitivity, and reliability (2005, unknown): “15-25) than in the smaller subsets (5-10).”
Retrieval evaluation with incomplete information (2004, background): “For larger test collections such as the TREC collections that are built through the use of pooling [13], the assumption is not strictly true, but sufficient documents are judged such that completeness is closely approximated.”