Test Collection Based Evaluation of Information Retrieval Systems

By Mark Sanderson. Foundations and Trends in Information Retrieval, 2010

482 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/625686866788358b426da13cddd12b6862ce27d2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Management Science and Operations Research

Citations in the corpus

Learning to Rank (2005, unknown): “First, creating such datasets is expensive and therefore infeasible for smaller search engines, such as small web-store search engines [24].”
Reusing historical interaction data for faster online learning to rank for IR (2013, background): “is often expensive and, because annotators may interpret queries differently from actual users, it may not accurately capture users’ preferences [22].”
Click Models for Web Search (2015, methodology): “If we assume that there are five possible relevance labels, as is customary in web search settings [Sanderson, 2010], then this model has 11 parameters in total: five attractiveness, five satisfaction and one continuation parameter.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “discussion about IR evaluation, the readers are suggested to read more focused references [75], [76], [79], [80].”
Information Retrieval Evaluation (2011, methodology): “Readers particularly interested in the early history should also see [134, 147, 167].”
Modeling Diverse Relevance Patterns in Ad-hoc Retrieval (2018, methodology): “As revealed by the evaluation policy in TREC ad-hoc task [6, 32], “a document is judged relevant if any piece of it is relevant (regardless of how small the piece is in relation to the rest of the document) 1 ”.”
Information Retrieval for E-Discovery (2013, background): “• Sanderson [117] is a comprehensive history of test collection based evaluation in information retrieval, with special attention to studies of the reliability of the methodology.”
Online Evaluation for Information Retrieval (2016, methodology): “The Cranfield studies led to the collection-based experimentation paradigm [Sanderson, 2010] and to the TREC tradition [Voorhees and Harman, 2005].”
Continuous Online Evaluation of Recommendation Strategies in Social Science Academic Search (2026, unknown): “However, prior research has highlighted several limitations of static test collections, particularly their inability to reflect evolving document corpora, changing user needs, and advances in retrieval methods [66, 35, 38].”