The Philosophy of Information Retrieval Evaluation

By Ellen M. Voorhees. 2001

526 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

Multi-Task Retrieval-Augmented Text Generation with Relevance Sampling (2022, background): “The coverage sparsity of relevance judgements of large collections and the resulting reliability issues are well studied, yet still a timely problem in the retrieval community (Zobel, 1998; Voorhees, 2001; Craswell et al., 2021; Hofstätter et al., 2021).”
Multi-Stage Document Ranking with BERT (2019, methodology): “This basic approach has received much interest in academia (Matveeva et al., 2006; Wang et al., 2011; Asadi and Lin, 2013; Chen et al., 2017; Mackenzie et al., 2018) as well as industry.”
Retrieval evaluation with incomplete information (2004, methodology): “The predominate research methodology for information retrieval system building is the Cranfield paradigm [17]: using test collections to compare the quality of different alternative retrieval methods.”
Crowdsourcing Subjective Tasks: The Case Study of Understanding Toxicity in Online Discussions (2019, background): “Research indicates that subjective assessment tasks are highly prone to disagreement between judges [43, 44].”
Information Retrieval for E-Discovery (2013, background): “As with other fields of information retrieval, research and development in e-discovery relies on the evaluation of retrieval effectiveness [156].”
Feeling lucky?: multi-armed bandits for ordering judgements in pooling-based evaluation (2016, methodology): “First, we evaluated several document selection strategies that have been proposed in the past: DocID : The standard evaluation sequence followed by TREC assessors [19].”
User performance versus precision measures for simple search tasks (2006, methodology): “However, mean average precision, while including a recall component, evaluates systems predom-1 inantly using precision [6, 19].”
An Information Retrieval Approach to Short Text Conversation (2014, methodology): “We employ a pooling strategy widely used in information retrieval for getting the instances to label (Voorhees, 2002).”
Expected reciprocal rank for graded relevance (2009, background): “This is largely due to the prevalence and popularity of Cranﬁeld and TREC-style evaluations [10, 25].”
TREC-COVID (2020, background): “7See summary papers such as [5] or TREC Overview papers for details on pooling.”
On information retrieval metrics designed for evaluation with incomplete relevance assessments (2008, methodology): “The Boostrap Sensitivity Method is closely related to the swap method proposed by Voorhees and Buckley (2002) which compares two systems using two different topics sets and examines whether the two outcomes are the same.”