Variations in relevance judgments and the measurement of retrieval effectiveness

By Ellen M. Voorhees. 1998

Subfields: Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

The TREC question answering track (2001, neutral): “The results of the TREC-8 assessment process demonstrated that assessors do have legitimate differences of opinion as to what constitutes an acceptable answer even for these deliberately constrained questions (Voorhees and Tice 2000b).”
Information Retrieval: A Biomedical and Health Perspective (2020, background): “Equally important, however, they defined a number of additional requirements and other statements for biomedical publishing [127]:”
The Effects of Human Variation in DUC Summarization Evaluation (2004, background): “Whereas the judgment effect does not change the relative ranking of systems in the TREC information retrieval task (Voorhees, 1998), the task in coverage evaluation is much more cognitively difﬁcult and needed further exploration.”
The Notion of Relevance in Information Science: Everybody knows what relevance is. But, what is it really? (2016, background): “Ø Voorhees (2000) (also in Voorhees and Harman, 2005, pp.”
Test Collection Based Evaluation of Information Retrieval Systems (2010, methodology): “Voorhees conducted a larger scale study using TRECs 4 and 6 run data [268], for which multiple relevance assessments, qrels, were available.”
Relevance assessment: are judges exchangeable and does it matter (2008, background): “judges for TREC-4, and NIST and University of Waterloo judges for a set of overlapped judgements on part of the TREC-6 dataset [22].”
Novelty and diversity in information retrieval evaluation (2008, background): “Human assessors are known to be inconsistent in their judgments [27].”
Assessor Differences and User Preferences in Tweet Timeline Generation (2015, methodology): “In the same way that Voorhees [31] demonstrated the stability of system rankings with respect to assessors’ divergent document-level relevance judgments, the implicit assumption is that for these semantic clustering tasks, assessor differences don’t matter if our goal is to induce a set of system co”
Document categorization in legal electronic discovery: computer classification vs. manual review (2010, result): “Other studies outside of TREC Legal have found similar levels of (dis)agreement (e.g., Barnett, Godjevac, Renders, Privault, Schneider, &Wickstrom, 2009; Borko, 1964; Tonta, 1991; Voorhees, 1998).”
Information Retrieval Evaluation (2011, unknown): “Voorhees [185] investigated this by using different subsets of the relevance judgments from TREC-4.”
Evaluating sentence-level relevance feedback for high-recall information retrieval (2018, background): “The synthetic labels used to simulate user feedback were imperfect, but of comparable quality, according to recall and precision, to what has been observed for human users (Voorhees, 2000).”
How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation (2010, methodology): “Voorhees [28] analyses the influence of changes in relevance judgments on the evaluation of retrieval results utilizing the Kendall τ correlation coefficient.”
Less is More Probabilistic Models for Retrieving Fewer Relevant Documents (2006, unknown): “For TREC-4 and TREC-6, multiple independent annotators were asked to make relevance judgments for the same set of topics, and over the same corpus [18, 7, 17].”
Overview of TREC 2021 (2021, unknown): “Despite the idiosyncratic nature of relevance, test collections are useful abstractions because the comparative effectiveness of different retrieval methods is stable in the face of changes to the relevance judgments [9].”