Relevance assessment: are judges exchangeable and does it matter

By Peter Bailey, Nick Craswell, Ian Soboroff, Paul Thomas, A. D. Vries, Emine Yilmaz. 2008

216 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Management Science and Operations Research

Citations in the corpus

Up next: retrieval methods for large scale related video suggestion (2014, methodology): “Even for relatively objective evaluation tasks, such as document retrieval [6] the inter-judge agreement is low.”
Assessing Top- Preferences (2020, background): “Another large body of literature explores methods for crowdsourcing relevance judgments [1, 2, 26], including preference judgments.”
Information Retrieval: A Biomedical and Health Perspective (2020, methodology): “• Cross-Language Evaluation Forum, later named the Conference and Labs of the Evaluation Forum (CLEF)23—Focused on European languages, CLEF features a number of tracks that mimic those in TREC, such as ad hoc and Web searching [128].”
Information Retrieval Evaluation (2011, background): “at SIGIR 2008 [16] not only summarizes this work but adds new observations comparing the CSIRO assessors (the gold standard) in the TREC 2007 Enterprise track to a silver standard group (science communicators outside of CSIRO) and TREC participants (the bronze standard), with the latter being shown ”
Information Retrieval for E-Discovery (2013, unknown): “[6].”
Assessor Differences and User Preferences in Tweet Timeline Generation (2015, background): “In the same way that assessors disagree over relevance judgments (see [6] for a nice summary), humans also disagree about whether two pieces of text have the same semantic content.”
The anatomy of online deception: what makes automated text convincing? (2016, background): “To ﬁll this panel, three judges were highlighted by Bailey et al. [2] – for two distinct groups: social media daily but are not experienced with technology”
Methods for Evaluating Interactive Information Retrieval Systems with Users (2009, background): “Although traditionally the behavior of assessors has been of little concern, several researchers have started to investigate assessors [14, 226], thus treating them more as research subjects.”
Test Collection Based Evaluation of Information Retrieval Systems (2010, background): “’s work [19] on gold, silver, and bronze judgments suggested that such an approach to gathering judgments was not without its risks.”