Pairwise Crowd Judgments: Preference, Absolute, and Ratio

By Ziying Yang, Alistair Moffat, Andrew Turpin. 2018

16 citations recorded by the source corpus.

Relevance judgments are conventionally formed by small numbers of experts using ordinal relevance scales defined by two or more relevance categories. Such judgments often contain many ties: documents in the same category that cannot be separated by relevance. Here we explore the use of crowd-sourcing and combined three-way relevance assessments using pairwise preference, absolute relevance, and relevance ratio, with forced choice testing and embedded quality control processes, seeking to reduce assessment ties, and to increase judgment consistency. In particular, the crowd-sourced judgments from these three approaches were normalized into numeric relevance scores, and compared against judgments arising via three previous techniques: NIST binary; Sormunen; and magnitude estimation. The relationship between generated judgment reliability and number of document pairs assessed was also explored, as was the effect that factors such as document length, topic difficulty, number of documents judged, and assessment time, have on assessment reliability. Lastly, we investigate the extent to which the methodology used to collect judgments affects the ability of an experiment to discriminate between IR systems.

Subfields: Artificial Intelligence, Computer Science Applications, Information Systems, Statistics and Probability, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Information Systems

In Statistics and Probability

In Human-Computer Interaction

Citations in the corpus

Evaluation Measures Based on Preference Graphs (2021, background): “Preferences may provide a more solid foundation for offline evaluation [3, 7, 11, 17, 18, 22, 26, 29, 30].”
Preferences on a Budget: Prioritizing Document Pairs when Crowdsourcing Relevance Judgments (2022, methodology): “We validated this method using the dataset by Yang et al. [31] that started from native pairwise comparisons: the agreement between the BTL and the pairwise comparisons was higher than the proposed linearization method (81 .”