On Crowdsourcing Relevance Magnitudes for Information Retrieval Evaluation

By Eddy Maddalena, Stefano Mizzaro, Falk Scholer, Andrew Turpin. 2017

96 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Science Applications, Information Systems, Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Information Systems

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

Human Preferences as Dueling Bandits (2022, background): “Preferences also bypass the difficult problem of converting ordinal relevance grades into a ratio scale in order to compute NDCG and other standard evaluation measures [14, 18, 24, 28, 29].”
Shallow pooling for sparse labels (2021, background): “[22] estimated relevance magnitudes by showing assessors sequences of eight documents, so that comparisons could be made between them.”
On Fine-Grained Relevance Scales (2018, methodology): “In terms of quality checks, we performed the same checks as [15] (i.”
Evaluation Measures Based on Preference Graphs (2021, methodology): “b) Grades must be mapped to gain values to compute these evaluation measures, but no mapping is known to be suitable for all contexts, and there is no established method for developing and validating the suitability of these mappings in a particular context [21].”
Preferences on a Budget: Prioritizing Document Pairs when Crowdsourcing Relevance Judgments (2022, methodology): “It is easier for assessors to make relative judgments stating that one document is more relevant than another rather than making absolute judgments of relevance [21] and preference judgments have shown to be more reliable than absolute (i.”
Assessing Top- Preferences (2020, background): “ed on top-k learning-to-rank methods [25, 28, 34]. Another large body of literature explores methods for crowdsourcing relevance judgments [1, 2, 26], including preference judgments. Maddalena et al. [27] crowdsource relevance magnitudes through a process in which assessors view a series of document”