On the effect of relevance scales in crowdsourcing relevance assessments for Information Retrieval evaluation

By Kevin Roitero, Eddy Maddalena, Stefano Mizzaro, Falk Scholer. 2021

24 citations recorded by the source corpus.

Subfields: Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

1. Nick Craswell
2. C. Clarke
3. Cheng-hsin Luo
4. Gianluca Demartini
5. Kevin Roitero
6. Xinyi Yan
7. Alessandro Checco
8. Stefano Mizzaro

Citations in the corpus

• Human Preferences as Dueling Bandits (2022, background): “Preferences also bypass the difficult problem of converting ordinal relevance grades into a ratio scale in order to compute NDCG and other standard evaluation measures [14, 18, 24, 28, 29].”
• Preferences on a Budget: Prioritizing Document Pairs when Crowdsourcing Relevance Judgments (2022, methodology): “• S100 [25], S4 [26], and S2 [26]: three datasets made by reassessing the same documents assessed in ME keeping the same experimental setting and using an evaluation scale with values in the [0, 100], [0, 3], and [0, 1] ranges, respectively.”