On Fine-Grained Relevance Scales

By Kevin Roitero, Eddy Maddalena, Gianluca Demartini, Stefano Mizzaro. 2018

28 citations recorded by the source corpus.

In Information Retrieval evaluation, the classical approach of adopting binary relevance judgments has been replaced by multi-level relevance judgments and by gain-based metrics leveraging such multi-level judgment scales. Recent work has also proposed and evaluated unbounded relevance scales by means of Magnitude Estimation (ME) and compared them with multi-level scales. While ME brings advantages like the ability for assessors to always judge the next document as having higher or lower relevance than any of the documents they have judged so far, it also comes with some drawbacks. For example, it is not a natural approach for human assessors to judge items as they are used to do on the Web (e.g., 5-star rating). In this work, we propose and experimentally evaluate a bounded and fine-grained relevance scale having many of the advantages and dealing with some of the issues of ME. We collect relevance judgments over a 100-level relevance scale (S100) by means of a large-scale crowdsourcing experiment and compare the results with other relevance scales (binary, 4-level, and ME) showing the benefit of fine-grained scales over both coarse-grained and unbounded scales as well as highlighting some new results on ME. Our results show that S100 maintains the flexibility of unbounded scales like ME in providing assessors with ample choice when judging document relevance (i.e., assessors can fit relevance judgments in between of previously given judgments). It also allows assessors to judge on a more familiar scale (e.g., on 10 levels) and to perform efficiently since the very first judging task.

Subfields: Computer Science Applications, Information Systems, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems

In Information Retrieval & Search

Citations in the corpus

Human Preferences as Dueling Bandits (2022, background): “Preferences also bypass the difficult problem of converting ordinal relevance grades into a ratio scale in order to compute NDCG and other standard evaluation measures [14, 18, 24, 28, 29].”
Pairwise Crowd Judgments: Preference, Absolute, and Ratio (2018, methodology): “For S100 and binary judging, the documents are assessed against an absolute scale, without forcing the assessor to consider documents relative to one another.”