Good Evaluation Measures based on Document Preferences

By Tetsuya Sakai, Zhaohao Zeng. 2020

13 citations recorded by the source corpus.

For offline evaluation of IR systems, some researchers have proposed to utilise pairwise document preference assessments instead of relevance assessments of individual documents, as it may be easier for assessors to make relative decisions rather than absolute ones. Simple preference-based evaluation measures such as ppref and wpref have been proposed, but the past decade did not see any wide use of such measures. One reason for this may be that, while these new measures have been reported to behave more or less similarly to traditional measures based on absolute assessments, whether they actually align with the users' perception of search engine result pages (SERPs) has been unknown. The present study addresses exactly this question, after formally defining two classes of preference-based measures called Pref measures and Δ-measures. We show that the best of these measures perform at least as well as an average assessor in terms of agreement with users' SERP preferences, and that implicit document preferences (i.e., those suggested by a SERP that retrieves one document but not the other) play a much more important role than explicit preferences (i.e., those suggested by a SERP that retrieves one document above the other). We have released our data set containing 119,646 document preferences, so that the feasibility of document preferenced-based evaluation can be further pursued by the IR community.

Subfields: Information Systems, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Evaluation & Benchmarks

Citations in the corpus

Human Preferences as Dueling Bandits (2022, methodology): “Many papers assume that all pairs in a judging pool will be judged once [2, 7, 32].”
Evaluation Measures Based on Preference Graphs (2021, methodology): “Recent work by Sakai and Zeng [25, 26] took a”
Shallow pooling for sparse labels (2021, background): “Sakai and Zeng [32] extend and explore a family of measures first defined by Carterette et al.”
Assessing Top- Preferences (2020, background): “Preference judgments [7, 16, 30, 32, 38] have long been proposed as an alternative to graded relevance judgments for the oine evaluation of search and related ranking tasks, including recommendation and question answering.”
Preferences on a Budget: Prioritizing Document Pairs when Crowdsourcing Relevance Judgments (2022, background): “Sakai and Zeng [28, 29] study how different evaluation measures (both classic / pointwise and novel / pairwise ones) are aligned with web search result preferences; their results show that although some variants of the novel pairwise metrics are promising in identifying statistically significant…”