Evaluation measures for preference judgments

By Ben Carterette, Paul N. Bennett. 2008

47 citations recorded by the source corpus.

Subfields: Information Systems, Management Science and Operations Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Management Science and Operations Research

In Evaluation & Benchmarks

Citations in the corpus

Human Preferences as Dueling Bandits (2022, methodology): “Others propose and explore measures based on counts of correctly ordered items – according to the preference judgments — or measures that convert preferences to gain values for use with NDCG and other standard evaluation measures [6, 32, 35, 36].”
Preferences on a Budget: Prioritizing Document Pairs when Crowdsourcing Relevance Judgments (2022, background): “Previous research has also proposed measures to evaluate IR systems in the presence of incomplete preference judgments [4, 30].”
What Makes Them Click: Empirical Analysis of Consumer Demand for Search Advertising (2012, methodology): “Another frequently used frameworks is a “cascade model” (Craswell, Zoeter, Taylor, and Ramsey (2008), Papadimitriou and Zhang (2008)).”
Low-Cost Preference Judgment via Ties (2017, background): “In other words, the relative rankings between them are not considered in the evaluation as in [2].”
Shallow pooling for sparse labels (2021, background): “Extensive work by Carterette and various collaborator also indicate potential benefits from pairwise preference judgments [5, 45, 3, 4], including improved assessment speed and accuracy.”
Assessing Top- Preferences (2020, background): “In a recent SIGIR 2020 paper, Sakai and Zeng [32] propose and explore two broad families of measures intended to support preference judgments. e rst family is based on counts of concordant pairs, generalizing and extending ideas proposed by Cartere e et al. [7] and Cartere e and Benne [6]. e second ”
Evaluation Measures Based on Preference Graphs (2021, methodology): “This effort to establish preference based evaluation directly inspired the work of Sakai and Zeng [26] who greatly expand and explore the family of measures defined in Carterette and Bennett [6].”