Alternatives to Bpref

By Tetsuya Sakai. 2007

184 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Recommendations as Treatments: Debiasing Learning and Evaluation (2016, methodology): “Rankings for DCG@50 were created by sorting items according to ˆ Y i for each user.”
Test Collection Based Evaluation of Information Retrieval Systems (2010, background): “Sakai [207] tested a number of alternatives including one that considered graded judgments (RPref).”
Condensed List Relevance Models (2015, methodology): “We refer to this method as the condensed list relevance model (CLRM) because we remove unscored documents in the same way condensed list evaluation metrics remove unjudged documents [11].”
On information retrieval metrics designed for evaluation with incomplete relevance assessments (2008, methodology): “However, it appears that it was never properly examined until Yilmaz and Aslam (2006) and Sakai (2007a) rediscovered it.”
A simple and efficient sampling method for estimating AP and NDCG (2008, background): “INTRODUCTION We consider the problem of large scale retrieval evaluation, in particular, retrieval evaluation with incomplete relevance judgments.”
Generalizing Translation Models in the Probabilistic Relevance Framework (2016, methodology): “Therefore, in order to provide a fairer evaluation framework, we consider MAP and NDCG over the condensed lists [32] 3 .”
Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction (2022, methodology): “We follow Sakai [40] to correct our metric measurements for pool bias by observing only measuring effectiveness on judged passages, which means removing all retrieved passages that are not judged and then re-assigning the ranks of the remaining ones.”
Expected reciprocal rank for graded relevance (2009, background): “Finally, various researchers have showed how the completeness assumption could be relaxed by inferring the relevance of missing judgments in various ways [3, 5, 7], ignoring unjudged documents [21], or intelligently choosing which un-judged documents to obtain judgments for [6].”
TU Wien at TREC DL and Podcast 2021: Simple Compression for Dense Retrieval (2021, methodology): “…for nDCG@10 Run TREC-Podcast 2020 (out-of-pool) TREC-Podcast 2021 (in-pool) J@10 nDCG@10 J@30 nDCG@30 nDCG–J@10 bpref J@10 nDCG@10 J@30 nDCG@30 bpref In Table 7, we present the evaluation results for our 2021-runs for both TREC-Podcast years using common approaches to tackle pool bias [2, 19].”
Assessing efficiency–effectiveness tradeoffs in multi-stage retrieval systems without using relevance judgments (2015, background): “One way of handling unjudged documents is to remove them from the ranking and form a condensed sequence [21]; the more common mechanism is to assume that unjudged documents are non-relevant.”
Shallow pooling for sparse labels (2021, background): “Unjudged items are typically assumed to provide zero gain [31].”