On information retrieval metrics designed for evaluation with incomplete relevance assessments

By Tetsuya Sakai, Noriko Kando. 2008

154 citations recorded by the source corpus.

Modern information retrieval (IR) test collections have grown in size, but the available manpower for relevance assessments has more or less remained constant. Hence, how to reliably evaluate and compare IR systems using incomplete relevance data, where many documents exist that were never examined by the relevance assessors, is receiving a lot of attention. This article compares the robustness of IR metrics to incomplete relevance assessments, using four different sets of graded-relevance test collections with submitted runs—the TREC 2003 and 2004 robust track data and the NTCIR-6 Japanese and Chinese IR data from the crosslingual task. Following previous work, we artificially reduce the original relevance data to simulate IR evaluation environments with extremely incomplete relevance data. We then investigate the effect of this reduction on discriminative power, which we define as the proportion of system pairs with a statistically significant difference for a given probability of Type I Error, and on Kendall’s rank correlation, which reflects the overall resemblance of two system rankings according to two different metrics or two different relevance data sets. According to these experiments, Q′, nDCG′ and AP′ proposed by Sakai are superior to bpref proposed by Buckley and Voorhees and to Rank-Biased Precision proposed by Moffat and Zobel. We also point out some weaknesses of bpref and Rank-Biased Precision by examining their formal definitions.

Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

Human Preferences as Dueling Bandits (2022, background): “The goal is to make the collection re-usable, so that future experiments conducted on the collection provide meaningful results, even if these new experimental runs surface unjudged documents in their top ranks[5, 31].”
Rank-Biased Precision Reloaded: Reproducibility and Generalization (2015, background): “Indeed, to the best of our knowledge, the only one other systematic analysis of RBP on different experimental collections is the one by [13], even if it does not concern the original RBP as defined in the reference paper under examination but its extension to multi-graded relevance judgements.”
Assessing efficiency–effectiveness tradeoffs in multi-stage retrieval systems without using relevance judgments (2015, background): “…temporally; P () is a query-free scoring regime based on story recency and story importance as determined by non-query features, such as user profile or past reading behavior; and the retrieval system is of the form P (D, k), where k ≈ 10, for example, is the number of stories to be presented.”
Test Collection Based Evaluation of Information Retrieval Systems (2010, background): “[142] N. Kando, K. Kuriyama, T. Nozue, K. Eguchi, H. Kato, and S. Hidaka, “Overview of IR tasks at the first NTCIR workshop,” in Proceedings of the First NTCIR Workshop on Research in Japanese Text Retrieval and Term Recognition, pp. 11–44, 1999.”
Efficient and effective spam filtering and re-ranking for large web datasets (2010, unknown): “Unadjudicated documents are elided, and P@10 is computed on the top-ranked 10 documents that are adjudicated (Sakai and Kando 2008).”