IR evaluation methods for retrieving highly relevant documents

By Kalervo Järvelin, Jaana Kekäläinen. 2000

1,437 citations recorded by the source corpus.

This paper proposes evaluation methods based on the use of non-dichotomous relevance judgements in IR experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is desirable from the user point of view in modern large IR environments. The proposed methods are (1) a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance, and (2) two novel measures computing the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. We then demonstrate the use of these evaluation methods in a case study on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents of various degrees of relevance. The test was run with a best match retrieval system (In-Query1) in a text database consisting of newspaper articles. The results indicate that the tested strong query structures are most effective in retrieving highly relevant documents. The differences between the query types are practically essential and statistically significant. More generally, the novel evaluation methods and the case demonstrate that non-dichotomous relevance assessments are applicable in IR experiments, may reveal interesting phenomena, and allow harder testing of IR methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Systems

In Information Retrieval & Search

Citations in the corpus

Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval (2015, methodology): “Our model satisﬁes both with Models NDCG@1 NDCG@3 NDCG@10 TF-IDF model (BM25) 0.305 0.328 0.388 Unigram Language Model (Zhai and Lafferty, 2001) 0.304 0.327 0.385 PLSA(Topic=100) (Hofmann, 1999; Gao et al., 2011) 0.305 0.335 0.402 PLSA(Topic=500) (Hofmann, 1999; Gao et al., 2011) 0.308 0.337 0.402 L”
Deep Exponential Families (2014, methodology): “Further, we also report the commonly-used multi-level ranking measure (un-truncated) NDCG [17] for all methods.”
A Meta-Learning Method to Select the Kernel Width in Support Vector Regression (2004, methodology): “This kind of top-N evaluation is commonly used in other fields, like Information Retrieval and recommender systems (Järvelin & Kekäläinen, 2000; Karypis, 2001), where both benefits and costs of solutions are usually taken into account.”
A Theoretical Analysis of NDCG Type Ranking Measures (2013, methodology): “The focus of this work is the Normalized Discounted Cumulative Gain (NDCG) which is one of the most popular evaluation measures in Web search [21, 22].”
Multi-task learning for boosting with application to web search ranking (2010, methodology): “The performance of various ranking models is measured on the test set using Discounted Cumulative Gain [10] at fifth position, which we refer to as DCG-5.”
Bagging gradient-boosted trees for high precision, low variance ranking models (2011, methodology): “For our experiments we work with three public data sets: TD2004 and MQ2007 from LETOR data sets [24] and the recently published MSLR-WEB10K data set from Microsoft Research [1].”
Adapting boosting for information retrieval measures (2010, positive): “We consider retrieval problems with five levels of relevance and we use the normalized discounted cumulative gain (NDCG) relevance measure (Jarvelin and Kekalainen 2000), which is suitable for non-binary relevance measures and which emphasizes the top returned results.”
Learning to Rank by Optimizing NDCG Measure (2009, methodology): “The NDCG value for ranking function F ( d, q ) is then computed as following: where Z k is the normalization factor [1].”
Learning to rank for information retrieval (2009, neutral): “Discounted cumulative gain (DCG): While the aforementioned measures are mainly designed for binary judgments, DCG [65, 66] can leverage the relevance judgment in terms of multiple ordered categories, and has an explicit position discount factor in its definition.”
Three-dimensional shape searching: state-of-the-art review and future trends (2005, neutral): “A large body of work has been done in the information retrieval (IR) area to evaluate information retrieval systems [140–142].”
From RankNet to LambdaRank to LambdaMART: An Overview (2010, background): “We end with the following note: [11] contains a relatively simple method for linearly combining two rankers such that out of all possible linear combinations , the resulting ranker gets the highest possible NDCG (in fact this can be done for any IR measure).”
Alors: An algorithm recommender system (2017, background): “V j ), such that matrices U and V maximize the Normalized Discounted Cumulative gain (NDCG, (J¨arvelin and Kek¨al¨ainen, 2000)), deﬁned as follows.”
Statistical Analysis of Bayes Optimal Subset Ranking (2008, methodology): “This metric, described in [17] as DCG (discounted cumulated gain), is one of the main metrics used in the evaluation of internet search systems, including the”