Cumulated gain-based evaluation of IR techniques

By Kalervo Järvelin, Jaana Kekäläinen. 2002

5,590 citations recorded by the source corpus.

Subfields: Information Systems, Information Retrieval & Search, Evaluation & Benchmarks, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Information Retrieval & Search

In Evaluation & Benchmarks

In Signal Processing

Citations in the corpus

Findings of the 2014 Workshop on Statistical Machine Translation (2014, background): “…in terms of standard IR evaluation measures: precision at a cut-off of 5 and 10 documents (P@5, P@10), normalized discounted cumulative gain (J¨arvelin and Kek¨al¨ainen, 2002) at 5 and 10 documents (NDCG@5, NDCG@10), mean average precision (MAP) (Voorhees and Harman, 2005), precision…”
Variational Autoencoders for Collaborative Filtering (2018, methodology): “Recommender systems are often evaluated using ranking-based measures, such as mean average precision and normalized discounted cumulative gain [21].”
On the challenges of learning with inference networks on sparse, high-dimensional data (2017, neutral): “We report two ranking-based metrics averaged over all held-out users: Recall@ N and truncated normalized discounted cumulative gain (NDCG@ N ) (J¨arvelin & Kek¨al¨ainen, 2002).”
BLENDER: Enabling Local Search with a Hybrid Differential Privacy Model (2017, background): “NDCG: NDCG is a standard measure of search quality [20, 38] that explicitly takes the ordering of the items in a results list into account.”
Counterfactual Estimation and Optimization of Click Metrics for Search Engines (2014, background): “The dominant approach is to collect relevance judgments for a collection of query-document pairs, and compare different retrieval/ranking functions against metrics like mean average precision [1] and normalized discounted cumulative gains [17].”
Relevance-based Word Embedding (2017, methodology): “To evaluate the eectiveness of query expansion models, we report three standard evaluation metrics: mean average precision (MAP) of the top ranked 1000 documents, precision of the top 20 retrieved documents (P@20), and normalized discounted cumulative gain [13] calculated for the top 20 retrieved d”
Exploring and Predicting Transferability across NLP Tasks (2020, background): “…ρ of the source task with the highest absolute transfer gain from Section 2’s experiments, and (2) the Normalized Discounted Cumulative Gain (NDCG; Järvelin and Kekäläinen, 2002), a common information retrieval measure that evaluates the quality of the entire ranking, not just the rank of…”
Learning Answer-Entailing Structures for Machine Comprehension (2015, methodology): “We consider two evaluation metrics: accuracy (proportion of questions correctly answered) and NDCG 4 (J¨arvelin and Kek¨al¨ainen, 2002).”
Ranking with Fairness Constraints (2017, background): “Common examples are Discounted Cumulative Gain (DCG) [JK02] where f ( i ) = 1 log( i +1) for all i , and its variants.”
Understanding the Sparse Vector Technique for Differential Privacy (2016, methodology): “We point out that since the SVT and EM algorithms output an unorder set of queries, we cannot use discounted cumulative gain or normalized discounted cumulative gain [13], which discount the gain of a query based on which position it is outputted.”
Neural Ranking Models with Weak Supervision (2017, methodology): “To evaluate retrieval eectiveness, we report three standard evaluation metrics: mean average precision (MAP) of the top-ranked 1000 documents, precision of the top 20 retrieved documents (P@20), and normalized discounted cumulative gain (nDCG) [19] calculated for the top 20 retrieved documents (nDC”
Explainable Matrix Factorization for Collaborative Filtering (2016, methodology): “To assess the accuracy of EMF in terms of rating prediction, we used the Root Mean Squared Error (RMSE) and Normalized Discounted Cumulative Gain (nDCG@10) [3] metrics.”
Ranking with non-random missing ratings: influence of popularity and positivity on evaluation metrics (2012, background): “Recently, [13, 9] measured their ranking performance in terms on Normalized Discounted Cumulative Gain (NDCG, see [5]), ignoring”
Rank-Biased Precision Reloaded: Reproducibility and Generalization (2015, background): “RBP has been compared against P@10, P@R (precision at the recall-base), and Average Precision (AP) [3]; Normalized Discounted Cumulated Gain (nDCG) [9], and Reciprocal Rank (RR) [17], by considering two pool depths 100 (the original depth of TREC-05) and 10.”
Expected reciprocal rank for graded relevance (2009, background): “First, J¨arvelin and Kek¨al¨ainen [15] looked at graded relevance judgments and proposed the DCG metric that can exploit such judgments.”