Expected reciprocal rank for graded relevance

By Olivier Chapelle, Donald Metlzer, Ya Zhang, Pierre Grinspan. 2009

926 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Signal Processing

Citations in the corpus

Off-policy evaluation for slate recommendation (2016, background): “We consider two slate rewards: NDCG from Example 3, and the expected reciprocal rank, ERR [9], which does not satisfy linearity, and is defined as ERR(x, s) := ∑` r=1 1 r ∏r−1 i=1 (1−R(si))R(sr) , where R(a) = 2 −1 2maxrel with maxrel = 4.”
Evaluation Measures Based on Preference Graphs (2021, methodology): “PGC outperforms wpref6 on the Derived 2019 judgements, and would be tied for fifth (along with wpref2) on the Real judgments. nDCG outperforms all other measures on agreement, even ERR, which has outperformed nDCG on click metrics [9], but does poorly on this experiment, placing well below most pref”
Towards Meaningful Statements in IR Evaluation: Mapping Evaluation Measures to Interval Scales (2021, background): “In short, Ferrante et al. found that, for a single topic: • set-based evaluation measures: – binary relevance: precision, recall, F-measure are interval scales; – multi-graded relevance: Generalized Precision (gP) and Generalized Recall (gR) are interval scales only if the relevance degrees are on a”
Learning to rank code examples for code search engines (2017, background): “k-ERR (Chapelle et al. 2009) is defined as the expectation of the reciprocal of the position k at which a developer stops when looking through a result list, which can be denoted as ∑k i=1 1 k P (user stops at position k).”
Query Driven Algorithm Selection in Early Stage Retrieval (2017, background): “One obvious question arises when trying to measure trade-offs in multi-stage retrieval systems – how can we quantify the impact on effectiveness when modifying different components of the search system? One approach is to simply make changes to the system, and re-compute a standard information retri”
Learning to Reweight Terms with Distributed Representations (2015, methodology): “For web collections such as GOV2 and ClueWeb09B, where graded relavence judgments are available, we also report Normalized Discounted Cumulative Gain (NDCG) at 20 and Expected Reciprocal Rank (ERR)[4] at 20 values using the gdeval.pl tool provided by TREC.”
Rank and relevance in novelty and diversity metrics for recommender systems (2011, methodology): “For instance, drawing from the ERR metric scheme [8]: | , ~ 2 , − 1 2 (13)”
Learning Concept Embeddings for Query Expansion by Quantum Entropy Minimization (2014, background): “Hence, we also report NDCG@10 and the recent ERR@10, which correlates better with click metrics than other editorial metrics (Chapelle et al. 2009).”
Calibrated recommendations (2018, background): “This idea has resulted in the ranking metric called expected reciprocal rank (ERR) [7], as well as in approaches for generating a more diverse ranked list [20, 27].”
A Theoretical Analysis of NDCG Type Ranking Measures (2013, background): “There seem to be many ways to deﬁne ranking measures and several evaluation measures have been proposed [11, 32, 4, 1, 28].”
TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank (2018, methodology): “For example, widely-utilized metrics such as Normalized Discounted Cumulative Gain (NDCG) [20], Expected Reciprocal Rank (ERR) [8], Mean Reciprocal Rank (MRR) [14], Mean Average Precision (MAP), and Average Relevance Position (ARP) [48] are designed to emphasize the items that are ranked higher in t”
Learning Scoring Functions with Order-Preserving Losses and Standardized Supervision (2011, methodology): “Both have a highly complex structure, and there is no consistent surrogate loss w.r.t. the ERR or the AP when considering order-preserving losses: Theorem 3 There is no standardization function for the ERR and for the AP.”
Content-Based Weak Supervision for Ad-Hoc Re-Ranking (2017, methodology): “Akin to [6, 7], dierent competing models are used to re-rank the search results from a query-likelihood retrieval model (QL), by examining the quality of the re-ranked search results in terms of ERR@20 [4] and nDCG@20 [8].”
Document Comprehensiveness and User Preferences in Novelty Search Tasks (2015, methodology): “Two of the most widely used metrics that make use of graded judgments are nDCG [10] and ERR [7].”
PACRR: A Position-Aware Neural IR Model for Relevance Matching (2017, methodology): “ERR@20 (Chapelle et al., 2009) and nDCG@20 (J¨arvelin and Kek¨al¨ainen, 2002) are employed as evaluation measures, and both are computed with the script from T REC 6 .”
Assessing Top- Preferences (2020, methodology): “is approach contrasts with the more established approach of converting independently assigned relevance grades into gain values to compute measures such as NDCG [4, 20] and ERR [10].”
From RankNet to LambdaRank to LambdaMART: An Overview (2010, background): “However for IR measures like NDCG and ERR that emphasize the top few results, this is not what we want.”