IR evaluation methods for retrieving highly relevant documents

By Kalervo Järvelin, Jaana Kekäläinen. 2000

453 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search, Evaluation & Benchmarks, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Retrieval & Search

In Evaluation & Benchmarks

In Signal Processing

Citations in the corpus

Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval (2015, methodology): “For example, MT-DNN achieves NDCG@1=0.334, outperforming the current state-of-the-art single-task DSSM (0.327) and the classic methods like PLSA (0.308) and BM25 (0.305).”
Offline A/B Testing for Recommender Systems (2018, methodology): “To solve these pitfalls, people historically relied on offline experiments based on some rank-based metrics, such as NDCG [13], MAP [1] or Precision@K.”
Deep Exponential Families (2014, methodology): “Further, we also report the commonly-used multi-level ranking measure (un-truncated) NDCG [17] for all methods.”
NAIS: Neural Attentive Item Similarity Model for Recommendation (2018, methodology): “6% over FISM in terms of NDCG on MovieLens and Pinterest, respectively.”
Visually Explainable Recommendation (2018, methodology): “• NDCG: To assess if the items that a user has actually consumed are ranked in higher positions in the recommendation list, we use normalized discounted cumulative gain (NDCG) to evaluate ranking performance by taking the positions of the correct items into consideration [14].”
Controllable Multi-Interest Framework for Recommendation (2020, background): “Normalized Discounted Cumulative Gain (NDCG) takes the positions of correct recommended items into consideration [23].”
A Neural Autoencoder Approach for Document Ranking and Query Refinement in Pharmacogenomic Information Retrieval (2018, methodology): “(nDCG) (Järvelin and Kekäläinen, 2000).”
Task-Oriented Intrinsic Evaluation of Semantic Textual Similarity (2016, methodology): “• The normalized Cumulative Gain (nCG) can be used to evaluate the ranking quality of STS scores (Järvelin and Kekäläinen, 2000).”
Deep semantic ranking based hashing for multi-label image retrieval (2015, methodology): “Since the similarity level r can be bigger than one, we compute a weighted mAP using ACG: where Π( .”
Personalized Fashion Recommendation with Visual Explanations based on Multimodal Attention Network: Towards Visually Explainable Recommendation (2019, methodology): “For comparison, the widely used metrics, including F 1 [19], Hit Ratio (HR) and Normalized Discounted Cumulative Gain (NDCG) [17], are utilized to evaluate different models.”
Beyond Independent Relevance (2003, methodology): “The S-precision and S-recall metrics are broadly similar to the cumulated gain (CG) measure proposed by Jarvelin and Kekalianen [ 6 ].”
Sequential Recommendation with User Memory Networks (2018, background): “• NDCG: The normalized discounted cumulative gain, which evaluates ranking performance by taking the positions of correct items into consideration [15]: where I ( x ) is the indicator function as above, and Z is a normalization constant, which is the maximum possible value of DCG @ N .”
TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank (2018, background): “=1 be the document list for this query. Each query-document pair becomes an example x j= (q;d j). For each x j, we have a corresponding relevance label y jthat can be a binary or graded integer value [12]. Given a list of examples Xand a corresponding list of ground-truth relevance labels Y, ˇ is th”
Convex Calibration Dimension for Multiclass Loss Matrices (2014, methodology): “xpress them as non-negative losses by subtracting them from a suitable constant. 21 5.1 Normalized Discounted Cumulative Gain (NDCG) The NDCG loss is widely used in information retrieval applications [13]. Here Yis the set of rdimensional relevance vectors with say srelevance levels, Y= f0;1;:::;s 1”
Improving Web Search Ranking by Incorporating User Behavior Information (2019, methodology): “• NDCG at K: NDCG is a retrieval measure devised specifically for web search evaluation [10].”
Design of fusion technique-based mining engine for smart business (2015, methodology): “The TREC and the metrics [44] for estimating the performance are shown as follows, where P represents the precision value and R indicates the recall value.”
Leveraging Passage-level Cumulative Gain for Document Ranking (2020, methodology): “The cumulative gain (CG) has been used to evaluate ranking performance at both the query-level [14, 15] and multiple session-level [16].”