A simple and efficient sampling method for estimating AP and NDCG

By Emine Yilmaz, Evangelos Kanoulas, J. Aslam. 2008

328 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Recommendations as Treatments: Debiasing Learning and Evaluation (2016, methodology): “Note that rankings of recommendations are similar to the set-based recommendation described above, and measures like Discounted Cumulative Gain (DCG), DCG@k, Precision at k (PREC@k), and others (Aslam et al., 2006; Yilmaz et al., 2008) also ﬁt in this setting.”
Relevance assessment: are judges exchangeable and does it matter (2008, methodology): “has recently been extended for non-random selection of documents [25].”
A Comparison of Word Embeddings for the Biomedical Natural Language Processing (2018, methodology): “Four official metrics, namely Inferred Normalized Discounted Cumulated Gain (infNDCG)[53], Inferred Average Precision (infAP)[53], Precision at 10 (P@10), and Mean Average Precision (MAP), were utilized to measure the IR performance.”
Counterfactual Evaluation and Learning for Search, Recommendation and Ad Placement (2016, background): “However, this research has happened in many rather disjoint fields, including statistics [16], economics [1], reinforcement learning [20], contextual bandit learning [21], Monte Carlo estimation [17] and information retrieval [23, 5, 4].”
TREC 2017 Common Core Track Overview (2017, methodology): “The experiments compared qrels built using depth-k pooling, qrels built through sampling in support of inferred measures [Yilmaz et al., 2008], and qrels built using the MaxMean bandit method for a range of values of k .”
On Building Fair and Reusable Test Collections using Bandit Techniques (2018, methodology): “The extended inferred measures sampling methodology [17] is an alternative collection-building technique that uses fewer judgments than pooling and calculates estimates of standard evaluation measures whose expected value will match the true value of the standard measure.”
Diagnosing New York city's noises with ubiquitous data (2014, methodology): “We then rank these locations in terms of the real noise level and the inferred values respectively, measuring the closeness of the two ranks using NDCG (Normalized Discounted cumulative gain) [22].”
Preferences on a Budget: Prioritizing Document Pairs when Crowdsourcing Relevance Judgments (2022, background): “Our next steps include the generalization of these findings to other IR evaluation setups, using native pairwise judgments as input, and considering more advanced pair selection strategies (e.g., based on stratified sampling rather then fix-depth as done for pointwise judgment pooling [18, 33]).”
Toward interactive training and evaluation (2011, methodology): “Stratified sampling using model predictions has also been applied to address specific evaluation problems in information retrieval [25].”
Overview of the TREC 2014 Clinical Decision Support Track (2014, methodology): “All of the 102 runs contributed to the judgment sets, which were constructed to be compatible with computing inferred retrieval measures (Yilmaz et al., 2008).”
Medical Question Answering for Clinical Decision Support (2016, methodology): “…(2) the inferred Normalized Discounted Cumulative Gain (iNDCG), wherein retrieved articles were randomly sampled and the NDCG was calculated as per [37]; (3) the R -Precision, which measures the precision of the highest R -retrieved documents, where R is the total number of relevant documents for…”