Estimating average precision with incomplete and imperfect judgments

By Emine Yilmaz, J. Aslam. International Conference on Information and Knowledge Management, 2006

365 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/421f60f316373875f8887bf58476b7feff4716bd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

TRECVID 2010 – An Introduction to the Goals , Tasks , Data , Evaluation Mechanisms , and Metrics (2010, methodology): “Work at Northeastern University (Yilmaz & Aslam, 2006) has resulted in methods for estimating standard system performance measures using relatively small samples of the usual judgment sets so that larger numbers of features can be evaluated using the same amount of judging effort.”
Medical Question Answering for Clinical Decision Support (2016, methodology): “…by TREC: (1) the inferred Average Precision (iAP), wherein retrieved articles were randomly sampled and the Average Precision was calculated as in [36]; (2) the inferred Normalized Discounted Cumulative Gain (iNDCG), wherein retrieved articles were randomly sampled and the NDCG was calculated as…”
Test Collection Based Evaluation of Information Retrieval Systems (2010, background): “[51] B. Carterette, V. Pavlu, E. Kanoulas, J. A. Aslam, and J. Allan, “Evaluation over thousands of queries,” in Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 651–658, New York, NY, USA: ACM, 2008.”
Information Retrieval Evaluation (2011, methodology): “This third pool was used for a new evaluation measure inferred average precision, infAP [202].”
Information Retrieval for E-Discovery (2013, methodology): “Evaluation by sampled assessments has been explored extensively in other applications of information retrieval [32, 153, 170].”
A new rank correlation coefficient for information retrieval (2008, methodology): “ANew Rank Correlation Coef.cient for Information Retrieval Emine Yilmaz* Javed A. Aslam * Stephen Robertson* eminey@microsoft.com jaa@ccs.neu.edu ser@microsoft.com College of Computer and Information Science *Microsoft Research Northeastern University 7 JJ Thomson Avenue 360 Huntington Ave, #202 W”
Relevance assessment: are judges exchangeable and does it matter (2008, methodology): “The measures of correlation and RMS errors originally reported [24] are based on scores averaged over all topics.”
The THUMOS challenge on action recognition for videos "in the wild" (2016, neutral): “Since only a sub-set of test data is annotated, inferred Average Precision is used for evaluation (infAP) ( Yilmaz and Aslam, 2006 ) of each concept.”
On information retrieval metrics designed for evaluation with incomplete relevance assessments (2008, background): “However, it appears that it was never properly examined until Yilmaz and Aslam (2006) and Sakai (2007a) rediscovered it.”
TRECVID 2011 - An Overview of the Goals, Tasks, Data,Evaluation Mechanisms, and Metrics (2011, methodology): “Work at Northeastern University (Yilmaz & Aslam, 2006) has resulted in methods for estimating standard system performance measures using relatively small samples of the usual judgment sets so that larger numbers of features can be evaluated using the same amount of judging effort.”
Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media (2017, background): “This is similar to how missing relevance judgements are handled in the Information Retrieval literature [48].”