A statistical method for system evaluation using incomplete judgments

By J. Aslam, Virgil Pavlu, Emine Yilmaz. 2006

199 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Recommendations as Treatments: Debiasing Learning and Evaluation (2016, methodology): “Note that rankings of recommendations are similar to the set-based recommendation described above, and measures like Discounted Cumulative Gain (DCG), DCG@k, Precision at k (PREC@k), and others (Aslam et al., 2006; Yilmaz et al., 2008) also ﬁt in this setting.”
Information Retrieval Evaluation (2011, methodology): “The documents that were judged for each topic did not come from traditional pooling but were sampled using one of two different methods, Minimal Test Collections (MTC) [41] or statAP [12, 13], using the appropriate evaluation measures for the results.”
A comparison of user and system query performance predictions (2010, methodology): “In the case of ClueWeb09, AP and P@30 were estimated according to [2], which is the evaluation measure/procedure for this corpus at TREC.”
Test Collection Based Evaluation of Information Retrieval Systems (2010, background): “[13] described experiments with pool sampling showing that one could sample as little as 4% of a TREC ad hoc pool and still produce accurate results.”
Bias and the limits of pooling for large collections (2007, background): “The final two sections offer possible ways forward in building large, fair test collections and summarize the consequences of possible bias on retrieval experiments.”
Information Retrieval: A Biomedical and Health Perspective (2020, unknown): “This was borne out in an analysis that assessed 10 years of articles in NEJM that concerned a medical practice and tested a new practice [142].”
Relevance assessment: are judges exchangeable and does it matter (2008, background): “[2]; each stratum can be thought of as containing only one document.”
Information Retrieval for E-Discovery (2013, background): “[4] describe an alternative, more complex unequal sampling technique for estimating average precision, but it does not enforce a fixed sample size.”