Retrieval evaluation with incomplete information

By Chris Buckley, Ellen M. Voorhees. 2004

811 citations recorded by the source corpus.

Subfields: Information Systems, Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

Findings of the 2014 Workshop on Statistical Machine Translation (2014, background): “The results are provided in terms of standard IR evaluation measures: precision at a cut-off of 5 and 10 documents (P@5, P@10), normalized discounted cumulative gain (Järvelin and Kekäläinen, 2002) at 5 and 10 documents (NDCG@5, NDCG@10), mean average precision (MAP) (Voorhees and Harman, 2005), pre”
Human Preferences as Dueling Bandits (2022, neutral): “The goal is to make the collection re-usable, so that future experiments conducted on the collection provide meaningful results, even if these new experimental runs surface unjudged documents in their top ranks[5, 31].”
CNN architectures for large-scale audio classification (2016, background): “ of the Average Precision (AP), which is the proportion of positive items in a ranked list of trials (i.e., Precision) averaged across lists just long enough to include each individual positive trial [28]. AP is widely used as an indicator of precision that does not require a particular retrieval li”
End-to-End Retrieval in Continuous Space (2018, methodology): “This issue is discussed at length by Buckley and Voorhees (2004), who show that the Mean Average Precision (MAP) metric computed on an incomplete evaluation set correlates reasonably well with the MAP metric computed on a (signiﬁcantly more) complete version of that evaluation set.”
Information Retrieval Evaluation (2011, background): “A new measure bpref [35] specifically designed for incomplete collections was tried, and the pools were made larger (top 100 documents) in 2005.”
Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction (2022, background): “in IR evaluation like influence of effect sizes [11, 53], relying on too shallow pooled collections [2, 33, 54], not accounting for pool bias in old collections [5, 41, 42], and aggregating metrics over different collections which are not comparable [46].”
When will information retrieval be "good enough"? (2005, background): “HARD track [2], because it has been shown both to track mean average precision in ranking system output, and to be a highly stable measure when relevance judgments are sparse [7].”
Automatic Keyphrase Extraction via Topic Decomposition (2010, background): “One metric is binary preference measure (Bpref) (Buckley and Voorhees, 2004).”
A Closer Look at Weak Label Learning for Audio Events (2018, methodology): “We use average precision (AP) [46] and area under ROC curves for each event (AUC) [47] as evaluation metrics.”
DeepInf: Social Influence Prediction with Deep Learning (2018, methodology): “To evaluate our framework quantitatively, we use the following performance metrics: Prediction Performance We evaluate the predictive performance ofDeepInfinterms ofArea UnderCurve(AUC) [8],Precision(Prec.)”
Tangent-CFT: An Embedding Model for Mathematical Formulas (2019, background): “For instance, considering n-grams, n = [3, 6], the word ‘system’ (with no boundary symbols) is presented as sys, yst, ste, tem, syst, yste, stem, syste, ystem and system.”
SeCoST: Sequential Co-Supervision for Weakly Labeled Audio Event Detection (2019, methodology): “Similar to existing AED works [6], [10], [7], Average precision (AP) and area under ROC (AUC) are used to measure the performance [28], [29].”
Expertise modeling for matching papers with reviewers (2007, methodology): “As a result, we use pooled relevance judgments [6].”
Social influence analysis in large-scale networks (2009, methodology): “We evaluate the performance of different methods in terms of Precision@5 (P@5), P@10, P@20, R-precision (R-Pre), and mean average precision (MAP) [4, 7].”
Bias and the limits of pooling for large collections (2007, methodology): “For both collections, MAP and bpref—a measure designed to be used in the presence of incomplete judgments and shown to be highly correlated with MAP when given relatively complete judgments (Buckley and Voorhees 2004)— ranked systems almost identically.”
Introduction to the special topic section on mining Web resources for enhancing information retrieval (2007, methodology): “For both retrieval tasks, the retrieval performances are evaluated by mean average precision (MAP) and B-pref ( Buckley & Voorhees, 2004 ), which are two of the most frequently used metrics in Web search research (they are also adopted by TREC as the major metrics).”
Alternatives to Bpref (2007, methodology): “The contants 1 and 10 have been copied from [3], representing the SIGIR 2007 Proceedings Session 3: Evaluation”