Information retrieval system evaluation: effort, sensitivity, and reliability

By Mark Sanderson, Justin Zobel. 2005

403 citations recorded by the source corpus.

Subfields: Computer Science Applications, Information Systems, Management Science and Operations Research, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems

In Management Science and Operations Research

In Information Retrieval & Search

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “s signiﬁcant or meaningful is more than just a matter of running a standard signiﬁcance test, but must consider a multitude of other factors, including all the ones discussed in Section 2.2 and more [Sanderson and Zobel, 2005]. Details of how this is accomplished depend on the task and vary from tra”
Information Retrieval Evaluation (2011, neutral): “Use of statistical testing can be done (see for example [97, 150, 159]) but has not been heavily adopted by the community.”
Test Collection Based Evaluation of Information Retrieval Systems (2010, methodology): “Later, Sanderson and Zobel [222] using Zobel’s [295] methodology examined the t-test, Wilcoxon and Sign tests.”
Learning to Reweight Terms with Distributed Representations (2015, methodology): “05 rather than the Wilcoxon signed rank test that was used in prior research [1], which is prone to type I errors and considered not reliable for ad hoc IR [17, 18].”
On Building Fair and Reusable Test Collections using Bandit Techniques (2018, background): “Other suggested ways to reduce the number of judgments required include recommendations to use many topics with relatively few judgments [11] per topic, or to find a small set of topics that will adequately compare runs [6–8].”
Tie-Breaking Bias: Effect of an Uncontrolled Parameter on Information Retrieval Evaluation (2010, methodology): “Following on from studies on IR evaluation methodology such as Voorhees’s [8] and Zobel’s [9] we identified an uncontrolled parameter in Trec through trec eval: evaluation results not only depend on retrieved documents, but also on how they were named in case of ties (i.e., ex aequo documents).”
Information Retrieval for E-Discovery (2013, background): “Studies on early TREC collections indicate that in such collections pooling does manage to find up to half the relevant documents; that, although absolute scores may be inaccurate (particularly if they include recall) and sensitive to variability in pooling, comparative scores are fairly stable; and”
Redundant documents and search effectiveness (2005, methodology): “In order to discount the e.ect of poorly performing runs we adopted the methodology used by Voorhees & Buckley (2002) and Sanderson & Zobel (2005) and discarded the bottom 25% of runs, leaving us in this case with 54 runs.”
Feeling lucky?: multi-armed bandits for ordering judgements in pooling-based evaluation (2016, background): “With a given budget for doing assessments, the more relevant documents we find, the more reliable the evaluation will be [16].”
Formal models for expert finding in enterprise corpora (2006, methodology): “expert finding approaches are statistically significant, we use Student’s t-test, and look for significant improvements (onetailed) at a significance level of 0.95 [ 16 ].”
MS MARCO: Benchmarking Ranking Models in the Large-Data Regime (2021, background): “Previous work [35, 67] has indicated that violation of certain assumptions by some”
Improvements that don't add up: ad-hoc retrieval results since 1998 (2009, methodology): “The validity of the new technique is tested by comparing its score to the baseline, and any apparent improvement is then tested with a statistical signiﬁcance test [Zobel, 1998, Sanderson and Zobel, 2005, Smucker et al., 2007].”
Score Aggregation Techniques in Retrieval Experimentation (2009, methodology): “The vertical dotted lines on the curves indicate the means of the four density distributions.”