Some Common Mistakes In IR Evaluation, And How They Can Be Avoided

By Norbert Fuhr. 2018

154 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

TILDE: Term Independent Likelihood moDEl for Passage Re-ranking (2021, unknown): “6Despite recent, somewhat differing opinions expressed in the community [9, 23], and recent empirical evidence [34], MRR@10 is still the official (and only) measure used for reporting on MS MARCO, and thus we adapt to this convention to allow our results to be easily compared to past and future work”
Towards Meaningful Statements in IR Evaluation: Mapping Evaluation Measures to Interval Scales (2021, background): “On the other hand, Sakai [75] has recently disagreed with Fuhr: 1. in general, on the fact that means should not be computed for an ordinal scale, using arguments similar to those discussed in Section 2.3; 2. in particular, on the use of RR which Sakai finds quite useful from a practical point of vi”
Improving Hate Speech Detection with Deep Learning Ensembles (2018, background): “Similar concerns regarding details of neural network conﬁgurations have recently been raised in the information retrieval community as well (Fuhr, 2017).”
MS MARCO: Benchmarking Ranking Models in the Large-Data Regime (2021, background): “[23] have furthered the argument made by Fuhr [28] to point out that indeed for many well-known and commonly used IR metrics it is inappropriate to compute their average.”
Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction (2022, background): “Our main aim is to present an analysis grounded in robust evaluation [50, 58] that does not fall for common problematic shortcuts in IR evaluation like influence of effect sizes [11, 52], relying on too shallow pooled collections [2, 31, 53], not accounting for pool bias in old collections [5, 40, 4”