The Effect of Content-Equivalent Near-Duplicates on the Evaluation of Search Engines

By Maik Fröbe, J. P. Bittner, Martin Potthast, Matthias Hagen. 2020

23 citations recorded by the source corpus.

Current best practices for the evaluation of search engines do not take into account duplicate documents. Dependent on their prevalence, not discounting duplicates during evaluation artificially inflates performance scores, and, it penalizes those whose search systems diligently filter them. Although these negative effects have already been demonstrated a long time ago by Bernstein and Zobel [4], we find that this has failed to move the community. In this paper, we reproduce the aforementioned study and extend it to incorporate all TREC Terabyte, Web, and Core tracks. The worst-case penalty of having filtered duplicates in any of these tracks were losses between 8 and 53 ranks.

Subfields: Information Systems, Sociology and Political Science, Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Sociology and Political Science

Andrew Yates

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “cle was retrieved more than once and thus entered the judgment pool multiple times. While researchers have previously studied the impact of near-duplicate documents, e.g., [Bernstein and Zobel, 2005, Fröbe et al., 2020], a thorough analysis has not been applied to TREC-COVID. Takeaway Lessons. We ha”
Overview of Touché 2022: Argument Retrieval (2022, unknown): “From the initial 1,286,977 passages we removed near-duplicates with CopyCat [29] to mitigate negative impacts [30, 31], resulting in the final collection of 868,655 passages.”