The Impact of Score Ties on Repeatability in Document Ranking

By Jimmy J. Lin, Peilin Yang. 2018

22 citations recorded by the source corpus.

Document ranking experiments should be repeatable. However, the interaction between multi-threaded indexing and score ties during retrieval may yield non-deterministic rankings, making repeatability not as trivial as one might imagine. In the context of the open-source Lucene search engine, score ties are broken by internal document ids, which are assigned at index time. Due to multi-threaded indexing, which makes experimentation with large modern document collections practical, internal document ids are not assigned consistently between different index instances of the same collection, and thus score ties are broken unpredictably. This short paper examines the effectiveness impact of such score ties, quantifying the variability that can be attributed to this phenomenon. The obvious solution to this non-determinism and to ensure repeatable document ranking is to break score ties using external collection document ids. This approach, however, comes with measurable efficiency costs due to the necessity of consulting external identifiers during query evaluation.

Subfields: Artificial Intelligence, Computer Networks and Communications, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jimmy J. Lin
2. Sean MacAvaney
3. Nazli Goharian
4. Nicola Tonellotto
5. Ophir Frieder
6. Zeynep Akkalyoncu Yilmaz
7. Sebastian Hofstätter
8. Franco Maria Nardini

In Computer Networks and Communications

1. Hui Fang
2. Peilin Yang
3. Jimmy J. Lin
4. Ophir Frieder
5. Sean MacAvaney

In Information Systems

1. Jimmy J. Lin
2. Sean MacAvaney
3. Nazli Goharian
4. Ophir Frieder
5. Peilin Yang
6. Hui Fang
7. Nicola Tonellotto
8. Franco Maria Nardini

Citations in the corpus

• Anserini (2018, methodology): “For ClueWeb collections, query evaluation is around 20% slower compared to an arbitrary tie-breaking approach [32].”
• Training Curricula for Open Domain Answer Re-Ranking (2020, result): “restobeconsideredsimilarly(oridentically) in terms of difficulty. In the case of identical scores,norm allows to improve the reproducibility of the CL approach compared to curricula that rely on rank [22]. To account for various ranges in which ranking scores can appear, we apply min-max normalizati”