Which BM25 Do You Mean? A Large-Scale Reproducibility Study of Scoring Variants

By Chris Kamphuis, A. D. Vries, Leonid Boytsov, Jimmy J. Lin. 2020

67 citations recorded by the source corpus.

When researchers speak of BM25, it is not entirely clear which variant they mean, since many tweaks to Robertson et al.’s original formulation have been proposed. When practitioners speak of BM25, they most likely refer to the implementation in the Lucene open-source search library. Does this ambiguity “matter”? We attempt to answer this question with a large-scale reproducibility study of BM25, considering eight variants. Experiments on three newswire collections show that there are no significant effectiveness differences between them, including Lucene’s often maligned approximation of document length. As an added benefit, our empirical approach takes advantage of databases for rapid IR prototyping, which validates both the feasibility and methodological advantages claimed in previous work.

Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “We discuss these below: The dense passage retriever (DPR) of Karpukhin et al. [2020], reported in April 2020, adopts an architecture quite similar to Sentence-BERT to improve both passage retrieval as well as end-to-end effectiveness in open-domain question answering.”
Efficiency Implications of Term Weighting for Passage Retrieval (2020, result): “As expected, we are able to closely replicate the effectiveness scores suggested by earlier work [5, 19], with minor differences arising due to the underlying BM25 computation [10].”
Supporting Interoperability Between Open-Source Search Engines with the Common Index File Format (2020, background): “In short, we have replicated previous replicability studies [6, 14], but in a manner that supports cross-system comparisons.”
Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies (2026, unknown): “Okapi Best Match 25, more frequently referred to as just BM25 (Robertson et al., 1995), is a ubiquitous method in classical information retrieval which remains surprisingly competitive against modern neural methods, especially with proper per-dataset parameter tuning (Kamphuis et al., 2020).”