Toward Reproducible Baselines: The Open-Source IR Reproducibility Challenge

By Jimmy J. Lin, Matt Crane, Andrew Trotman, Jamie Callan, Ishan Chattopadhyaya, John Foley, Grant Ingersoll, Craig Macdonald, Sebastiano Vigna. 2016

111 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Multi-Stage Document Ranking with BERT (2019, background): “It is a well-known fact in IR that different systems implementing the same scoring function might report very different results (Mühleisen et al., 2014; Lin et al., 2016), owing to details such as tokenization, stopword selection, stemming, and parameter tuning.”
Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations (2021, methodology): “Our group’s efforts to promote and support replicable IR research dates back to 2015 [4, 14], and the landscape has changed quite a bit since then.”
Anserini: Enabling the Use of Lucene for Information Retrieval Research (2017, background): “ese results are consistent with ndings from the recent Open-Source IR Reproducibility Challenge [10].”
Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations (2021, methodology): “Pyserini began as the Python interface to Anserini [38, 39], which our group has been developing for several years, with its roots in a community-wide reproducibility exercise dating back to 2015 [15].”
Query Driven Algorithm Selection in Early Stage Retrieval (2017, methodology): “shown that using an exhaustive ρ results in effective top-k retrieval, however, using a heuristic ρ can give similar effectiveness, yet much more efficient retrieval [31, 32].”
Supporting Interoperability Between Open-Source Search Engines with the Common Index File Format (2020, methodology): “this approach requires n ×( n − 1 ) distinct wrappers to support inter-operability between n systems, as every system would need to wrap”
A Comparison of Document-at-a-Time and Score-at-a-Time Query Evaluation (2017, methodology): “One variant of JASS was the fastest in a recent reproducibility evaluation comparing several open-source search engines [21], and thus JASS represents the state of the art.”
The Lucene for Information Access and Retrieval Research (LIARR) Workshop at SIGIR 2017 (2017, background): “With respect to the second and third points, these perceptions are no longer accurate today: empirical studies [11, 23] have shown that the eectiveness of Lucene’s baseline retrieval models is at least as good as those in academic IR systems, and that Lucene is capable of high-throughput multi-thre”
BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models (2021, methodology): “We replicate the setup of Nogueira and Lin [47] and generate 40 queries for each document and use BM25 with default Anserini parameters.”
Applying BERT to Document Retrieval with Birch (2019, background): “Further standardizing on a common platform within the academic community can foster greater replicability and reproducibility, a growing concern in the community (Lin et al., 2016).”
To Clean or Not to Clean (2018, methodology): “Given the importance of being able to reproduce experimental results in IR, it is not surprising that the issue of reproducibility has become a specific focus area at a number of recent workshops and conferences [4, 32, 33, 40, 49].”
Identification of Reproducible Subsets for Data Citation, Sharing and Re-Use (2016, background): “Currently, there is a lack of reproducibility and many workflows cannot be re-executed [28, 17, 13], although recent initiatives aim to demand the reproducibility of submitted contributions [16, 9].”
Metrology for AI: From Benchmarks to Instruments (2019, background): “(Arguello et al. 2016) and (Lin et al. 2016) have identified issues with reproducibility in the information retrieval community.”
The Impact of Score Ties on Repeatability in Document Ranking (2018, methodology): “The system evolved from previous IR reproducibility experiments [2, 5] where”
BERT-QE: Contextualized Query Expansion for Document Re-ranking (2020, methodology): “- BM25+RM3 is also used as baseline, which follows the experimental settings from (Yilmaz et al., 2019), and the implementation from Anserini (Lin et al., 2016) with default settings is used.”