Improvements that don't add up: ad-hoc retrieval results since 1998

By Timothy G. Armstrong, Alistair Moffat, William Webber, Justin Zobel. 2009

260 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Management Science and Operations Research, Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Management Science and Operations Research

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “Usually, the gains from the combination of two innovations is smaller than the sum of the gain for each applied independently; see Armstrong et al. [2009] for more discussion.”
Troubling Trends in Machine Learning Scholarship (2018, background): “y for abandoning self-discipline, warning prophetically that “if we can’t criticize ourselves, someone else will save us the trouble”. Similar discussions recurred throughout the 80s, 90s, and aughts [13, 38, 2]. In other ﬁelds such as psychology, poor experimental standards have eroded trust in the”
MS MARCO: Benchmarking Ranking Models in the Large-Data Regime (2021, result): “There have been some claims that the field has a problem with this kind of validity [2] although that paper was not questioning that BM25 was an improvement, but rather questioning whether subsequent studies improved on methods such as BM25 from the 1990s.”
Cross-Domain Sentence Modeling for Relevance Transfer with BERT (2019, background): “Comparisons to weak baselines pervade the information retrieval literature [4, 34].”
To Clean or Not to Clean (2018, background): “Secondly, an improper choice of parsing method can lead to poor results or weak baselines, which may in turn lead to incorrect conclusions about the benefits of using various retrieval methodologies [5] (such as query expansion, for example).”
Improvements to BM25 and Language Models Examined (2014, unknown): “Keywords Relevance Ranking, Document Retrieval, Procrastination 1.”
A Study of Information Retrieval Weighting Schemes for Sentiment Analysis (2010, background): “…is a very active research area in information retrieval and it is outside the scope of this paper to provide an in-depth analysis but signiﬁ-cant research can be found in Salton and McGill (1986), Robertson et al. (2004), Manning et al. (2008) or Armstrong et al. (2009) for a more recent study.”
Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval (2019, unknown): “Comparisons to weak baselines that inﬂate the merits of an approach are not new problems in information retrieval (Armstrong et al., 2009), and researchers have in fact observed similar issues in the recommender systems literature as well (Ren-dle et al., 2019; Dacrema et al., 2019).”
Concept-Based Information Retrieval Using Explicit Semantic Analysis (2011, background): “As [Armstrong et al. 2009] recently pointed out, it is not sufficient for IR researchers to show improvement over their own baseline, rather they should strive to show that their method can improve over systems that are already highly effective.”
The neural hype, justified! (2019, background): “[2] seems to have received renewed vigor and attention, but whether this leads to concrete action and cultural change, it’s still too early to tell.”